esh-pfi-infrastructure/docs/orientation.md

# Fleet orientation — start here

Read this first if you're a fresh session joining this project. It
summarizes the fleet, the conventions, the known gotchas, and points
at the deeper docs / memory files that cover each area in detail.

## What this repo is

A **reference + management workspace** for PFI infrastructure — a
small multi-site fleet of Proxmox hypervisors, Linux VMs/LXCs, a
bare-metal GPU host, and two NAS boxes. Authoritative copies of
Docker compose files, backup configs, and homepage dashboard config
live **on the hosts themselves**; this repo mirrors them for version
control, planning, and cross-host orchestration.

Primary file you'll read automatically on session start:

- `CLAUDE.md` — conventions, servers table, scripts tour, placement rules.
- `~/.claude/projects/.../memory/MEMORY.md` + linked memory files —
  durable cross-session facts.

This file (`docs/orientation.md`) is the narrative overview that ties
them together.

## Sites + hosts at a glance

Three physical locations plus one WireGuard-tunneled satellite:

| Site | Subnet | Physical boxes | Notes |
|---|---|---|---|
| **Anaheim (ANA)** | `10.250.0.0/16` | Dell R750xs (pfi-pve hypervisor, iDRAC 10.250.250.30), Supermicro (ana-ml2 bare-metal GPU, BMC 10.250.250.50), Dell R630 (sfsrv-ana SureFire tenant, iDRAC 10.250.250.110) | Primary production site |
| **NH3** | `10.100.0.0/16` | nh3-pve hypervisor, Synology RS2418+ (`nh3-nas`, 42 TB) | Secondary + DR target |
| **ESH home lab** | `esteban.net`, `10.0.50.0/24` | esh-pve, esh-pve-nas (split-role hypervisors), esh-vm-db (DB VM) | Non-production, shares backup tooling |
| **Irvine (IRV)** | `10.100.79.0/24` (WG tunnel only) | irv-ml1 bare-metal GPU box | Reachable ONLY via WireGuard tunnel from NH3 |

Full host list with roles: see the **Servers table** in `CLAUDE.md`.

## Backup architecture (two layers)

This is the most fleet-impacting system. Every host gets backed up
via one or both layers.

### Layer 1 — PBS (VM-image, block-level dedupe)

- **Primary:** `pbs-ana` (VM on pfi-pve, NFS datastore on ana-nas).
- **DR mirror:** `pbs-nh3` (VM on nh3-pve, NFS datastore on nh3-nas Synology).
- **Sync:** pbs-nh3 pulls from pbs-ana at 06:00 daily. One-way.
- **Namespaces:** per-hypervisor (`pfi-pve`, `nh3-pve`, `esh-pve`,
  `esh-pve-nas`, `sfsrv-ana`) because VMIDs are NOT globally unique.
- **Scope:** all 5 hypervisors back up to pbs-ana nightly.

Deployment runbook: `docs/runbooks/pbs-deployment.md` (9 phases,
captured during the 2026-04-22 deployment).

### Layer 2 — restic (file-level, encrypted, DB-consistent)

- **Target A:** `rest-server-ana` container on ana-docker. Data dir
  NFS-mounted from ana-nas. Used by: ana-docker, ana-ml2,
  esh-docker-vm, vm-esh-nas, esh-vm-db.
- **Target B:** `rest-server-nh3` container on nh3-nas (Synology).
  Used by: nh3-docker, nh3-dev, irv-ml1.
- **Mode:** both run `--append-only --private-repos` — clients can't
  delete their own history, and each client's URL path must match
  their HTTP basic-auth user.
- **Scope:** 8 hosts have resticprofile + systemd timers firing at
  01:00 daily.
- **DB-consistent dumps:** hosts with databases have a
  `pre-backup.sh` that runs pg_dump/mongodump/sqlite3 .backup into
  `/var/lib/restic/stage/` before restic sweeps.

### Cross-site restic mirror (rsync)

- **ana-nas → nh3-nas** daily at 04:00 (runs on ana-nas as lkraven).
- **nh3-nas → ana-nas** daily at 05:00 (runs on nh3-nas as root —
  rest-server-nh3's container writes files mode 400 owned by admin,
  only root can read them on the Synology side).

Tracked at `configs/rsync/{ana-nas-to-nh3,nh3-nas-to-ana}/`.

## Governing principles (durable rules)

Saved in feedback-type memory files; apply by default.

1. **Backups must not risk production.** If a backup process could
   crash the host it's backing up, change the strategy or skip the
   backup. Reference: CT 109 (ana-nas) self-backup crash of
   2026-04-23. Memory: `feedback_backups_must_not_risk_production.md`.
2. **DB data lives on local disk, not NFS.** Decision 2026-04-23
   after pfi-postgres migration off `/mnt/db`. New DB deployments
   default to local storage; shared NFS for DB data is an anti-pattern
   in this environment. Memory: `project_db_migrate_off_nfs.md`.
3. **Use `ssh -t` when the remote runs sudo.** Read-only probes and
   piped scripts can stay plain `ssh`. Memory: `feedback_ssh_sudo.md`.
4. **Handle git commands autonomously.** Stage + commit at natural
   boundaries without asking; verify risky changes first; never
   force-push or push without explicit approval. Memory:
   `feedback_git_autonomous.md`.
5. **No Claude attribution in commits.** No `Co-Authored-By`,
   "Generated with...", robot emojis, etc. Memory:
   `feedback_git_commits.md`.

## Known gotchas (save yourself time)

These caught us once; don't let them catch you twice.

### NFS / storage

- **ana-nas is `CT 109` (LXC), not `VM 100`.** Persistent confusion
  historically — the Proxmox VM name was `PFI-ANA-TRUENAS` for ages
  but the host was reprovisioned to Debian, and it's an LXC container
  not a VM. Memory: `storage_ana_nas.md`.
- **ana-nas is a SPOF for rest-server-ana + PBS-ANA datastore.** When
  multiple fleet services fail together, check `ping 10.250.50.50`
  FIRST. Memory: `incident_ana_nas_spof.md`. Recovery runbook:
  `docs/runbooks/disaster-recovery.md`.
- **NFS root_squash bites in two directions:**
  - **Sudo-root on the client writing to an NFS file** gets mapped to
    nobody. Doesn't matter who owns the file.
  - **rsync-as-root of NFS-owned data** fails the same way. Workaround
    for postgres-on-NFS migration: run rsync as the `postgres` user,
    whose uid has legitimate NFS access.
- **ZFS with `casesensitivity=insensitive` + NFSv4 = EACCES on writes.**
  PBS-ANA's datastore uses NFSv3 mount to work around this. Don't
  "upgrade" to NFSv4 without fixing the dataset first.
- **Synology `.htpasswd` edits must be done as the file owner** (uid
  1000 = lkraven for rest-server-ana), NOT via sudo. sudo-root gets
  squashed to nobody and can't read/write the file.
- **Synology Advanced Permissions (syno_acl)** imposes NFSv4 ACLs that
  shadow POSIX perms. "Enable Advanced Permissions = off" doesn't
  reset it once set. Fix for PBS data: `chmod 777` on the share to
  flatten to "Linux mode" (confirm with `synoacltool -get`).
- **DSM Container Manager** doesn't expose `sudo docker` in root's
  PATH. Use DSM web UI to restart containers, or find the actual
  binary via `find`.
- **scp to DSM** requires `-O` flag (legacy scp protocol) — default
  sftp subsystem is disabled.

### systemd

- **DSM ships a pre-v247 systemd.** Refuses `Restart=on-failure` on
  `Type=oneshot`, doesn't recognize `StartLimitBurst/IntervalSec`,
  `systemctl --now` flag absent. Write unit files with plain oneshot
  semantics for DSM; the daily timer retry compensates for missed
  runs.
- **fstab with bare `defaults`** doesn't auto-retry a failed NFS
  mount. If the NFS server wasn't up at boot, `mnt-*.mount` sits in
  failed state until manual `systemctl reset-failed` + `systemctl start`
  (or simpler: `mount -a`).
- **Ghost files on NFS mount points.** When NFS unmounts while writes
  are still landing at the mount point path, the writes go to the
  underlying local directory. When NFS remounts, the local ghost is
  shadowed. When NFS later unmounts again, the ghost reappears and
  breaks whatever was reading that path. See `incident_ana_nas_spof.md`
  for the exact case on ana-docker.

### Naming / identifiers

- **Hardware vs OS distinction** for servers with iDRAC/BMC:
  - `pfi-pve` (Dell R750xs) ↔ `PFI-r750xs-iDRAC` at 10.250.250.30
  - `sfsrv-ana` (Dell R630) ↔ `SF-R630-iDRAC` at 10.250.250.110
  - `ana-ml2` (Supermicro) ↔ `PFI-ANA-ML2 BMC` at 10.250.250.50
  Each `servers/<name>/` dir documents the OS side; hardware/iDRAC
  entries exist in homepage but are separate logical identities.
- **irv-ml1 was `ana-ml1`** before a physical move; OS hostname still
  says `ana-ml1` pending an explicit rename. Doesn't affect services.

### Git / gitea

- **Colo/fleet hosts must reach gitea over the INTERNAL route, not the
  public IP.** `gitea.phasefinal.com` resolves to the **public** IP
  `38.120.12.44` (ana-srv1); gitea itself is a container on **ana-docker**
  with git-SSH at **`10.250.50.70:222`** (`222→22`) and HTTP at `:3000`.
  A fleet host that egresses to the public `:22` gets its egress IP
  **fail2ban-banned** after any retrying git/deploy loop, which silently
  wedges automation — e.g. a gitea-webhook auto-deploy whose `git fetch`
  then times out under `set -euo pipefail` and never reaches the `reset`.
  Point each host's gitea ssh alias at `HostName 10.250.50.70` /
  `Port 222` with the repo deploy key; the internal route is ban-immune
  and treats the cause. Bit irv-ml1's arbo deploy on 2026-06-13 (the
  `gitea-arbo` alias pointed at the public host → fetch timeout → the
  v0.11.7 frontend wouldn't serve until the alias was repointed internal).
- **`:22` on `10.250.50.70` is ana-docker's HOST sshd, not gitea.** A
  gitea deploy key there returns `Permission denied (publickey)` — gitea's
  git-SSH is the container port `:222`. (HTTP/clone-over-HTTPS is `:3000`.)

### Workflow

- **Terminal word-wrap breaks long pasted commands.** Never embed a
  580-char pubkey or long URL inline in a multi-line shell command —
  terminal wraps the line, shell captures literal newlines, and the
  command fails in bizarre ways. Use `cat` from a file, `printf`, or
  a tempfile + scp.
- **zsh interactive mode doesn't parse `#` as comment by default.**
  Paste multi-line blocks with comment lines and zsh tries to execute
  `#` as a command. Either tell user to `setopt interactivecomments`
  or keep blocks comment-free.
- **`ssh -t` with piped stdin fails** — `-t` allocates a tty, stdin
  from a pipe can't be a tty. Options: don't use `-t` (give up
  interactive sudo prompt), or scp the file first then `ssh -t` for
  the sudo bits.

## Where to look for what

| Need | Go here |
|---|---|
| Is host X alive? | `servers/<host>/system-details.txt` (snapshot). Refresh with `scripts/refresh-server-info.sh <host>`. |
| What does host X run? | `servers/<host>/README.md` |
| What's the backup strategy for host X? | `configs/restic/<host>/README.md` + `profiles.yaml` |
| How do I recover from host X going down? | `docs/runbooks/disaster-recovery.md` |
| How was PBS deployed? | `docs/runbooks/pbs-deployment.md` |
| How do I run the quarterly NH3 restic prune? | `docs/runbooks/nh3-prune-ritual.md` (or just `scripts/restic-prune.sh nh3`) |
| What's currently open / in-flight? | `STATUS.md` |
| What do I need to know that isn't in current code? | `MEMORY.md` + the `.md` files it links |
| Why did we do X? | Check memory files + `STATUS.md` session milestones at the bottom |
| **I need to quantize / requant a model** | **`docs/pfi/model-quantization-playbook.md` — READ IT FIRST.** Consolidated hard-won lessons (scheme choice, the recurring landmines, the acceptance gate, superseded claims). Per-model runbooks are worked examples, not the general guide. |
| What sampler/serve settings for model X? | `docs/pfi/recommended-model-settings.md` |
| Which model is on which GPU seat? | `servers/ana-ml2/README.md` + `stacks/<seat>/README.md` |

## Inventory + automation scripts

Under `scripts/`:

- `server_inspect.sh` — pipes over SSH to generate `system-details.txt`
  (Docker hosts mostly; works on any Debian-ish Linux).
- `refresh-server-info.sh <host>` or `all` — fleet-wide refresh
  driver. `--validate-only` for dry-run.
- `proxmox_inspect.sh` + `refresh-proxmox-info.sh` — Proxmox-flavored
  variant that captures `qm list`, `pct list`, zpool, storage.
- `discover-fortigate.sh`, `discover-unifi.sh`, `discover-gaps.sh` —
  network-level inventory diff against `servers/*/`.
- `sync-stacks.sh` / `deploy-stack.sh` — compose mirror pull/push.
- `add-host.sh <name> <user@ip>` — canonical host registration.

## Current state (2026-04-23 snapshot)

- **Host inventory:** ~22 tracked hosts under `servers/` across ANA,
  NH3, ESH, IRV. Every host has SSH config alias for convenience.
- **Backup coverage:** 8 restic clients, 5 hypervisors on PBS with
  DR mirror, cross-site rsync both directions operational.
- **Recent migrations:** pfi-postgres → local disk (removes ana-nas
  cascade); offen sidecars retired on esh-docker-vm (16 GB reclaimed);
  irv-ml1 + esh-vm-db fully onboarded.
- **Recent secret rotations:** 6 credentials — all gitea/vaultwarden/
  paperless DB passwords + 3 rest-server htpasswd + repo passphrases.

Active watchlist (see `STATUS.md` for full detail):

- pfi-pve backplane repair pending tech visit
- Postgres NFS retirement (block 4+5) pending 24h burn-in
- Tomorrow 01:00: first scheduled restic runs for esh-vm-db + irv-ml1
- PBS Phase 7-8 (retire legacy vzdump) gated on 2026-04-29

## First commands a fresh session should be comfortable running

```bash
# Fleet validation (read-only, safe)
scripts/refresh-server-info.sh --validate-only all

# Specific host refresh
scripts/refresh-server-info.sh <hostname>

# Look at current fleet state
cat STATUS.md

# Explore memory
ls /home/lkraven/.claude/projects/-home-lkraven-development-eshpfi-management/memory/
cat /home/lkraven/.claude/projects/-home-lkraven-development-eshpfi-management/memory/MEMORY.md

# Probe any host
ssh <host>  # alias resolves via ~/.ssh/config
```

## When in doubt

- Read `CLAUDE.md` for conventions and the Servers table.
- Read `STATUS.md` for what's in-flight right now.
- Check `memory/MEMORY.md` for durable facts about the user's
  preferences + project history.
- For recovery questions: `docs/runbooks/disaster-recovery.md`.
- Ask. User is experienced — doesn't need hand-holding, but a clear
  question is faster than a wrong assumption.