# Fleet orientation — start here Read this first if you're a fresh session joining this project. It summarizes the fleet, the conventions, the known gotchas, and points at the deeper docs / memory files that cover each area in detail. ## What this repo is A **reference + management workspace** for PFI infrastructure — a small multi-site fleet of Proxmox hypervisors, Linux VMs/LXCs, a bare-metal GPU host, and two NAS boxes. Authoritative copies of Docker compose files, backup configs, and homepage dashboard config live **on the hosts themselves**; this repo mirrors them for version control, planning, and cross-host orchestration. Primary file you'll read automatically on session start: - `CLAUDE.md` — conventions, servers table, scripts tour, placement rules. - `~/.claude/projects/.../memory/MEMORY.md` + linked memory files — durable cross-session facts. This file (`docs/orientation.md`) is the narrative overview that ties them together. ## Sites + hosts at a glance Three physical locations plus one WireGuard-tunneled satellite: | Site | Subnet | Physical boxes | Notes | |---|---|---|---| | **Anaheim (ANA)** | `10.250.0.0/16` | Dell R750xs (pfi-pve hypervisor, iDRAC 10.250.250.30), Supermicro (ana-ml2 bare-metal GPU, BMC 10.250.250.50), Dell R630 (sfsrv-ana SureFire tenant, iDRAC 10.250.250.110) | Primary production site | | **NH3** | `10.100.0.0/16` | nh3-pve hypervisor, Synology RS2418+ (`nh3-nas`, 42 TB) | Secondary + DR target | | **ESH home lab** | `esteban.net`, `10.0.50.0/24` | esh-pve, esh-pve-nas (split-role hypervisors), esh-vm-db (DB VM) | Non-production, shares backup tooling | | **Irvine (IRV)** | `10.100.79.0/24` (WG tunnel only) | irv-ml1 bare-metal GPU box | Reachable ONLY via WireGuard tunnel from NH3 | Full host list with roles: see the **Servers table** in `CLAUDE.md`. ## Backup architecture (two layers) This is the most fleet-impacting system. Every host gets backed up via one or both layers. ### Layer 1 — PBS (VM-image, block-level dedupe) - **Primary:** `pbs-ana` (VM on pfi-pve, NFS datastore on ana-nas). - **DR mirror:** `pbs-nh3` (VM on nh3-pve, NFS datastore on nh3-nas Synology). - **Sync:** pbs-nh3 pulls from pbs-ana at 06:00 daily. One-way. - **Namespaces:** per-hypervisor (`pfi-pve`, `nh3-pve`, `esh-pve`, `esh-pve-nas`, `sfsrv-ana`) because VMIDs are NOT globally unique. - **Scope:** all 5 hypervisors back up to pbs-ana nightly. Deployment runbook: `docs/runbooks/pbs-deployment.md` (9 phases, captured during the 2026-04-22 deployment). ### Layer 2 — restic (file-level, encrypted, DB-consistent) - **Target A:** `rest-server-ana` container on ana-docker. Data dir NFS-mounted from ana-nas. Used by: ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas, esh-vm-db. - **Target B:** `rest-server-nh3` container on nh3-nas (Synology). Used by: nh3-docker, nh3-dev, irv-ml1. - **Mode:** both run `--append-only --private-repos` — clients can't delete their own history, and each client's URL path must match their HTTP basic-auth user. - **Scope:** 8 hosts have resticprofile + systemd timers firing at 01:00 daily. - **DB-consistent dumps:** hosts with databases have a `pre-backup.sh` that runs pg_dump/mongodump/sqlite3 .backup into `/var/lib/restic/stage/` before restic sweeps. ### Cross-site restic mirror (rsync) - **ana-nas → nh3-nas** daily at 04:00 (runs on ana-nas as lkraven). - **nh3-nas → ana-nas** daily at 05:00 (runs on nh3-nas as root — rest-server-nh3's container writes files mode 400 owned by admin, only root can read them on the Synology side). Tracked at `configs/rsync/{ana-nas-to-nh3,nh3-nas-to-ana}/`. ## Governing principles (durable rules) Saved in feedback-type memory files; apply by default. 1. **Backups must not risk production.** If a backup process could crash the host it's backing up, change the strategy or skip the backup. Reference: CT 109 (ana-nas) self-backup crash of 2026-04-23. Memory: `feedback_backups_must_not_risk_production.md`. 2. **DB data lives on local disk, not NFS.** Decision 2026-04-23 after pfi-postgres migration off `/mnt/db`. New DB deployments default to local storage; shared NFS for DB data is an anti-pattern in this environment. Memory: `project_db_migrate_off_nfs.md`. 3. **Use `ssh -t` when the remote runs sudo.** Read-only probes and piped scripts can stay plain `ssh`. Memory: `feedback_ssh_sudo.md`. 4. **Handle git commands autonomously.** Stage + commit at natural boundaries without asking; verify risky changes first; never force-push or push without explicit approval. Memory: `feedback_git_autonomous.md`. 5. **No Claude attribution in commits.** No `Co-Authored-By`, "Generated with...", robot emojis, etc. Memory: `feedback_git_commits.md`. ## Known gotchas (save yourself time) These caught us once; don't let them catch you twice. ### NFS / storage - **ana-nas is `CT 109` (LXC), not `VM 100`.** Persistent confusion historically — the Proxmox VM name was `PFI-ANA-TRUENAS` for ages but the host was reprovisioned to Debian, and it's an LXC container not a VM. Memory: `storage_ana_nas.md`. - **ana-nas is a SPOF for rest-server-ana + PBS-ANA datastore.** When multiple fleet services fail together, check `ping 10.250.50.50` FIRST. Memory: `incident_ana_nas_spof.md`. Recovery runbook: `docs/runbooks/disaster-recovery.md`. - **NFS root_squash bites in two directions:** - **Sudo-root on the client writing to an NFS file** gets mapped to nobody. Doesn't matter who owns the file. - **rsync-as-root of NFS-owned data** fails the same way. Workaround for postgres-on-NFS migration: run rsync as the `postgres` user, whose uid has legitimate NFS access. - **ZFS with `casesensitivity=insensitive` + NFSv4 = EACCES on writes.** PBS-ANA's datastore uses NFSv3 mount to work around this. Don't "upgrade" to NFSv4 without fixing the dataset first. - **Synology `.htpasswd` edits must be done as the file owner** (uid 1000 = lkraven for rest-server-ana), NOT via sudo. sudo-root gets squashed to nobody and can't read/write the file. - **Synology Advanced Permissions (syno_acl)** imposes NFSv4 ACLs that shadow POSIX perms. "Enable Advanced Permissions = off" doesn't reset it once set. Fix for PBS data: `chmod 777` on the share to flatten to "Linux mode" (confirm with `synoacltool -get`). - **DSM Container Manager** doesn't expose `sudo docker` in root's PATH. Use DSM web UI to restart containers, or find the actual binary via `find`. - **scp to DSM** requires `-O` flag (legacy scp protocol) — default sftp subsystem is disabled. ### systemd - **DSM ships a pre-v247 systemd.** Refuses `Restart=on-failure` on `Type=oneshot`, doesn't recognize `StartLimitBurst/IntervalSec`, `systemctl --now` flag absent. Write unit files with plain oneshot semantics for DSM; the daily timer retry compensates for missed runs. - **fstab with bare `defaults`** doesn't auto-retry a failed NFS mount. If the NFS server wasn't up at boot, `mnt-*.mount` sits in failed state until manual `systemctl reset-failed` + `systemctl start` (or simpler: `mount -a`). - **Ghost files on NFS mount points.** When NFS unmounts while writes are still landing at the mount point path, the writes go to the underlying local directory. When NFS remounts, the local ghost is shadowed. When NFS later unmounts again, the ghost reappears and breaks whatever was reading that path. See `incident_ana_nas_spof.md` for the exact case on ana-docker. ### Naming / identifiers - **Hardware vs OS distinction** for servers with iDRAC/BMC: - `pfi-pve` (Dell R750xs) ↔ `PFI-r750xs-iDRAC` at 10.250.250.30 - `sfsrv-ana` (Dell R630) ↔ `SF-R630-iDRAC` at 10.250.250.110 - `ana-ml2` (Supermicro) ↔ `PFI-ANA-ML2 BMC` at 10.250.250.50 Each `servers//` dir documents the OS side; hardware/iDRAC entries exist in homepage but are separate logical identities. - **irv-ml1 was `ana-ml1`** before a physical move; OS hostname still says `ana-ml1` pending an explicit rename. Doesn't affect services. ### Workflow - **Terminal word-wrap breaks long pasted commands.** Never embed a 580-char pubkey or long URL inline in a multi-line shell command — terminal wraps the line, shell captures literal newlines, and the command fails in bizarre ways. Use `cat` from a file, `printf`, or a tempfile + scp. - **zsh interactive mode doesn't parse `#` as comment by default.** Paste multi-line blocks with comment lines and zsh tries to execute `#` as a command. Either tell user to `setopt interactivecomments` or keep blocks comment-free. - **`ssh -t` with piped stdin fails** — `-t` allocates a tty, stdin from a pipe can't be a tty. Options: don't use `-t` (give up interactive sudo prompt), or scp the file first then `ssh -t` for the sudo bits. ## Where to look for what | Need | Go here | |---|---| | Is host X alive? | `servers//system-details.txt` (snapshot). Refresh with `scripts/refresh-server-info.sh `. | | What does host X run? | `servers//README.md` | | What's the backup strategy for host X? | `configs/restic//README.md` + `profiles.yaml` | | How do I recover from host X going down? | `docs/runbooks/disaster-recovery.md` | | How was PBS deployed? | `docs/runbooks/pbs-deployment.md` | | How do I run the quarterly NH3 restic prune? | `docs/runbooks/nh3-prune-ritual.md` (or just `scripts/restic-prune.sh nh3`) | | What's currently open / in-flight? | `STATUS.md` | | What do I need to know that isn't in current code? | `MEMORY.md` + the `.md` files it links | | Why did we do X? | Check memory files + `STATUS.md` session milestones at the bottom | ## Inventory + automation scripts Under `scripts/`: - `server_inspect.sh` — pipes over SSH to generate `system-details.txt` (Docker hosts mostly; works on any Debian-ish Linux). - `refresh-server-info.sh ` or `all` — fleet-wide refresh driver. `--validate-only` for dry-run. - `proxmox_inspect.sh` + `refresh-proxmox-info.sh` — Proxmox-flavored variant that captures `qm list`, `pct list`, zpool, storage. - `discover-fortigate.sh`, `discover-unifi.sh`, `discover-gaps.sh` — network-level inventory diff against `servers/*/`. - `sync-stacks.sh` / `deploy-stack.sh` — compose mirror pull/push. - `add-host.sh ` — canonical host registration. ## Current state (2026-04-23 snapshot) - **Host inventory:** ~22 tracked hosts under `servers/` across ANA, NH3, ESH, IRV. Every host has SSH config alias for convenience. - **Backup coverage:** 8 restic clients, 5 hypervisors on PBS with DR mirror, cross-site rsync both directions operational. - **Recent migrations:** pfi-postgres → local disk (removes ana-nas cascade); offen sidecars retired on esh-docker-vm (16 GB reclaimed); irv-ml1 + esh-vm-db fully onboarded. - **Recent secret rotations:** 6 credentials — all gitea/vaultwarden/ paperless DB passwords + 3 rest-server htpasswd + repo passphrases. Active watchlist (see `STATUS.md` for full detail): - pfi-pve backplane repair pending tech visit - Postgres NFS retirement (block 4+5) pending 24h burn-in - Tomorrow 01:00: first scheduled restic runs for esh-vm-db + irv-ml1 - PBS Phase 7-8 (retire legacy vzdump) gated on 2026-04-29 ## First commands a fresh session should be comfortable running ```bash # Fleet validation (read-only, safe) scripts/refresh-server-info.sh --validate-only all # Specific host refresh scripts/refresh-server-info.sh # Look at current fleet state cat STATUS.md # Explore memory ls /home/lkraven/.claude/projects/-home-lkraven-development-eshpfi-management/memory/ cat /home/lkraven/.claude/projects/-home-lkraven-development-eshpfi-management/memory/MEMORY.md # Probe any host ssh # alias resolves via ~/.ssh/config ``` ## When in doubt - Read `CLAUDE.md` for conventions and the Servers table. - Read `STATUS.md` for what's in-flight right now. - Check `memory/MEMORY.md` for durable facts about the user's preferences + project history. - For recovery questions: `docs/runbooks/disaster-recovery.md`. - Ask. User is experienced — doesn't need hand-holding, but a clear question is faster than a wrong assumption.