vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
graphify-tooling
esh-pfi-infrastructure/docs/orientation.md
T
vh ad2b0e97aa docs/runbooks/nh3-prune-ritual: write up the quarterly NH3 prune ceremony 
New runbook captures the three-phase process:

  Phase 1 — Drop --append-only via DSM Container Manager web UI
  Phase 2 — sudo resticprofile forget --prune --verbose on each of
            nh3-docker, nh3-dev, irv-ml1 (interactive sudo per host)
  Phase 3 — Restore --append-only via DSM

Why each phase looks the way it does, what to expect (largely no-op
runs for the first 6 months while no snapshots have aged out of the
keep window), how to verify each phase non-destructively (curl 401
on the rest-server root proves the container's up + serving), what
to do if Phase 2 fails with `repository is configured as append-only`
(skipped Phase 1 / DSM didn't apply), and the path to future
automation (find docker bin path on DSM, NOPASSWD-lock syncuser to
the specific recreate command).

Includes a "last run history" table seeded with today's first
post-pipeline run (no-op, irv-ml1 only had 3 snapshots due to the
04-25→27 CUDA stall).

Cross-referenced from docs/README.md (runbook tree), docs/
orientation.md (where-to-look table), and STATUS.md item 9 (which
now points at the runbook + records the next-round date 2026-07-27).
2026-04-27 20:54:08 -07:00

12 KiB
Raw Permalink Blame History

Fleet orientation — start here

Read this first if you're a fresh session joining this project. It summarizes the fleet, the conventions, the known gotchas, and points at the deeper docs / memory files that cover each area in detail.

What this repo is

A reference + management workspace for PFI infrastructure — a small multi-site fleet of Proxmox hypervisors, Linux VMs/LXCs, a bare-metal GPU host, and two NAS boxes. Authoritative copies of Docker compose files, backup configs, and homepage dashboard config live on the hosts themselves; this repo mirrors them for version control, planning, and cross-host orchestration.

Primary file you'll read automatically on session start:

  • CLAUDE.md — conventions, servers table, scripts tour, placement rules.
  • ~/.claude/projects/.../memory/MEMORY.md + linked memory files — durable cross-session facts.

This file (docs/orientation.md) is the narrative overview that ties them together.

Sites + hosts at a glance

Three physical locations plus one WireGuard-tunneled satellite:

Site Subnet Physical boxes Notes
Anaheim (ANA) 10.250.0.0/16 Dell R750xs (pfi-pve hypervisor, iDRAC 10.250.250.30), Supermicro (ana-ml2 bare-metal GPU, BMC 10.250.250.50), Dell R630 (sfsrv-ana SureFire tenant, iDRAC 10.250.250.110) Primary production site
NH3 10.100.0.0/16 nh3-pve hypervisor, Synology RS2418+ (nh3-nas, 42 TB) Secondary + DR target
ESH home lab esteban.net, 10.0.50.0/24 esh-pve, esh-pve-nas (split-role hypervisors), esh-vm-db (DB VM) Non-production, shares backup tooling
Irvine (IRV) 10.100.79.0/24 (WG tunnel only) irv-ml1 bare-metal GPU box Reachable ONLY via WireGuard tunnel from NH3

Full host list with roles: see the Servers table in CLAUDE.md.

Backup architecture (two layers)

This is the most fleet-impacting system. Every host gets backed up via one or both layers.

Layer 1 — PBS (VM-image, block-level dedupe)

  • Primary: pbs-ana (VM on pfi-pve, NFS datastore on ana-nas).
  • DR mirror: pbs-nh3 (VM on nh3-pve, NFS datastore on nh3-nas Synology).
  • Sync: pbs-nh3 pulls from pbs-ana at 06:00 daily. One-way.
  • Namespaces: per-hypervisor (pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana) because VMIDs are NOT globally unique.
  • Scope: all 5 hypervisors back up to pbs-ana nightly.

Deployment runbook: docs/runbooks/pbs-deployment.md (9 phases, captured during the 2026-04-22 deployment).

Layer 2 — restic (file-level, encrypted, DB-consistent)

  • Target A: rest-server-ana container on ana-docker. Data dir NFS-mounted from ana-nas. Used by: ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas, esh-vm-db.
  • Target B: rest-server-nh3 container on nh3-nas (Synology). Used by: nh3-docker, nh3-dev, irv-ml1.
  • Mode: both run --append-only --private-repos — clients can't delete their own history, and each client's URL path must match their HTTP basic-auth user.
  • Scope: 8 hosts have resticprofile + systemd timers firing at 01:00 daily.
  • DB-consistent dumps: hosts with databases have a pre-backup.sh that runs pg_dump/mongodump/sqlite3 .backup into /var/lib/restic/stage/ before restic sweeps.

Cross-site restic mirror (rsync)

  • ana-nas → nh3-nas daily at 04:00 (runs on ana-nas as lkraven).
  • nh3-nas → ana-nas daily at 05:00 (runs on nh3-nas as root — rest-server-nh3's container writes files mode 400 owned by admin, only root can read them on the Synology side).

Tracked at configs/rsync/{ana-nas-to-nh3,nh3-nas-to-ana}/.

Governing principles (durable rules)

Saved in feedback-type memory files; apply by default.

  1. Backups must not risk production. If a backup process could crash the host it's backing up, change the strategy or skip the backup. Reference: CT 109 (ana-nas) self-backup crash of 2026-04-23. Memory: feedback_backups_must_not_risk_production.md.
  2. DB data lives on local disk, not NFS. Decision 2026-04-23 after pfi-postgres migration off /mnt/db. New DB deployments default to local storage; shared NFS for DB data is an anti-pattern in this environment. Memory: project_db_migrate_off_nfs.md.
  3. Use ssh -t when the remote runs sudo. Read-only probes and piped scripts can stay plain ssh. Memory: feedback_ssh_sudo.md.
  4. Handle git commands autonomously. Stage + commit at natural boundaries without asking; verify risky changes first; never force-push or push without explicit approval. Memory: feedback_git_autonomous.md.
  5. No Claude attribution in commits. No Co-Authored-By, "Generated with...", robot emojis, etc. Memory: feedback_git_commits.md.

Known gotchas (save yourself time)

These caught us once; don't let them catch you twice.

NFS / storage

  • ana-nas is CT 109 (LXC), not VM 100. Persistent confusion historically — the Proxmox VM name was PFI-ANA-TRUENAS for ages but the host was reprovisioned to Debian, and it's an LXC container not a VM. Memory: storage_ana_nas.md.
  • ana-nas is a SPOF for rest-server-ana + PBS-ANA datastore. When multiple fleet services fail together, check ping 10.250.50.50 FIRST. Memory: incident_ana_nas_spof.md. Recovery runbook: docs/runbooks/disaster-recovery.md.
  • NFS root_squash bites in two directions:
    • Sudo-root on the client writing to an NFS file gets mapped to nobody. Doesn't matter who owns the file.
    • rsync-as-root of NFS-owned data fails the same way. Workaround for postgres-on-NFS migration: run rsync as the postgres user, whose uid has legitimate NFS access.
  • ZFS with casesensitivity=insensitive + NFSv4 = EACCES on writes. PBS-ANA's datastore uses NFSv3 mount to work around this. Don't "upgrade" to NFSv4 without fixing the dataset first.
  • Synology .htpasswd edits must be done as the file owner (uid 1000 = lkraven for rest-server-ana), NOT via sudo. sudo-root gets squashed to nobody and can't read/write the file.
  • Synology Advanced Permissions (syno_acl) imposes NFSv4 ACLs that shadow POSIX perms. "Enable Advanced Permissions = off" doesn't reset it once set. Fix for PBS data: chmod 777 on the share to flatten to "Linux mode" (confirm with synoacltool -get).
  • DSM Container Manager doesn't expose sudo docker in root's PATH. Use DSM web UI to restart containers, or find the actual binary via find.
  • scp to DSM requires -O flag (legacy scp protocol) — default sftp subsystem is disabled.

systemd

  • DSM ships a pre-v247 systemd. Refuses Restart=on-failure on Type=oneshot, doesn't recognize StartLimitBurst/IntervalSec, systemctl --now flag absent. Write unit files with plain oneshot semantics for DSM; the daily timer retry compensates for missed runs.
  • fstab with bare defaults doesn't auto-retry a failed NFS mount. If the NFS server wasn't up at boot, mnt-*.mount sits in failed state until manual systemctl reset-failed + systemctl start (or simpler: mount -a).
  • Ghost files on NFS mount points. When NFS unmounts while writes are still landing at the mount point path, the writes go to the underlying local directory. When NFS remounts, the local ghost is shadowed. When NFS later unmounts again, the ghost reappears and breaks whatever was reading that path. See incident_ana_nas_spof.md for the exact case on ana-docker.

Naming / identifiers

  • Hardware vs OS distinction for servers with iDRAC/BMC:
    • pfi-pve (Dell R750xs) ↔ PFI-r750xs-iDRAC at 10.250.250.30
    • sfsrv-ana (Dell R630) ↔ SF-R630-iDRAC at 10.250.250.110
    • ana-ml2 (Supermicro) ↔ PFI-ANA-ML2 BMC at 10.250.250.50 Each servers/<name>/ dir documents the OS side; hardware/iDRAC entries exist in homepage but are separate logical identities.
  • irv-ml1 was ana-ml1 before a physical move; OS hostname still says ana-ml1 pending an explicit rename. Doesn't affect services.

Workflow

  • Terminal word-wrap breaks long pasted commands. Never embed a 580-char pubkey or long URL inline in a multi-line shell command — terminal wraps the line, shell captures literal newlines, and the command fails in bizarre ways. Use cat from a file, printf, or a tempfile + scp.
  • zsh interactive mode doesn't parse # as comment by default. Paste multi-line blocks with comment lines and zsh tries to execute # as a command. Either tell user to setopt interactivecomments or keep blocks comment-free.
  • ssh -t with piped stdin fails — -t allocates a tty, stdin from a pipe can't be a tty. Options: don't use -t (give up interactive sudo prompt), or scp the file first then ssh -t for the sudo bits.

Where to look for what

Need Go here
Is host X alive? servers/<host>/system-details.txt (snapshot). Refresh with scripts/refresh-server-info.sh <host>.
What does host X run? servers/<host>/README.md
What's the backup strategy for host X? configs/restic/<host>/README.md + profiles.yaml
How do I recover from host X going down? docs/runbooks/disaster-recovery.md
How was PBS deployed? docs/runbooks/pbs-deployment.md
How do I run the quarterly NH3 restic prune? docs/runbooks/nh3-prune-ritual.md (or just scripts/restic-prune.sh nh3)
What's currently open / in-flight? STATUS.md
What do I need to know that isn't in current code? MEMORY.md + the .md files it links
Why did we do X? Check memory files + STATUS.md session milestones at the bottom

Inventory + automation scripts

Under scripts/:

  • server_inspect.sh — pipes over SSH to generate system-details.txt (Docker hosts mostly; works on any Debian-ish Linux).
  • refresh-server-info.sh <host> or all — fleet-wide refresh driver. --validate-only for dry-run.
  • proxmox_inspect.sh + refresh-proxmox-info.sh — Proxmox-flavored variant that captures qm list, pct list, zpool, storage.
  • discover-fortigate.sh, discover-unifi.sh, discover-gaps.sh — network-level inventory diff against servers/*/.
  • sync-stacks.sh / deploy-stack.sh — compose mirror pull/push.
  • add-host.sh <name> <user@ip> — canonical host registration.

Current state (2026-04-23 snapshot)

  • Host inventory: ~22 tracked hosts under servers/ across ANA, NH3, ESH, IRV. Every host has SSH config alias for convenience.
  • Backup coverage: 8 restic clients, 5 hypervisors on PBS with DR mirror, cross-site rsync both directions operational.
  • Recent migrations: pfi-postgres → local disk (removes ana-nas cascade); offen sidecars retired on esh-docker-vm (16 GB reclaimed); irv-ml1 + esh-vm-db fully onboarded.
  • Recent secret rotations: 6 credentials — all gitea/vaultwarden/ paperless DB passwords + 3 rest-server htpasswd + repo passphrases.

Active watchlist (see STATUS.md for full detail):

  • pfi-pve backplane repair pending tech visit
  • Postgres NFS retirement (block 4+5) pending 24h burn-in
  • Tomorrow 01:00: first scheduled restic runs for esh-vm-db + irv-ml1
  • PBS Phase 7-8 (retire legacy vzdump) gated on 2026-04-29

First commands a fresh session should be comfortable running

# Fleet validation (read-only, safe)
scripts/refresh-server-info.sh --validate-only all

# Specific host refresh
scripts/refresh-server-info.sh <hostname>

# Look at current fleet state
cat STATUS.md

# Explore memory
ls /home/lkraven/.claude/projects/-home-lkraven-development-eshpfi-management/memory/
cat /home/lkraven/.claude/projects/-home-lkraven-development-eshpfi-management/memory/MEMORY.md

# Probe any host
ssh <host>  # alias resolves via ~/.ssh/config

When in doubt

  • Read CLAUDE.md for conventions and the Servers table.
  • Read STATUS.md for what's in-flight right now.
  • Check memory/MEMORY.md for durable facts about the user's preferences + project history.
  • For recovery questions: docs/runbooks/disaster-recovery.md.
  • Ask. User is experienced — doesn't need hand-holding, but a clear question is faster than a wrong assumption.
Reference in New Issue View Git Blame Copy Permalink