vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
4979869731649332a63176bb9107dea1e8be0ce7
esh-pfi-infrastructure/docs/runbooks/disaster-recovery.md
T
vh 91bda3c480 fv-ml1: complete the cutover — rename, renumber, DNS, and the LiteLLM repoint 
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16,
and serving inference again. This lands the repo half of that.

Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq
reservation so the address the runbook, DNS and LiteLLM all assume is the
address it actually has. Its headscale node is renamed too.

The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the
reason this diff touches current-state files and not the record. Dated
persistent-memory entries, archival-memory and incident notes still say
ana-ml2 in 31 and 62 places respectively, because that is what the box was
when those things happened. Rewriting them would make the history lie.

LiteLLM was the load-bearing piece and needed more than the api_base sed the
runbook describes. Twenty api_base entries repointed, but a grep-and-verify
pass also caught a LIVE pass_through_endpoints target for the scalar-judge
reward route still on the old address -- an api_base-only substitution would
have left it dead. Four prose references describing current state were
repointed as well; one historical note recording where a hand-test was run
is deliberately left pointing at 10.250.50.54.

Two facts in the server tables were wrong and are corrected here. The site is
Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell
Max-Q, not two -- verified by nvidia-smi -L and independently by PCI
enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196,
which changes what fits on it.

DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback
approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all
three resolvers. The BMC record carries a warning that its 802.1q VLAN tag
must stay disabled -- it shipped tagging VLAN 250 into an untagged port,
which made it invisible to every network-side diagnostic and is the reason
it appeared dead through several cable changes.

Verified end to end: summarizer and sec both answer through the Anaheim
gateway across the mesh to FV seats on different ports.
2026-09-12 22:00:50 -07:00

12 KiB
Raw Blame History

Fleet disaster-recovery runbook

What breaks when a given host/service goes down, and how to recover. Ordered by blast-radius severity — read top-down.

First action for ANY multi-service outage: check ana-nas reachability. Many fleet-wide "things are broken" events trace back to ana-nas's NFS exports going stale. See ana-nas SPOF memory for details, or just ping 10.250.50.50 first.

Governing principle: backups must not risk production

Any backup strategy that could take down the host it's backing up is worse than the risk it mitigates. Specifically applies to:

  • NFS-serving hosts (CT 109 / ana-nas): scheduled vzdump of the container itself is prohibited. The rootfs is trivially rebuildable; the data lives on ospool bind-mounts captured via ZFS snapshot at the host level. If rootfs config is worth preserving, capture via rsync /etc /root to a known location — no snapshot, no freeze.
  • DB servers on NFS storage (VM 105 / pfi-postgres): prefer pg_dump + restic over full vzdump. fs-freeze on NFS-backed PGDATA is a failure mode waiting to happen.
  • Any host with heavy I/O concurrency during the backup window: schedule its own backup outside the fleet window OR use mode=stop (planned brief downtime beats random-freeze risk).

Incident reference: 2026-04-23 — CT 109, 112, 113 all went offline on pfi-pve during morning backup window. Backup suspected as trigger (full root cause TBD). See memory/incident_ana_nas_spof.md.

Tier 0 — Storage fabric (catastrophic blast radius)

ana-nas (CT 109 on pfi-pve, 10.250.50.50)

Blast radius:

  • pfi-postgres (VM 105) — PGDATA on /mnt/db — migrated to local disk 2026-04-23. vaultwarden, gitea, paperless-ng, zammad no longer cascade on ana-nas outage. Left in history for the recovery pre-migration.
  • ana-docker rest-server-ana — repo data on /mnt/backup → all ana-side restic clients fail (ana-docker, fv-ml1, esh-docker-vm, vm-esh-nas)
  • PBS-ANA datastore — NFS-backed on /mnt/backup/pbs-ana → fleet vzdumps fail, PBS-NH3 sync fails
  • ana-docker NFS mounts for /mnt/docker, /mnt/compose, /mnt/pve-VMStorage if used → various stack misbehavior

Recovery (see full procedure in memory/incident_ana_nas_spof.md):

  1. pct start 109 on pfi-pve if stopped; pct console 109 to see boot if stuck
  2. Once responsive, remount NFS on each consumer and restart services:
    • pfi-postgres: umount -lf /mnt/db; mount /mnt/db; systemctl restart postgresql
    • ana-docker: sudo mount -a; sudo docker restart rest-server — usually enough. mount -a re-attempts all fstab entries and bypasses the "failed" state that mnt-backup.mount gets stuck in (fstab uses bare defaults without auto-retry). If rest-server still errors with /data/.htpasswd: permission denied, you've got a ghost file on the local mount point — see incident memory for the clean-up procedure (stop rest-server, unmount NFS, rm the ghost, remount, restart).
    • PBS-ANA: umount -lf /mnt/pbs-datastore; mount /mnt/pbs-datastore; systemctl restart proxmox-backup proxmox-backup-proxy
  3. Re-run failed PBS vzdump job for affected VMs (Datacenter → Backup → Run Now)
  4. Restic clients recover automatically on next 01:00 schedule

Prevention:

  • Re-schedule ana-nas's own vzdump OUTSIDE the 03:00 fleet window (concurrent NFS load during fleet backup + self-backup is the suspected 2026-04-23 crash root cause)
  • Consider moving pfi-postgres PGDATA to local VM disk (reduces SPOF)

nh3-nas (Synology RS2418+, 10.100.50.50)

Blast radius:

  • nh3-docker, nh3-dev restic clients — writes to rest-server-nh3 fail
  • PBS-NH3 datastore — mirror sync can't write new chunks
  • ana-nas → nh3-nas rsync (04:00 daily) — fails

Recovery:

  1. Power-cycle via DSM web UI or physical button if unresponsive
  2. DSM boots 3-5 minutes; NFS exports auto-start
  3. Verify PBS-NH3 datastore recovered: 
    ssh pbs-nh3 'ls /mnt/pbs-datastore/.chunks | head; systemctl restart proxmox-backup proxmox-backup-proxy'
  4. ana-nas-side rsync retries at next 04:00; no manual action unless you want to trigger now

Notable: nh3-nas outage is LESS severe than ana-nas because:

  • No production DB depends on it
  • It's a mirror/DR tier, not primary
  • Ana-side backups keep running independently

Tier 1 — Hypervisors (regional blast radius)

pfi-pve (Proxmox VE, 10.250.250.31)

Blast radius — everything hosted on it:

  • ana-docker (VM), ana-nas (CT 109), pfi-postgres (VM 105), pfi-ana-webhost, ana-filebot, pfi-pteradactyl, pfi-tacticalrmm, ana-wg, PBS-ANA (VM), + others

Essentially all Anaheim primary services go offline. Because ana-nas lives here too, Tier-0 cascade applies simultaneously.

Recovery:

  1. Is the Proxmox host itself up? iDRAC/IPMI console for hardware diagnosis.
  2. If the host is up but VMs/CTs aren't starting: systemctl status pve-cluster qemu-server pve-container on the host.
  3. Start VMs/CTs in dependency order:
    • First: CT 109 (ana-nas) — everything else on the host that uses NFS depends on it
    • Second: PBS-ANA VM (resume backup target)
    • Third: VM 105 (pfi-postgres) — many apps wait on this
    • Fourth: ana-docker, then other VMs
  4. After all VMs/CTs up, re-apply the Tier-0 ana-nas recovery steps for any consumers with stale NFS mounts.

Prevention: PBS-ANA → PBS-NH3 nightly sync means VM image backups are recoverable at NH3 if pfi-pve is unrecoverable; can restore key VMs to nh3-pve and cut over (cold DR).

nh3-pve (10.100.250.60)

Blast radius:

  • nh3-docker (VM) — NH3 site's Docker stacks
  • PBS-NH3 (VM) — DR mirror target
  • Other NH3 VMs

Recovery: same pattern as pfi-pve. PBS-NH3 being down breaks DR replication but doesn't affect primary backups (ANA side is authoritative).

esh-pve + esh-pve-nas (10.0.250.35 + 10.0.50.55)

Blast radius:

  • All ESH home-lab workloads: esh-docker-vm, vm-esh-nas, media services, home automation stacks

Recovery: restore VMs from PBS-ANA (cross-site pull). Not production-critical; no SLA.

sfsrv-ana (Dell R630, 10.250.250.115, iDRAC 10.250.250.110)

Blast radius: SureFire tenant workloads.

  • PBS-ANA namespace sfsrv-pve has daily snapshots
  • Coordinate with tenant before any recovery action per hosting agreement

Recovery: restore from PBS-ANA sfsrv-pve namespace. Tenant may have app-level recovery procedures that take precedence.

Tier 2 — Docker hosts (per-stack blast radius)

ana-docker (VM 10.250.50.70)

Blast radius — services hosted on it:

  • rest-server-ana (fleet restic target; depends on ana-nas NFS)
  • Backrest UI, Beszel hub, Dozzle hub, Traefik (ANA), CrowdSec, Gitea, Vaultwarden, Seafile, Synapse, OpenWebUI, SearXNG, IT Tools, others

Recovery:

  1. VM restart on pfi-pve: qm stop <vmid>; qm start <vmid>
  2. After boot, stacks auto-start via docker compose up -d (compose files at /opt/docker/compose/*/)
  3. Check each stack with docker ps / dockge
  4. If /mnt/backup (NFS) is stale after pfi-pve reboot, force remount

nh3-docker (VM 10.100.50.40)

Blast radius:

  • rest-server-nh3 (if migrated there — currently on nh3-nas Synology directly)
  • NH3-only stacks

Recovery: VM restart on nh3-pve. Cross-site Beszel/Dozzle agents on ana-docker keep reporting independently.

esh-docker-vm + vm-esh-nas

Blast radius: ESH home-lab Docker workloads. Paperless, HA, CWA, pgadmin (esh-docker-vm); dockge, filezilla, agents (vm-esh-nas).

Recovery: VM restart on respective ESH hypervisors. Not production-critical.

Tier 3 — Workload VMs/LXCs

pfi-postgres (VM 105 on pfi-pve)

Blast radius: vaultwarden, gitea, paperless-ng databases.

Recovery: usually it's actually an ana-nas issue (see Tier 0). If postgres itself is the problem:

  1. Check logs: sudo tail -200 /var/lib/postgresql/*/main/log/postgresql-*.log
  2. If clean: sudo systemctl restart postgresql
  3. If WAL corruption: restore from PBS vzdump (most recent snapshot)

ana-wg (CT 113 on pfi-pve) — WireGuard VPN

Blast radius: remote-access VPN down; remote admin sessions drop but on-prem ops continue.

Recovery: pct start 113 on pfi-pve. Restart wg-quick service if needed.

pfi-pteradactyl (VM 107 on pfi-pve) — game panel

Blast radius: hosted game servers offline.

Recovery: VM restart. Not operationally critical.

pfi-tacticalrmm (VM 111 on pfi-pve) — RMM tooling

Blast radius: RMM agent dashboards; agents continue running on endpoints but can't phone home.

Recovery: VM restart; TacticalRMM services auto-start.

ana-filebot (CT 112 on pfi-pve) — file automation

Blast radius: scheduled file operations (rename, organize); low-criticality.

Recovery: pct start 112.

pfi-ana-webhost (VM on pfi-pve) — web workload

Blast radius: hosted website(s) offline.

Recovery: VM restart; web server auto-start.

Tier 4 — Specialty workloads

fv-ml1 (bare metal Supermicro, 10.251.50.54, BMC 10.251.250.50)

Blast radius: AI inference services (llama-swap, vllm). Consumer-facing chat/embedding/reward-scoring endpoints fail.

Recovery:

  1. Check OS via SSH. If unresponsive, BMC console at https://10.251.250.50.
  2. If hardware issue: BMC logs, power cycle via IPMI, check GPU health (nvidia-smi).
  3. Docker stacks auto-start via compose restart: unless-stopped.

Notable: no Proxmox hypervisor involved — recovery is pure bare-metal. File-level restic runs from inside the host. No vzdump coverage (it's not a VM). Backups via restic to rest-server-ana.

SureFire hosts (sfsrv-ana, sf-ana-container, sf-r630)

Blast radius: tenant workloads (scoped to SureFire client).

Recovery: coordinate with tenant. PFI responsibility is hardware + OS layer; application recovery may need tenant input. See servers/sfsrv-ana/README.md for hosting agreement scope.

Tier 5 — Networking

FortiGate 60F (ESH gateway, 10.0.250.1)

Blast radius: ESH site WAN + inter-site VPN to ANA/NH3.

Recovery: physical console access; restore config from FortiManager if needed.

UniFi controllers (ESH 10.0.0.1, PFI 10.100.0.1)

Blast radius: UniFi AP/switch management (existing config persists on devices; only changes need the controller).

Recovery: device power cycle; controller reboot. Not critical for ongoing operations.

Cross-cutting: what to check FIRST for ambiguous outages

When symptoms are vague ("lots of things are down"), run this triage in order:

# 1. Is the Anaheim NAS alive?
ping -c 2 10.250.50.50

# 2. Are the hypervisors alive?
for h in 10.250.250.31 10.100.250.60 10.0.250.35 10.0.50.55 10.250.250.115; do
  ping -c 1 -W 2 $h >/dev/null && echo "$h OK" || echo "$h DOWN"
done

# 3. WAN reachability between sites?
#    (from nh3-dev to ana-side IP; from ana-side to nh3-side IP)
ping -c 2 10.250.50.70  # ana-docker from NH3

The first DOWN in the hypervisor list narrows blast radius to that site / that hypervisor's guests.

What this runbook does NOT cover

  • Application-level restores from PBS / restic snapshots — that's per-service and lives in respective stack READMEs
  • Full site-failover (e.g., "move PFI fleet to NH3 entirely") — requires Phase-9-style DR plan, not built
  • Data-level integrity recovery (DB corruption, ZFS pool damage) — beyond this runbook's scope

Maintenance note: update this file when:

  • New host gets registered (add to applicable tier)
  • New SPOF discovered (add blast-radius note)
  • A recovery procedure changes in practice
Reference in New Issue View Git Blame Copy Permalink