vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7e7130172e81c8d96ad8b8af1b6cf8cfad729cfc
esh-pfi-infrastructure/docs/runbooks/disaster-recovery.md
T
vh 7e7130172e vllm: rename stack from vllm-qwen3 → vllm + add Skywork reward classifier 
Two related changes shipped together. The stack rename is independent
but adding `vllm-reward` to the existing `vllm-qwen3` would have made
that name actively misleading.

**Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo
references (README.md root, servers/ana-ml2/, stacks/llama-swap/,
configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two
intentional history mentions retained (servers/ana-ml2 + stacks/vllm
README).

**Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ
on port 8003. The AWQ output is a locally-quantized model (not from HF),
so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the
shared HF cache. Model config.json declares LlamaForSequenceClassification
which vLLM's pooling runner picks up automatically — produces a single
reward score per input via /classify.

**Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1
deprecated --task in favor of --runner pooling (model architecture in
config.json drives the classification head). Compose uses --runner
pooling with a comment explaining the substitution.

**GPU memory:** no rebalance needed — production had already tuned
EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now
matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB
headroom on the 48 GB Ada.

**Server-side:** brought existing vllm-qwen3 down, mv'd
/opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended
REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed
new compose via scripts/deploy-stack.sh, brought all 3 services up.

**Smoke tests:**
- /health on 8001/8002/8003 → 200
- /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
  with max_model_len 16384
- /classify with a sample conversation → returns LABEL_0 with prob 0.9999
  (single-output regression-style reward score, expected shape for a
  reward model)
2026-05-13 22:00:26 -07:00

12 KiB
Raw Blame History

Fleet disaster-recovery runbook

What breaks when a given host/service goes down, and how to recover. Ordered by blast-radius severity — read top-down.

First action for ANY multi-service outage: check ana-nas reachability. Many fleet-wide "things are broken" events trace back to ana-nas's NFS exports going stale. See ana-nas SPOF memory for details, or just ping 10.250.50.50 first.

Governing principle: backups must not risk production

Any backup strategy that could take down the host it's backing up is worse than the risk it mitigates. Specifically applies to:

  • NFS-serving hosts (CT 109 / ana-nas): scheduled vzdump of the container itself is prohibited. The rootfs is trivially rebuildable; the data lives on ospool bind-mounts captured via ZFS snapshot at the host level. If rootfs config is worth preserving, capture via rsync /etc /root to a known location — no snapshot, no freeze.
  • DB servers on NFS storage (VM 105 / pfi-postgres): prefer pg_dump + restic over full vzdump. fs-freeze on NFS-backed PGDATA is a failure mode waiting to happen.
  • Any host with heavy I/O concurrency during the backup window: schedule its own backup outside the fleet window OR use mode=stop (planned brief downtime beats random-freeze risk).

Incident reference: 2026-04-23 — CT 109, 112, 113 all went offline on pfi-pve during morning backup window. Backup suspected as trigger (full root cause TBD). See memory/incident_ana_nas_spof.md.

Tier 0 — Storage fabric (catastrophic blast radius)

ana-nas (CT 109 on pfi-pve, 10.250.50.50)

Blast radius:

  • pfi-postgres (VM 105) — PGDATA on /mnt/db — migrated to local disk 2026-04-23. vaultwarden, gitea, paperless-ng, zammad no longer cascade on ana-nas outage. Left in history for the recovery pre-migration.
  • ana-docker rest-server-ana — repo data on /mnt/backup → all ana-side restic clients fail (ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas)
  • PBS-ANA datastore — NFS-backed on /mnt/backup/pbs-ana → fleet vzdumps fail, PBS-NH3 sync fails
  • ana-docker NFS mounts for /mnt/docker, /mnt/compose, /mnt/pve-VMStorage if used → various stack misbehavior

Recovery (see full procedure in memory/incident_ana_nas_spof.md):

  1. pct start 109 on pfi-pve if stopped; pct console 109 to see boot if stuck
  2. Once responsive, remount NFS on each consumer and restart services:
    • pfi-postgres: umount -lf /mnt/db; mount /mnt/db; systemctl restart postgresql
    • ana-docker: sudo mount -a; sudo docker restart rest-server — usually enough. mount -a re-attempts all fstab entries and bypasses the "failed" state that mnt-backup.mount gets stuck in (fstab uses bare defaults without auto-retry). If rest-server still errors with /data/.htpasswd: permission denied, you've got a ghost file on the local mount point — see incident memory for the clean-up procedure (stop rest-server, unmount NFS, rm the ghost, remount, restart).
    • PBS-ANA: umount -lf /mnt/pbs-datastore; mount /mnt/pbs-datastore; systemctl restart proxmox-backup proxmox-backup-proxy
  3. Re-run failed PBS vzdump job for affected VMs (Datacenter → Backup → Run Now)
  4. Restic clients recover automatically on next 01:00 schedule

Prevention:

  • Re-schedule ana-nas's own vzdump OUTSIDE the 03:00 fleet window (concurrent NFS load during fleet backup + self-backup is the suspected 2026-04-23 crash root cause)
  • Consider moving pfi-postgres PGDATA to local VM disk (reduces SPOF)

nh3-nas (Synology RS2418+, 10.100.50.50)

Blast radius:

  • nh3-docker, nh3-dev restic clients — writes to rest-server-nh3 fail
  • PBS-NH3 datastore — mirror sync can't write new chunks
  • ana-nas → nh3-nas rsync (04:00 daily) — fails

Recovery:

  1. Power-cycle via DSM web UI or physical button if unresponsive
  2. DSM boots 3-5 minutes; NFS exports auto-start
  3. Verify PBS-NH3 datastore recovered: 
    ssh pbs-nh3 'ls /mnt/pbs-datastore/.chunks | head; systemctl restart proxmox-backup proxmox-backup-proxy'
  4. ana-nas-side rsync retries at next 04:00; no manual action unless you want to trigger now

Notable: nh3-nas outage is LESS severe than ana-nas because:

  • No production DB depends on it
  • It's a mirror/DR tier, not primary
  • Ana-side backups keep running independently

Tier 1 — Hypervisors (regional blast radius)

pfi-pve (Proxmox VE, 10.250.250.31)

Blast radius — everything hosted on it:

  • ana-docker (VM), ana-nas (CT 109), pfi-postgres (VM 105), pfi-ana-webhost, ana-filebot, pfi-pteradactyl, pfi-tacticalrmm, ana-wg, PBS-ANA (VM), + others

Essentially all Anaheim primary services go offline. Because ana-nas lives here too, Tier-0 cascade applies simultaneously.

Recovery:

  1. Is the Proxmox host itself up? iDRAC/IPMI console for hardware diagnosis.
  2. If the host is up but VMs/CTs aren't starting: systemctl status pve-cluster qemu-server pve-container on the host.
  3. Start VMs/CTs in dependency order:
    • First: CT 109 (ana-nas) — everything else on the host that uses NFS depends on it
    • Second: PBS-ANA VM (resume backup target)
    • Third: VM 105 (pfi-postgres) — many apps wait on this
    • Fourth: ana-docker, then other VMs
  4. After all VMs/CTs up, re-apply the Tier-0 ana-nas recovery steps for any consumers with stale NFS mounts.

Prevention: PBS-ANA → PBS-NH3 nightly sync means VM image backups are recoverable at NH3 if pfi-pve is unrecoverable; can restore key VMs to nh3-pve and cut over (cold DR).

nh3-pve (10.100.250.60)

Blast radius:

  • nh3-docker (VM) — NH3 site's Docker stacks
  • PBS-NH3 (VM) — DR mirror target
  • Other NH3 VMs

Recovery: same pattern as pfi-pve. PBS-NH3 being down breaks DR replication but doesn't affect primary backups (ANA side is authoritative).

esh-pve + esh-pve-nas (10.0.250.35 + 10.0.50.55)

Blast radius:

  • All ESH home-lab workloads: esh-docker-vm, vm-esh-nas, media services, home automation stacks

Recovery: restore VMs from PBS-ANA (cross-site pull). Not production-critical; no SLA.

sfsrv-ana (Dell R630, 10.250.250.115, iDRAC 10.250.250.110)

Blast radius: SureFire tenant workloads.

  • PBS-ANA namespace sfsrv-pve has daily snapshots
  • Coordinate with tenant before any recovery action per hosting agreement

Recovery: restore from PBS-ANA sfsrv-pve namespace. Tenant may have app-level recovery procedures that take precedence.

Tier 2 — Docker hosts (per-stack blast radius)

ana-docker (VM 10.250.50.70)

Blast radius — services hosted on it:

  • rest-server-ana (fleet restic target; depends on ana-nas NFS)
  • Backrest UI, Beszel hub, Dozzle hub, Traefik (ANA), CrowdSec, Gitea, Vaultwarden, Seafile, Synapse, OpenWebUI, SearXNG, IT Tools, others

Recovery:

  1. VM restart on pfi-pve: qm stop <vmid>; qm start <vmid>
  2. After boot, stacks auto-start via docker compose up -d (compose files at /opt/docker/compose/*/)
  3. Check each stack with docker ps / dockge
  4. If /mnt/backup (NFS) is stale after pfi-pve reboot, force remount

nh3-docker (VM 10.100.50.40)

Blast radius:

  • rest-server-nh3 (if migrated there — currently on nh3-nas Synology directly)
  • NH3-only stacks

Recovery: VM restart on nh3-pve. Cross-site Beszel/Dozzle agents on ana-docker keep reporting independently.

esh-docker-vm + vm-esh-nas

Blast radius: ESH home-lab Docker workloads. Paperless, HA, CWA, pgadmin (esh-docker-vm); dockge, filezilla, agents (vm-esh-nas).

Recovery: VM restart on respective ESH hypervisors. Not production-critical.

Tier 3 — Workload VMs/LXCs

pfi-postgres (VM 105 on pfi-pve)

Blast radius: vaultwarden, gitea, paperless-ng databases.

Recovery: usually it's actually an ana-nas issue (see Tier 0). If postgres itself is the problem:

  1. Check logs: sudo tail -200 /var/lib/postgresql/*/main/log/postgresql-*.log
  2. If clean: sudo systemctl restart postgresql
  3. If WAL corruption: restore from PBS vzdump (most recent snapshot)

ana-wg (CT 113 on pfi-pve) — WireGuard VPN

Blast radius: remote-access VPN down; remote admin sessions drop but on-prem ops continue.

Recovery: pct start 113 on pfi-pve. Restart wg-quick service if needed.

pfi-pteradactyl (VM 107 on pfi-pve) — game panel

Blast radius: hosted game servers offline.

Recovery: VM restart. Not operationally critical.

pfi-tacticalrmm (VM 111 on pfi-pve) — RMM tooling

Blast radius: RMM agent dashboards; agents continue running on endpoints but can't phone home.

Recovery: VM restart; TacticalRMM services auto-start.

ana-filebot (CT 112 on pfi-pve) — file automation

Blast radius: scheduled file operations (rename, organize); low-criticality.

Recovery: pct start 112.

pfi-ana-webhost (VM on pfi-pve) — web workload

Blast radius: hosted website(s) offline.

Recovery: VM restart; web server auto-start.

Tier 4 — Specialty workloads

ana-ml2 (bare metal Supermicro, 10.250.50.54, BMC 10.250.250.50)

Blast radius: AI inference services (llama-swap, vllm). Consumer-facing chat/embedding/reward-scoring endpoints fail.

Recovery:

  1. Check OS via SSH. If unresponsive, BMC console at https://10.250.250.50.
  2. If hardware issue: BMC logs, power cycle via IPMI, check GPU health (nvidia-smi).
  3. Docker stacks auto-start via compose restart: unless-stopped.

Notable: no Proxmox hypervisor involved — recovery is pure bare-metal. File-level restic runs from inside the host. No vzdump coverage (it's not a VM). Backups via restic to rest-server-ana.

SureFire hosts (sfsrv-ana, sf-ana-container, sf-r630)

Blast radius: tenant workloads (scoped to SureFire client).

Recovery: coordinate with tenant. PFI responsibility is hardware + OS layer; application recovery may need tenant input. See servers/sfsrv-ana/README.md for hosting agreement scope.

Tier 5 — Networking

FortiGate 60F (ESH gateway, 10.0.250.1)

Blast radius: ESH site WAN + inter-site VPN to ANA/NH3.

Recovery: physical console access; restore config from FortiManager if needed.

UniFi controllers (ESH 10.0.0.1, PFI 10.100.0.1)

Blast radius: UniFi AP/switch management (existing config persists on devices; only changes need the controller).

Recovery: device power cycle; controller reboot. Not critical for ongoing operations.

Cross-cutting: what to check FIRST for ambiguous outages

When symptoms are vague ("lots of things are down"), run this triage in order:

# 1. Is the Anaheim NAS alive?
ping -c 2 10.250.50.50

# 2. Are the hypervisors alive?
for h in 10.250.250.31 10.100.250.60 10.0.250.35 10.0.50.55 10.250.250.115; do
  ping -c 1 -W 2 $h >/dev/null && echo "$h OK" || echo "$h DOWN"
done

# 3. WAN reachability between sites?
#    (from nh3-dev to ana-side IP; from ana-side to nh3-side IP)
ping -c 2 10.250.50.70  # ana-docker from NH3

The first DOWN in the hypervisor list narrows blast radius to that site / that hypervisor's guests.

What this runbook does NOT cover

  • Application-level restores from PBS / restic snapshots — that's per-service and lives in respective stack READMEs
  • Full site-failover (e.g., "move PFI fleet to NH3 entirely") — requires Phase-9-style DR plan, not built
  • Data-level integrity recovery (DB corruption, ZFS pool damage) — beyond this runbook's scope

Maintenance note: update this file when:

  • New host gets registered (add to applicable tier)
  • New SPOF discovered (add blast-radius note)
  • A recovery procedure changes in practice
Reference in New Issue View Git Blame Copy Permalink