Fleet backup architecture & freshness runbook

The map that was missing: what backs up what, where it lands, and how to check in 2 minutes whether backups are actually fresh. Companion to disaster-recovery.md (which covers recovery when a host/service is down). Read this one first when the question is "are we backed up?"

Why this exists: on 2026-06-20 diagnosing "are backups OK?" took a long exploration because the topology lived only in scattered memory. The ana-side restic layer had been failing silently for ~6.5 weeks (last good snapshot 2026-05-06) and nobody knew. This doc + a future freshness alert is the fix.

TL;DR — coverage matrix

Two independent layers. PBS = whole-VM images. restic = granular file+DB. A host is well-covered if it has either a current PBS image or a current restic snapshot; the danger zone is a host whose only layer has failed.

Host	Kind	PBS (VM image)	restic (file+DB)	Sole net?
ana-docker	VM (pfi-pve)	✅ `ana-pve`	✅ → rest-server-ana	no
ana-ml2	bare metal	❌ none (not a VM)	✅ → rest-server-ana	⚠️ restic is the ONLY net
irv-ml1	bare metal	❌ none (not a VM)	✅ → rest-server-nh3	⚠️ restic is the ONLY net
nh3-docker	VM (nh3-pve)	✅ `nh3-pve`	✅ → rest-server-nh3	no
esh-docker-vm	VM (esh-pve)	✅ `esh-pve`	✅ → rest-server-ana	no
esh-vm-db	VM (esh-pve-nas)	❌ none (esh-pve-nas not a PBS source)	✅ → rest-server-ana	⚠️ restic-only (a DB!)
vm-esh-nas	VM (esh-pve-nas)	❌ none (esh-pve-nas not a PBS source)	✅ → rest-server-ana	⚠️ restic-only
other pfi-pve / nh3-pve VMs/CTs	VM/CT	✅ respective ns	(PBS only)	no
SureFire `sfsrv-pve`	tenant VMs	✅ `sfsrv-pve` ns	(PBS only)	no

Bare-metal hosts have NO PBS coverage (PBS only backs up Proxmox guests). Their restic snapshot is the entire safety net — keep an eye on it. ana-ml2 → rest-server-ana; irv-ml1 → rest-server-nh3.

Layer 1 — PBS (whole-VM/CT images)

PBS-ANA (pbs-ana, 10.250.50.90) — fleet primary. Datastore is NFS-backed: 10.250.50.50:/mnt/backup/pbs-ana mounted at /mnt/pbs-datastore (~20 TB). Backs up Proxmox guests via vzdump, organised by namespace per source hypervisor:
- ana-pve — pfi-pve guests (ana-docker, pfi-postgres VM105, ana-nas CT109, webhost, filebot, pteradactyl, tacticalrmm, ana-wg, …)
- esh-pve — esh-pve guests
- nh3-pve — nh3-pve guests
- sfsrv-pve — SureFire tenant
- ⚠️ there is no esh-pve-nas namespace — guests on that hypervisor (vm-esh-nas, likely esh-vm-db) are not PBS-covered.
PBS-NH3 (pbs-nh3, 10.100.50.90) — DR mirror; syncs from PBS-ANA (datastore on nh3-nas).
Schedule: vzdump jobs defined in Proxmox (Datacenter → Backup), staggered through the early morning.

Layer 2 — restic (granular file + DB)

restic clients push to one of two rest-server endpoints (HTTP, basic auth, append-only, private repos). The split is by site:

rest-server	Endpoint	Backing store	Clients
rest-server-ana	`http://10.250.50.70:8000` (container `rest-server` on ana-docker)	`ana-nas:/mnt/backup/restic/repo/ana` (NFS bind → `/data`)	ana-docker, ana-ml2, esh-docker-vm, esh-vm-db, vm-esh-nas
rest-server-nh3	`http://10.100.50.50:8000` (on nh3-nas)	`nh3-nas:/volume1/Backup/restic/<client>`	irv-ml1, nh3-docker

Per-client repos live as subdirs of the rest-server data dir (.../repo/ana/<client>/ for the ana side); the shared .htpasswd for ana sits at .../repo/ana/.htpasswd.
Scheduler = resticprofile systemd timers on each client host, NOT Backrest:
- resticprofile-backup@profile-default.timer — daily 01:00 PDT
- resticprofile-check@profile-default.timer — weekly (Sun 05:00)
Backrest (container on ana-docker, UI) is only a repo viewer here — it has 0 plans. Do not assume "Backrest healthy" means "backups running." The timers are the source of truth.
⚠️ Failures are silent — a timer fires, restic errors against a down endpoint, and nothing alerts. (See Known gaps.)

The 2-minute freshness check

Run these any time you need to answer "are we backed up?"

# --- restic ANA side: newest snapshot per client (want: today/yesterday) ---
ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do
  echo -n "$c: "; ls -t /mnt/backup/restic/repo/ana/$c/snapshots/ 2>/dev/null | head -1 \
    | xargs -I{} stat -c "%y" /mnt/backup/restic/repo/ana/$c/snapshots/{} 2>/dev/null || echo MISSING
done'

# --- restic NH3 side ---
ssh nh3-nas 'for c in irv-ml1 nh3-docker; do
  echo -n "$c: "; ls -lt /volume1/Backup/restic/$c/snapshots/ 2>/dev/null | sed -n 2p
done'

# --- rest-server endpoints healthy? (401 = up & serving; Restarting = broken) ---
ssh infra-ops@ana-docker 'sudo docker ps --format "{{.Names}}\t{{.Status}}" | grep rest-server'
curl -s -o /dev/null -w 'rest-server-ana: %{http_code}\n' http://10.250.50.70:8000/
curl -s -o /dev/null -w 'rest-server-nh3: %{http_code}\n' http://10.100.50.50:8000/

# --- PBS: newest snapshot per guest, all namespaces ---
ssh pbs-ana 'for ns in /mnt/pbs-datastore/ns/*/; do nsn=$(basename "$ns")
  for d in vm ct; do for g in "$ns$d"/*/; do [ -d "$g" ] || continue
    echo "$nsn/$d/$(basename "$g") -> $(ls "$g" 2>/dev/null | grep ^20 | sort | tail -1)"
  done; done; done'

Force a backup now (don't wait for 01:00): on the client host, ssh infra-ops@<host> 'sudo systemctl start resticprofile-backup@profile-default.service' (it's an incremental against the existing repo — bounded even if stale).

Known failure mode: rest-server-ana crash-loop (the 2026-05-06 → 2026-06-20 outage)

Symptom: rest-server container on ana-docker stuck Restarting; logs show cannot load /data/.htpasswd: permission denied. All ana-side restic backups silently fail.

Root cause: ana-nas's NFS mount on ana-docker uses bare defaults in /etc/fstab (no _netdev, no retry). When the mount drops, mnt-backup.mount gets stuck failed, so /mnt/backup/restic/repo/ana resolves to an empty local ghost dir (no .htpasswd) and rest-server binds that. ana-nas itself is fine — the real repos are intact.

Recovery (needs root on ana-docker — use ssh infra-ops@ana-docker, which has NOPASSWD sudo; the default ssh ana-docker lands as lkraven without sudo):

ssh infra-ops@ana-docker '
  sudo mount -a                                   # re-attach the NFS (bypasses the failed unit)
  sudo systemctl reset-failed mnt-backup.mount    # clear the stuck unit state
  mount | grep /mnt/backup                         # confirm nfs4 attached
  sudo ls /mnt/backup/restic/repo/ana/.htpasswd    # real htpasswd now present
  cd /opt/docker/compose/rest-server-ana && sudo docker compose up -d --force-recreate
'                                                  # recreate so the bind re-resolves onto NFS
# verify: docker ps shows Up (healthy); curl :8000 -> 401; logs say "Loaded htpasswd file"

If the ghost dir blocks the mount, see disaster-recovery.md Tier-0 for the stop→umount→rm-ghost→remount→start variant.

Known gaps / TODO

Backup-freshness alerting — DONE (2026-06-20). scripts/check-backup-freshness.sh (the 2-min check, exit 1 on stale/down) + a daily systemd user timer on nh3-dev at 08:00 (scripts/install-backup-freshness-timer.sh) → backup-freshness-alert.sh posts an althing alert to infra-ops on any stale/down layer. Run the check by hand anytime. (Channel is althing for now — swap in email/ntfy if you want a louder one.)
fstab hardening — DONE (2026-06-20). ana-docker /mnt/backup → noauto,x-systemd.automount,x-systemd.mount-timeout=30 (autofs self-heals on a NAS blip instead of getting stuck failed; activates on next reboot). /etc/fstab.bak-pre-harden saved. /mnt/compose also hardened the same way and activated live (umount → mnt-compose.automount started → autofs verified remounting on access) — it binds no container, so it was safe to convert now; this also proved the autofs pattern works on ana-docker.
ana-ml2 has no PBS net (bare metal) — restic is its only layer; now healthy + alerted. Bulk /tank models are re-downloadable; bespoke quants/configs/scripts are the real loss-risk.
esh-pve-nas coverage — VERIFIED (2026-06-20): NOT PBS-covered. No esh-pve-nas namespace exists on PBS-ANA, so esh-vm-db (postgres+mongo) + vm-esh-nas are restic-only. For the DB VM, restic-with-dumps is the preferred method (vs a VM image) IF the resticprofile includes pg_dump/mongodump — confirm that. Optionally add esh-pve-nas as a PBS source. ESH is home-lab (no SLA).
Rotate rest-server repo passwords — the 5 per-repo basic-auth creds were exposed during the 2026-06-20 diagnosis. BELAYED — operator handling offline.

9.2 KiB Raw Blame History