esh-pfi-infrastructure/docs/runbooks/backups.md

Fleet backup architecture & freshness runbook

The map that was missing: what backs up what, where it lands, and how to check in 2 minutes whether backups are actually fresh. Companion to disaster-recovery.md (which covers recovery when a host/service is down). Read this one first when the question is "are we backed up?"

Why this exists: on 2026-06-20 diagnosing "are backups OK?" took a long exploration because the topology lived only in scattered memory. The ana-side restic layer had been failing silently for ~6.5 weeks (last good snapshot 2026-05-06) and nobody knew. This doc + a future freshness alert is the fix.

---

TL;DR — coverage matrix

Two independent layers. PBS = whole-VM images. restic = granular file+DB. A host is well-covered if it has either a current PBS image or a current restic snapshot; the danger zone is a host whose only layer has failed.

Host	Kind	PBS (VM image)	restic (file+DB)	Sole net?
ana-docker	VM (pfi-pve)	✅ ana-pve	✅ → rest-server-ana	no
ana-ml2	bare metal	❌ none (not a VM)	✅ → rest-server-ana	⚠️ restic is the ONLY net
irv-ml1	bare metal	❌ none (not a VM)	✅ → rest-server-nh3	⚠️ restic is the ONLY net
nh3-docker	VM (nh3-pve)	✅ nh3-pve	✅ → rest-server-nh3	no
esh-docker-vm	VM (esh-pve)	✅ esh-pve	✅ → rest-server-ana	no
esh-vm-db	VM (esh-pve-nas)	❌ none (esh-pve-nas not a PBS source)	✅ → rest-server-ana	⚠️ restic-only (a DB!)
vm-esh-nas	VM (esh-pve-nas)	❌ none (esh-pve-nas not a PBS source)	✅ → rest-server-ana	⚠️ restic-only
other pfi-pve / nh3-pve VMs/CTs	VM/CT	✅ respective ns	(PBS only)	no
SureFire sfsrv-pve	tenant VMs	✅ sfsrv-pve ns	(PBS only)	no

Bare-metal hosts have NO PBS coverage (PBS only backs up Proxmox guests). Their restic snapshot is the entire safety net — keep an eye on it. ana-ml2 → rest-server-ana; irv-ml1 → rest-server-nh3.

---

Layer 1 — PBS (whole-VM/CT images)

• PBS-ANA (pbs-ana, 10.250.50.90) — fleet primary. Datastore is NFS-backed: 10.250.50.50:/mnt/backup/pbs-ana mounted at /mnt/pbs-datastore (~20 TB). Backs up Proxmox guests via vzdump, organised by namespace per source hypervisor:
  • ana-pve — pfi-pve guests (ana-docker, pfi-postgres VM105, ana-nas CT109, webhost, filebot, pteradactyl, tacticalrmm, ana-wg, …)
  • esh-pve — esh-pve guests
  • nh3-pve — nh3-pve guests
  • sfsrv-pve — SureFire tenant
  • ⚠️ there is no esh-pve-nas namespace — guests on that hypervisor (vm-esh-nas, likely esh-vm-db) are not PBS-covered.
• PBS-NH3 (pbs-nh3, 10.100.50.90) — DR mirror; syncs from PBS-ANA (datastore on nh3-nas).
• Schedule: vzdump jobs defined in Proxmox (Datacenter → Backup), staggered through the early morning.

Layer 2 — restic (granular file + DB)

restic clients push to one of two rest-server endpoints (HTTP, basic auth, append-only, private repos). The split is by site:

rest-server	Endpoint	Backing store	Clients
rest-server-ana	http://10.250.50.70:8000 (container rest-server on ana-docker)	ana-nas:/mnt/backup/restic/repo/ana (NFS bind → /data)	ana-docker, ana-ml2, esh-docker-vm, esh-vm-db, vm-esh-nas
rest-server-nh3	http://10.100.50.50:8000 (on nh3-nas)	nh3-nas:/volume1/Backup/restic/<client>	irv-ml1, nh3-docker

• Per-client repos live as subdirs of the rest-server data dir (.../repo/ana/<client>/ for the ana side); the shared .htpasswd for ana sits at .../repo/ana/.htpasswd.
• Scheduler = resticprofile systemd timers on each client host, NOT Backrest:
  • resticprofile-backup@profile-default.timer — daily 01:00 PDT
  • resticprofile-check@profile-default.timer — weekly (Sun 05:00)
• Backrest (container on ana-docker, UI) is only a repo viewer here — it has 0 plans. Do not assume "Backrest healthy" means "backups running." The timers are the source of truth.
• ⚠️ Failures are silent — a timer fires, restic errors against a down endpoint, and nothing alerts. (See Known gaps.)

---

⚠ Answering "is X backed up?" — ask the repo, never the job

Coverage is a property of the backup SYSTEM, not of any one job's configured scope. Establish it by querying the repository for the path and seeing it in a real snapshot. Never by reading a job's SRC=.

# as infra-ops on the host, against its own profile
sudo -n bash -c 'set -a; . /etc/restic/restic.env; set +a
  export RESTIC_PASSWORD_FILE=/etc/restic/password
  id=$(restic snapshots --latest 1 --json | jq -r ".[0].short_id")
  restic ls "$id" /path/in/question | head'

The failure this rule exists to prevent (2026-09-22). A peer agent checked dev-backup.sh, found SRC="$HOME/development/", and reported — with specifics, unhedged, to the operator — that five home-directory paths including Miranda's entire conversation had "never been backed up anywhere". All five were in that night's restic snapshot. dev-backup is the hourly job for one directory; resticprofile is the daily job covering all of /home/lkraven, and it is what actually protects home. Checking one job and generalising to the system produced a confident, false, escalated claim.

The true finding underneath was much smaller and is a cadence question, not a coverage one: live conversation state has a 24-hour RPO because it rides the daily job rather than the hourly one.

The 2-minute freshness check

Run these any time you need to answer "are we backed up?"

# --- restic ANA side: newest snapshot per client (want: today/yesterday) ---
ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do
  echo -n "$c: "; ls -t /mnt/backup/restic/repo/ana/$c/snapshots/ 2>/dev/null | head -1 \
    | xargs -I{} stat -c "%y" /mnt/backup/restic/repo/ana/$c/snapshots/{} 2>/dev/null || echo MISSING
done'

# --- restic NH3 side ---
ssh nh3-nas 'for c in irv-ml1 nh3-docker; do
  echo -n "$c: "; ls -lt /volume1/Backup/restic/$c/snapshots/ 2>/dev/null | sed -n 2p
done'

# --- rest-server endpoints healthy? (401 = up & serving; Restarting = broken) ---
ssh infra-ops@ana-docker 'sudo docker ps --format "{{.Names}}\t{{.Status}}" | grep rest-server'
curl -s -o /dev/null -w 'rest-server-ana: %{http_code}\n' http://10.250.50.70:8000/
curl -s -o /dev/null -w 'rest-server-nh3: %{http_code}\n' http://10.100.50.50:8000/

# --- PBS: newest snapshot per guest, all namespaces ---
ssh pbs-ana 'for ns in /mnt/pbs-datastore/ns/*/; do nsn=$(basename "$ns")
  for d in vm ct; do for g in "$ns$d"/*/; do [ -d "$g" ] || continue
    echo "$nsn/$d/$(basename "$g") -> $(ls "$g" 2>/dev/null | grep ^20 | sort | tail -1)"
  done; done; done'

Force a backup now (don't wait for 01:00): on the client host, ssh infra-ops@<host> 'sudo systemctl start resticprofile-backup@profile-default.service' (it's an incremental against the existing repo — bounded even if stale).

---

Known failure mode: rest-server-ana crash-loop (the 2026-05-06 → 2026-06-20 outage)

Symptom: rest-server container on ana-docker stuck Restarting; logs show cannot load /data/.htpasswd: permission denied. All ana-side restic backups silently fail.

Root cause: ana-nas's NFS mount on ana-docker uses bare defaults in /etc/fstab (no _netdev, no retry). When the mount drops, mnt-backup.mount gets stuck failed, so /mnt/backup/restic/repo/ana resolves to an empty local ghost dir (no .htpasswd) and rest-server binds that. ana-nas itself is fine — the real repos are intact.

Recovery (needs root on ana-docker — use ssh infra-ops@ana-docker, which has NOPASSWD sudo; the default ssh ana-docker lands as lkraven without sudo):

ssh infra-ops@ana-docker '
  sudo mount -a                                   # re-attach the NFS (bypasses the failed unit)
  sudo systemctl reset-failed mnt-backup.mount    # clear the stuck unit state
  mount | grep /mnt/backup                         # confirm nfs4 attached
  sudo ls /mnt/backup/restic/repo/ana/.htpasswd    # real htpasswd now present
  cd /opt/docker/compose/rest-server-ana && sudo docker compose up -d --force-recreate
'                                                  # recreate so the bind re-resolves onto NFS
# verify: docker ps shows Up (healthy); curl :8000 -> 401; logs say "Loaded htpasswd file"

If the ghost dir blocks the mount, see disaster-recovery.md Tier-0 for the stop→umount→rm-ghost→remount→start variant.

---

Known gaps / TODO

✅ restic content assertion (2026-09-22, CLOSED)

⚠ I reported this gap wrongly first. I ran grep -ic restic against backup-freshness-alert.sh — the wrapper — got 0, and told the operator nothing on the fleet verified restic. The actual checking lives in check-backup-freshness.sh, which had always checked restic snapshot ages for seven hosts. Grepping one file of a two-file tool and generalising is the same error a peer had just made with dev-backup, made while correcting them.

The real gap was narrower and still real: the host list read irv-ml1 nh3-docker and omitted nh3-dev — the repo holding every Claude Code transcript, althing routes, hermes history and Miranda's conversation. /volume1/Backup/restic/nh3-dev/ had always existed and always been written; it was simply never enumerated, so a total failure of the one repo nobody could reconstruct would have gone unreported indefinitely.

Both are fixed. nh3-dev is in the list, and a content assertion now runs per the operator ruling of 2026-09-22 (assert snapshot content, not REST-server reachability):

• Conjunctive: latest snapshot newer than MAX_AGE_H AND the probe path present AND a restore returns non-zero bytes. Age alone was the old check's problem; content alone is the same problem rotated.
• It restores. Listing proves the index; only a restore proves the bytes come back, and metadata is exactly what survives a pruned or partially-written repo. ~493 KB from ~/.local/state/svos, sub-second. The large paths are not restored: a repo that returns one file will return others, one that cannot is broken for everything.
• Identity: the timer runs as lkraven (no NOPASSWD sudo here), so the probe hops through infra-ops@localhost, which has it. The credentials under /etc/restic are deliberately root-only.

⚠ restic ls always prints a header line, matched or not — a path absent from the repo returns 1 line, a real one returned 6. Counting with grep -c . read the header as a hit and made the ABSENT case fall through to "blobs gone", telling an operator the repository was corrupt when the truth was a mistyped path. Count grep -c '^/'.

Tested by making it fail, because a check only ever seen passing is untested: a bogus probe path reports absent; BACKUP_MAX_AGE_HOURS=1 reports too old; the healthy path reports the snapshot id, age, entry count and bytes restored.

• Backup-freshness alerting — DONE (2026-06-20). scripts/check-backup-freshness.sh (the 2-min check, exit 1 on stale/down) + a daily systemd user timer on nh3-dev at 08:00 (scripts/install-backup-freshness-timer.sh) → backup-freshness-alert.sh posts an althing alert to infra-ops on any stale/down layer. Run the check by hand anytime. (Channel is althing for now — swap in email/ntfy if you want a louder one.)
• fstab hardening — DONE (2026-06-20). ana-docker /mnt/backup → noauto,x-systemd.automount,x-systemd.mount-timeout=30 (autofs self-heals on a NAS blip instead of getting stuck failed; activates on next reboot). /etc/fstab.bak-pre-harden saved. /mnt/compose also hardened the same way and activated live (umount → mnt-compose.automount started → autofs verified remounting on access) — it binds no container, so it was safe to convert now; this also proved the autofs pattern works on ana-docker.
• ana-ml2 has no PBS net (bare metal) — restic is its only layer; now healthy + alerted. Bulk /tank models are re-downloadable; bespoke quants/configs/scripts are the real loss-risk.
• esh-pve-nas coverage — VERIFIED (2026-06-20): NOT PBS-covered. No esh-pve-nas namespace exists on PBS-ANA, so esh-vm-db (postgres+mongo) + vm-esh-nas are restic-only. For the DB VM, restic-with-dumps is the preferred method (vs a VM image) IF the resticprofile includes pg_dump/mongodump — confirm that. Optionally add esh-pve-nas as a PBS source. ESH is home-lab (no SLA).
• Rotate rest-server repo passwords — the 5 per-repo basic-auth creds were exposed during the 2026-06-20 diagnosis. BELAYED — operator handling offline.