esh-pfi-infrastructure/docs/runbooks/backups.md

Fleet backup architecture & freshness runbook

The map that was missing: what backs up what, where it lands, and how to check in 2 minutes whether backups are actually fresh. Companion to disaster-recovery.md (which covers recovery when a host/service is down). Read this one first when the question is "are we backed up?"

Why this exists: on 2026-06-20 diagnosing "are backups OK?" took a long exploration because the topology lived only in scattered memory. The ana-side restic layer had been failing silently for ~6.5 weeks (last good snapshot 2026-05-06) and nobody knew. This doc + a future freshness alert is the fix.

---

TL;DR — coverage matrix

Two independent layers. PBS = whole-VM images. restic = granular file+DB. A host is well-covered if it has either a current PBS image or a current restic snapshot; the danger zone is a host whose only layer has failed.

Host	Kind	PBS (VM image)	restic (file+DB)	Sole net?
ana-docker	VM (pfi-pve)	✅ ana-pve	✅ → rest-server-ana	no
ana-ml2	bare metal	❌ none (not a VM)	✅ → rest-server-ana	⚠️ restic is the ONLY net
irv-ml1	bare metal	❌ none (not a VM)	✅ → rest-server-nh3	⚠️ restic is the ONLY net
nh3-docker	VM (nh3-pve)	✅ nh3-pve	✅ → rest-server-nh3	no
esh-docker-vm	VM (esh-pve)	✅ esh-pve	✅ → rest-server-ana	no
esh-vm-db	VM (esh-pve-nas)	❌ none (esh-pve-nas not a PBS source)	✅ → rest-server-ana	⚠️ restic-only (a DB!)
vm-esh-nas	VM (esh-pve-nas)	❌ none (esh-pve-nas not a PBS source)	✅ → rest-server-ana	⚠️ restic-only
other pfi-pve / nh3-pve VMs/CTs	VM/CT	✅ respective ns	(PBS only)	no
SureFire sfsrv-pve	tenant VMs	✅ sfsrv-pve ns	(PBS only)	no

Bare-metal hosts have NO PBS coverage (PBS only backs up Proxmox guests). Their restic snapshot is the entire safety net — keep an eye on it. ana-ml2 → rest-server-ana; irv-ml1 → rest-server-nh3.

---

Layer 1 — PBS (whole-VM/CT images)

• PBS-ANA (pbs-ana, 10.250.50.90) — fleet primary. Datastore is NFS-backed: 10.250.50.50:/mnt/backup/pbs-ana mounted at /mnt/pbs-datastore (~20 TB). Backs up Proxmox guests via vzdump, organised by namespace per source hypervisor:
  • ana-pve — pfi-pve guests (ana-docker, pfi-postgres VM105, ana-nas CT109, webhost, filebot, pteradactyl, tacticalrmm, ana-wg, …)
  • esh-pve — esh-pve guests
  • nh3-pve — nh3-pve guests
  • sfsrv-pve — SureFire tenant
  • ⚠️ there is no esh-pve-nas namespace — guests on that hypervisor (vm-esh-nas, likely esh-vm-db) are not PBS-covered.
• PBS-NH3 (pbs-nh3, 10.100.50.90) — DR mirror; syncs from PBS-ANA (datastore on nh3-nas).
• Schedule: vzdump jobs defined in Proxmox (Datacenter → Backup), staggered through the early morning.

Layer 2 — restic (granular file + DB)

restic clients push to one of two rest-server endpoints (HTTP, basic auth, append-only, private repos). The split is by site:

rest-server	Endpoint	Backing store	Clients
rest-server-ana	http://10.250.50.70:8000 (container rest-server on ana-docker)	ana-nas:/mnt/backup/restic/repo/ana (NFS bind → /data)	ana-docker, ana-ml2, esh-docker-vm, esh-vm-db, vm-esh-nas
rest-server-nh3	http://10.100.50.50:8000 (on nh3-nas)	nh3-nas:/volume1/Backup/restic/<client>	irv-ml1, nh3-docker

• Per-client repos live as subdirs of the rest-server data dir (.../repo/ana/<client>/ for the ana side); the shared .htpasswd for ana sits at .../repo/ana/.htpasswd.
• Scheduler = resticprofile systemd timers on each client host, NOT Backrest:
  • resticprofile-backup@profile-default.timer — daily 01:00 PDT
  • resticprofile-check@profile-default.timer — weekly (Sun 05:00)
• Backrest (container on ana-docker, UI) is only a repo viewer here — it has 0 plans. Do not assume "Backrest healthy" means "backups running." The timers are the source of truth.
• ⚠️ Failures are silent — a timer fires, restic errors against a down endpoint, and nothing alerts. (See Known gaps.)

---

⚠ Answering "is X backed up?" — ask the repo, never the job

Coverage is a property of the backup SYSTEM, not of any one job's configured scope. Establish it by querying the repository for the path and seeing it in a real snapshot. Never by reading a job's SRC=.

# as infra-ops on the host, against its own profile
sudo -n bash -c 'set -a; . /etc/restic/restic.env; set +a
  export RESTIC_PASSWORD_FILE=/etc/restic/password
  id=$(restic snapshots --latest 1 --json | jq -r ".[0].short_id")
  restic ls "$id" /path/in/question | head'

The failure this rule exists to prevent (2026-09-22). A peer agent checked dev-backup.sh, found SRC="$HOME/development/", and reported — with specifics, unhedged, to the operator — that five home-directory paths including Miranda's entire conversation had "never been backed up anywhere". All five were in that night's restic snapshot. dev-backup is the hourly job for one directory; resticprofile is the daily job covering all of /home/lkraven, and it is what actually protects home. Checking one job and generalising to the system produced a confident, false, escalated claim.

The true finding underneath was much smaller and is a cadence question, not a coverage one: live conversation state has a 24-hour RPO because it rides the daily job rather than the hourly one.

The 2-minute freshness check

Run these any time you need to answer "are we backed up?"

# --- restic ANA side: newest snapshot per client (want: today/yesterday) ---
ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do
  echo -n "$c: "; ls -t /mnt/backup/restic/repo/ana/$c/snapshots/ 2>/dev/null | head -1 \
    | xargs -I{} stat -c "%y" /mnt/backup/restic/repo/ana/$c/snapshots/{} 2>/dev/null || echo MISSING
done'

# --- restic NH3 side ---
ssh nh3-nas 'for c in irv-ml1 nh3-docker; do
  echo -n "$c: "; ls -lt /volume1/Backup/restic/$c/snapshots/ 2>/dev/null | sed -n 2p
done'

# --- rest-server endpoints healthy? (401 = up & serving; Restarting = broken) ---
ssh infra-ops@ana-docker 'sudo docker ps --format "{{.Names}}\t{{.Status}}" | grep rest-server'
curl -s -o /dev/null -w 'rest-server-ana: %{http_code}\n' http://10.250.50.70:8000/
curl -s -o /dev/null -w 'rest-server-nh3: %{http_code}\n' http://10.100.50.50:8000/

# --- PBS: newest snapshot per guest, all namespaces ---
ssh pbs-ana 'for ns in /mnt/pbs-datastore/ns/*/; do nsn=$(basename "$ns")
  for d in vm ct; do for g in "$ns$d"/*/; do [ -d "$g" ] || continue
    echo "$nsn/$d/$(basename "$g") -> $(ls "$g" 2>/dev/null | grep ^20 | sort | tail -1)"
  done; done; done'

Force a backup now (don't wait for 01:00): on the client host, ssh infra-ops@<host> 'sudo systemctl start resticprofile-backup@profile-default.service' (it's an incremental against the existing repo — bounded even if stale).

---

Known failure mode: rest-server-ana crash-loop (the 2026-05-06 → 2026-06-20 outage)

Symptom: rest-server container on ana-docker stuck Restarting; logs show cannot load /data/.htpasswd: permission denied. All ana-side restic backups silently fail.

Root cause: ana-nas's NFS mount on ana-docker uses bare defaults in /etc/fstab (no _netdev, no retry). When the mount drops, mnt-backup.mount gets stuck failed, so /mnt/backup/restic/repo/ana resolves to an empty local ghost dir (no .htpasswd) and rest-server binds that. ana-nas itself is fine — the real repos are intact.

Recovery (needs root on ana-docker — use ssh infra-ops@ana-docker, which has NOPASSWD sudo; the default ssh ana-docker lands as lkraven without sudo):

ssh infra-ops@ana-docker '
  sudo mount -a                                   # re-attach the NFS (bypasses the failed unit)
  sudo systemctl reset-failed mnt-backup.mount    # clear the stuck unit state
  mount | grep /mnt/backup                         # confirm nfs4 attached
  sudo ls /mnt/backup/restic/repo/ana/.htpasswd    # real htpasswd now present
  cd /opt/docker/compose/rest-server-ana && sudo docker compose up -d --force-recreate
'                                                  # recreate so the bind re-resolves onto NFS
# verify: docker ps shows Up (healthy); curl :8000 -> 401; logs say "Loaded htpasswd file"

If the ghost dir blocks the mount, see disaster-recovery.md Tier-0 for the stop→umount→rm-ghost→remount→start variant.

---

Known gaps / TODO

✅ restic content assertion (2026-09-22, CLOSED)

⚠ I reported this gap wrongly first. I ran grep -ic restic against backup-freshness-alert.sh — the wrapper — got 0, and told the operator nothing on the fleet verified restic. The actual checking lives in check-backup-freshness.sh, which had always checked restic snapshot ages for seven hosts. Grepping one file of a two-file tool and generalising is the same error a peer had just made with dev-backup, made while correcting them.

The real gap was narrower and still real: the host list read irv-ml1 nh3-docker and omitted nh3-dev — the repo holding every Claude Code transcript, althing routes, hermes history and Miranda's conversation. /volume1/Backup/restic/nh3-dev/ had always existed and always been written; it was simply never enumerated, so a total failure of the one repo nobody could reconstruct would have gone unreported indefinitely.

Both are fixed. nh3-dev is in the list, and a content assertion now runs per the operator ruling of 2026-09-22 (assert snapshot content, not REST-server reachability):

• Conjunctive: latest snapshot newer than MAX_AGE_H AND the probe path present AND a restore returns non-zero bytes. Age alone was the old check's problem; content alone is the same problem rotated.
• It restores. Listing proves the index; only a restore proves the bytes come back, and metadata is exactly what survives a pruned or partially-written repo. ~493 KB from ~/.local/state/svos, sub-second. The large paths are not restored: a repo that returns one file will return others, one that cannot is broken for everything.
• Identity: the timer runs as lkraven (no NOPASSWD sudo here), so the probe hops through infra-ops@localhost, which has it. The credentials under /etc/restic are deliberately root-only.

Repos are DISCOVERED, not enumerated

The host list is no longer hand-written. check-backup-freshness.sh asks each NAS which repositories exist — a directory counts as one when it has a snapshots/ child, which distinguishes a real repo from a container dir (/volume1/Backup/restic/repo/ holds ana|esh|nh3 namespaces and no snapshots of its own).

Why (svos-dev's framing, 2026-09-22): adding nh3-dev fixed the instance; the class was that a hand-maintained list of things to watch, sitting beside a NAS that already knows which repos exist, means the next repo added is unwatched by default and nothing says so. The list of what to check can silently disagree with the set of what exists.

The hand-written list survives demoted, as EXPECTED_RESTIC_ANA / EXPECTED_RESTIC_NH3, used only to report a repo that has vanished. That keeps two facts distinct that would otherwise both read as silence:

fact	now
a repo exists that nobody watches	impossible — it is discovered
a repo we expected is gone	reported as EXPECTED REPO NOT FOUND

Currently 8 repos: 5 on ana-nas (repo/ana/*), 3 on nh3-nas. The esh and nh3 namespaces under repo/ on ana-nas are empty.

⚠ restic ls always prints a header line, matched or not — a path absent from the repo returns 1 line, a real one returned 6. Counting with grep -c . read the header as a hit and made the ABSENT case fall through to "blobs gone", telling an operator the repository was corrupt when the truth was a mistyped path. Count grep -c '^/'.

Tested by making it fail, because a check only ever seen passing is untested: a bogus probe path reports absent; BACKUP_MAX_AGE_HOURS=1 reports too old; the healthy path reports the snapshot id, age, entry count and bytes restored.

• Backup-freshness alerting — DONE (2026-06-20). scripts/check-backup-freshness.sh (the 2-min check, exit 1 on stale/down) + a daily systemd user timer on nh3-dev at 08:00 (scripts/install-backup-freshness-timer.sh) → backup-freshness-alert.sh posts an althing alert to infra-ops on any stale/down layer. Run the check by hand anytime. (Channel is althing for now — swap in email/ntfy if you want a louder one.)
• fstab hardening — DONE (2026-06-20). ana-docker /mnt/backup → noauto,x-systemd.automount,x-systemd.mount-timeout=30 (autofs self-heals on a NAS blip instead of getting stuck failed; activates on next reboot). /etc/fstab.bak-pre-harden saved. /mnt/compose also hardened the same way and activated live (umount → mnt-compose.automount started → autofs verified remounting on access) — it binds no container, so it was safe to convert now; this also proved the autofs pattern works on ana-docker.
• ana-ml2 has no PBS net (bare metal) — restic is its only layer; now healthy + alerted. Bulk /tank models are re-downloadable; bespoke quants/configs/scripts are the real loss-risk.
• esh-pve-nas coverage — VERIFIED (2026-06-20): NOT PBS-covered. No esh-pve-nas namespace exists on PBS-ANA, so esh-vm-db (postgres+mongo) + vm-esh-nas are restic-only. For the DB VM, restic-with-dumps is the preferred method (vs a VM image) IF the resticprofile includes pg_dump/mongodump — confirm that. Optionally add esh-pve-nas as a PBS source. ESH is home-lab (no SLA).
• Rotate rest-server repo passwords — the 5 per-repo basic-auth creds were exposed during the 2026-06-20 diagnosis. BELAYED — operator handling offline.