From a7b4a82dec4f254dedb530351fc66b80bc156f2d Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 20 Jun 2026 15:56:41 -0700 Subject: [PATCH] docs(backups): add backup architecture + freshness runbook; record rest-server-ana recovery + correct ana-docker sudo path --- docs/runbooks/backups.md | 168 +++++++++++++++++++++++++++++++++++++++ persistent-memory.md | 36 ++++++--- 2 files changed, 193 insertions(+), 11 deletions(-) create mode 100644 docs/runbooks/backups.md diff --git a/docs/runbooks/backups.md b/docs/runbooks/backups.md new file mode 100644 index 0000000..8a13da9 --- /dev/null +++ b/docs/runbooks/backups.md @@ -0,0 +1,168 @@ +# Fleet backup architecture & freshness runbook + +The map that was missing: what backs up what, where it lands, and **how +to check in 2 minutes whether backups are actually fresh.** Companion to +[`disaster-recovery.md`](disaster-recovery.md) (which covers *recovery* +when a host/service is down). Read this one first when the question is +"are we backed up?" + +> **Why this exists:** on 2026-06-20 diagnosing "are backups OK?" took a +> long exploration because the topology lived only in scattered memory. +> The ana-side restic layer had been failing **silently for ~6.5 weeks** +> (last good snapshot 2026-05-06) and nobody knew. This doc + a future +> freshness alert is the fix. + +--- + +## TL;DR — coverage matrix + +Two independent layers. **PBS = whole-VM images. restic = granular +file+DB.** A host is well-covered if it has *either* a current PBS image +*or* a current restic snapshot; the danger zone is a host whose **only** +layer has failed. + +| Host | Kind | PBS (VM image) | restic (file+DB) | Sole net? | +|---|---|---|---|---| +| ana-docker | VM (pfi-pve) | ✅ `ana-pve` | ✅ → rest-server-**ana** | no | +| **ana-ml2** | **bare metal** | ❌ none (not a VM) | ✅ → rest-server-**ana** | ⚠️ **restic is the ONLY net** | +| **irv-ml1** | **bare metal** | ❌ none (not a VM) | ✅ → rest-server-**nh3** | ⚠️ **restic is the ONLY net** | +| nh3-docker | VM (nh3-pve) | ✅ `nh3-pve` | ✅ → rest-server-**nh3** | no | +| esh-docker-vm | VM (esh-pve) | ✅ `esh-pve` | ✅ → rest-server-**ana** | no | +| esh-vm-db | VM (esh-pve-nas?) | ❓ **no `esh-pve-nas` ns — VERIFY** | ✅ → rest-server-**ana** | ⚠️ maybe restic-only (a DB!) | +| vm-esh-nas | VM (esh-pve-nas) | ❓ **no `esh-pve-nas` ns — VERIFY** | ✅ → rest-server-**ana** | ⚠️ maybe restic-only | +| other pfi-pve / nh3-pve VMs/CTs | VM/CT | ✅ respective ns | (PBS only) | no | +| SureFire `sfsrv-pve` | tenant VMs | ✅ `sfsrv-pve` ns | (PBS only) | no | + +**Bare-metal hosts have NO PBS coverage** (PBS only backs up Proxmox +guests). Their restic snapshot is the entire safety net — keep an eye on +it. ana-ml2 → rest-server-ana; irv-ml1 → rest-server-nh3. + +--- + +## Layer 1 — PBS (whole-VM/CT images) + +- **PBS-ANA** (`pbs-ana`, 10.250.50.90) — fleet primary. Datastore is + NFS-backed: `10.250.50.50:/mnt/backup/pbs-ana` mounted at + `/mnt/pbs-datastore` (~20 TB). Backs up Proxmox guests via vzdump, + organised by **namespace per source hypervisor**: + - `ana-pve` — pfi-pve guests (ana-docker, pfi-postgres VM105, ana-nas + CT109, webhost, filebot, pteradactyl, tacticalrmm, ana-wg, …) + - `esh-pve` — esh-pve guests + - `nh3-pve` — nh3-pve guests + - `sfsrv-pve` — SureFire tenant + - ⚠️ there is **no `esh-pve-nas` namespace** — guests on that + hypervisor (vm-esh-nas, likely esh-vm-db) are **not** PBS-covered. +- **PBS-NH3** (`pbs-nh3`, 10.100.50.90) — DR mirror; syncs from PBS-ANA + (datastore on nh3-nas). +- Schedule: vzdump jobs defined in Proxmox (Datacenter → Backup), + staggered through the early morning. + +## Layer 2 — restic (granular file + DB) + +restic clients push to one of **two rest-server endpoints** (HTTP, basic +auth, append-only, private repos). The split is by site: + +| rest-server | Endpoint | Backing store | Clients | +|---|---|---|---| +| **rest-server-ana** | `http://10.250.50.70:8000` (container `rest-server` on ana-docker) | `ana-nas:/mnt/backup/restic/repo/ana` (NFS bind → `/data`) | ana-docker, **ana-ml2**, esh-docker-vm, esh-vm-db, vm-esh-nas | +| **rest-server-nh3** | `http://10.100.50.50:8000` (on nh3-nas) | `nh3-nas:/volume1/Backup/restic/` | **irv-ml1**, nh3-docker | + +- Per-client repos live as subdirs of the rest-server data dir + (`.../repo/ana//` for the ana side); the shared `.htpasswd` + for ana sits at `.../repo/ana/.htpasswd`. +- **Scheduler = `resticprofile` systemd timers on each client host**, NOT + Backrest: + - `resticprofile-backup@profile-default.timer` — daily **01:00** PDT + - `resticprofile-check@profile-default.timer` — weekly (Sun **05:00**) +- **Backrest** (container on ana-docker, UI) is only a **repo viewer here + — it has 0 plans.** Do not assume "Backrest healthy" means "backups + running." The timers are the source of truth. +- ⚠️ **Failures are silent** — a timer fires, restic errors against a + down endpoint, and nothing alerts. (See Known gaps.) + +--- + +## The 2-minute freshness check + +Run these any time you need to answer "are we backed up?" + +```bash +# --- restic ANA side: newest snapshot per client (want: today/yesterday) --- +ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do + echo -n "$c: "; ls -t /mnt/backup/restic/repo/ana/$c/snapshots/ 2>/dev/null | head -1 \ + | xargs -I{} stat -c "%y" /mnt/backup/restic/repo/ana/$c/snapshots/{} 2>/dev/null || echo MISSING +done' + +# --- restic NH3 side --- +ssh nh3-nas 'for c in irv-ml1 nh3-docker; do + echo -n "$c: "; ls -lt /volume1/Backup/restic/$c/snapshots/ 2>/dev/null | sed -n 2p +done' + +# --- rest-server endpoints healthy? (401 = up & serving; Restarting = broken) --- +ssh infra-ops@ana-docker 'sudo docker ps --format "{{.Names}}\t{{.Status}}" | grep rest-server' +curl -s -o /dev/null -w 'rest-server-ana: %{http_code}\n' http://10.250.50.70:8000/ +curl -s -o /dev/null -w 'rest-server-nh3: %{http_code}\n' http://10.100.50.50:8000/ + +# --- PBS: newest snapshot per guest, all namespaces --- +ssh pbs-ana 'for ns in /mnt/pbs-datastore/ns/*/; do nsn=$(basename "$ns") + for d in vm ct; do for g in "$ns$d"/*/; do [ -d "$g" ] || continue + echo "$nsn/$d/$(basename "$g") -> $(ls "$g" 2>/dev/null | grep ^20 | sort | tail -1)" + done; done; done' +``` + +**Force a backup now (don't wait for 01:00):** on the client host, +`ssh infra-ops@ 'sudo systemctl start resticprofile-backup@profile-default.service'` +(it's an incremental against the existing repo — bounded even if stale). + +--- + +## Known failure mode: rest-server-ana crash-loop (the 2026-05-06 → 2026-06-20 outage) + +**Symptom:** `rest-server` container on ana-docker stuck `Restarting`; +logs show `cannot load /data/.htpasswd: permission denied`. All ana-side +restic backups silently fail. + +**Root cause:** ana-nas's NFS mount on ana-docker uses bare `defaults` in +`/etc/fstab` (no `_netdev`, no retry). When the mount drops, +`mnt-backup.mount` gets stuck `failed`, so `/mnt/backup/restic/repo/ana` +resolves to an **empty local ghost dir** (no `.htpasswd`) and rest-server +binds *that*. ana-nas itself is fine — the real repos are intact. + +**Recovery** (needs root on ana-docker — use **`ssh infra-ops@ana-docker`**, +which has NOPASSWD sudo; the default `ssh ana-docker` lands as `lkraven` +*without* sudo): + +```bash +ssh infra-ops@ana-docker ' + sudo mount -a # re-attach the NFS (bypasses the failed unit) + sudo systemctl reset-failed mnt-backup.mount # clear the stuck unit state + mount | grep /mnt/backup # confirm nfs4 attached + sudo ls /mnt/backup/restic/repo/ana/.htpasswd # real htpasswd now present + cd /opt/docker/compose/rest-server-ana && sudo docker compose up -d --force-recreate +' # recreate so the bind re-resolves onto NFS +# verify: docker ps shows Up (healthy); curl :8000 -> 401; logs say "Loaded htpasswd file" +``` + +If the ghost dir blocks the mount, see `disaster-recovery.md` Tier-0 for +the stop→umount→rm-ghost→remount→start variant. + +--- + +## Known gaps / TODO + +- [ ] **No backup-freshness alerting** — the systemic miss; the ana + outage went unnoticed ~6.5 weeks. Add a check (e.g. a daily job that + alerts if any repo's newest snapshot is > 48 h old, or wire + resticprofile's check/`--json` output to a notifier). +- [ ] **fstab hardening** — change ana-docker's `/mnt/backup` entry from + `defaults` to `_netdev,nofail,x-systemd.automount,x-systemd.mount-timeout=30,retry` + so it self-heals on reboot/NAS-blip instead of getting stuck. +- [ ] **ana-ml2 has no PBS net** (bare metal) — restic is its only layer. + Acceptable IF restic stays healthy + alerted; bulk `/tank` models are + re-downloadable, but bespoke quants/configs/scripts are not. +- [ ] **Verify esh-pve-nas coverage** — vm-esh-nas + **esh-vm-db (a + postgres+mongo DB VM)** have no PBS namespace. Confirm whether they + get a PBS image anywhere; if not, restic is their sole net. +- [ ] **Rotate rest-server repo passwords** — the 5 per-repo basic-auth + creds were exposed during the 2026-06-20 diagnosis (Backrest config + dump). Add to the disclosed-keys rotation queue. diff --git a/persistent-memory.md b/persistent-memory.md index 69c8144..4eea4b0 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -98,12 +98,14 @@ no longer deployed sidecars here. See Recent decisions.) playbooks run fully non-interactive over key SSH. - **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo - on most PFI Linux boxes (corviduo-dev included since 2026-06-15) — BUT - **on ana-docker the session lands as `lkraven` (docker-group, NO - passwordless sudo)**. So on ana-docker: docker commands work directly; - root-owned-file edits need the docker-as-root trick or infra-ops; but - the litellm config + most stack compose/conf are `lkraven`-owned → - plain `cp`/edit, no sudo. (Learned 2026-06-20 — see Tried-and-abandoned.) + on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On + ana-docker there are TWO identities: the **default `ssh ana-docker` = + `lkraven`** (docker-group, NO passwordless sudo — docker works, root-file + edits don't); **but `ssh infra-ops@ana-docker` HAS NOPASSWD root** (verified + 2026-06-20). **→ For any sudo op on ana-docker (mount, root-owned files, + service control), use `ssh infra-ops@ana-docker`, NOT the default session.** + lkraven-owned files (litellm config, most stack compose/conf) still take + plain `cp`/edit under either identity. ## Current state / in-flight @@ -190,6 +192,16 @@ _As of 2026-06-20:_ **⚠️ Disclosed creds:** diagnosing dumped the Backrest config, exposing the 5 per-repo rest-server basic-auth passwords into the session transcript → rotate after recovery (add to the disclosed-keys rotation queue). + **✅ RECOVERED (2026-06-20):** via `ssh infra-ops@ana-docker` (NOT the default + lkraven session) — `sudo mount -a` + `reset-failed mnt-backup.mount` re-attached + the NFS, then `docker compose up -d --force-recreate` rebound rest-server onto + the real `/data` (now `Loaded htpasswd file`, 401, healthy). Verified end-to-end: + triggered backups on the two no-PBS hosts' priority — **ana-ml2 snapshot landed + 22:53Z** (total 19→20); ana-docker's larger catch-up was still running at hand-off + (will land). esh-docker-vm/esh-vm-db/vm-esh-nas auto-catch-up at their next 01:00 + timers (or trigger manually). **Full topology + 2-min freshness-check now documented + in `docs/runbooks/backups.md`.** STILL OPEN (prevention): backup-freshness alert + + fstab `_netdev`/retry hardening + verify esh-pve-nas PBS gap + rotate the 5 creds. - **ana-docker disk incident RESOLVED — root cause was a 94 GB unrotated `langfuse-clickhouse` log, NOT (just) image bloat.** ClickHouse logs @@ -377,11 +389,13 @@ _101 older entries archived to archival-memory.md._ state). Prune them with `docker buildx prune --builder -af`, or `docker volume rm` orphaned ones (builder already removed). -- `[2026-06-20]` **On ana-docker I am `lkraven` (no NOPASSWD sudo), not - infra-ops.** A `sudo cp` of the litellm config silently failed (password - prompt) and I restarted the gateway on an unchanged config (one wasted - bounce). The config + most stack files are `lkraven`-owned → plain `cp`, - no sudo. Verify the SSH user/ownership before reaching for sudo. +- `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) — + but `ssh infra-ops@ana-docker` HAS NOPASSWD root** (corrected later same day). + Early on a `sudo cp` as lkraven silently failed (password prompt) → one + wasted gateway bounce; I then wrongly concluded "no sudo on ana-docker" and + nearly punted the rest-server-ana recovery to the operator. The real rule: + reach for `infra-ops@ana-docker` for sudo ops; lkraven-owned files (litellm + config, most stack compose/conf) take plain `cp` under either identity. - `[2026-06-19]` **litellm CONFIG-loaded models can't be hot-removed — `/model/delete` is DB-only.** Only a process restart drops a config model. The INVERSE works: `/model/new` live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce.