docs(backups): add backup architecture + freshness runbook; record rest-server-ana recovery + correct ana-docker sudo path
This commit is contained in:
@@ -0,0 +1,168 @@
|
||||
# Fleet backup architecture & freshness runbook
|
||||
|
||||
The map that was missing: what backs up what, where it lands, and **how
|
||||
to check in 2 minutes whether backups are actually fresh.** Companion to
|
||||
[`disaster-recovery.md`](disaster-recovery.md) (which covers *recovery*
|
||||
when a host/service is down). Read this one first when the question is
|
||||
"are we backed up?"
|
||||
|
||||
> **Why this exists:** on 2026-06-20 diagnosing "are backups OK?" took a
|
||||
> long exploration because the topology lived only in scattered memory.
|
||||
> The ana-side restic layer had been failing **silently for ~6.5 weeks**
|
||||
> (last good snapshot 2026-05-06) and nobody knew. This doc + a future
|
||||
> freshness alert is the fix.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR — coverage matrix
|
||||
|
||||
Two independent layers. **PBS = whole-VM images. restic = granular
|
||||
file+DB.** A host is well-covered if it has *either* a current PBS image
|
||||
*or* a current restic snapshot; the danger zone is a host whose **only**
|
||||
layer has failed.
|
||||
|
||||
| Host | Kind | PBS (VM image) | restic (file+DB) | Sole net? |
|
||||
|---|---|---|---|---|
|
||||
| ana-docker | VM (pfi-pve) | ✅ `ana-pve` | ✅ → rest-server-**ana** | no |
|
||||
| **ana-ml2** | **bare metal** | ❌ none (not a VM) | ✅ → rest-server-**ana** | ⚠️ **restic is the ONLY net** |
|
||||
| **irv-ml1** | **bare metal** | ❌ none (not a VM) | ✅ → rest-server-**nh3** | ⚠️ **restic is the ONLY net** |
|
||||
| nh3-docker | VM (nh3-pve) | ✅ `nh3-pve` | ✅ → rest-server-**nh3** | no |
|
||||
| esh-docker-vm | VM (esh-pve) | ✅ `esh-pve` | ✅ → rest-server-**ana** | no |
|
||||
| esh-vm-db | VM (esh-pve-nas?) | ❓ **no `esh-pve-nas` ns — VERIFY** | ✅ → rest-server-**ana** | ⚠️ maybe restic-only (a DB!) |
|
||||
| vm-esh-nas | VM (esh-pve-nas) | ❓ **no `esh-pve-nas` ns — VERIFY** | ✅ → rest-server-**ana** | ⚠️ maybe restic-only |
|
||||
| other pfi-pve / nh3-pve VMs/CTs | VM/CT | ✅ respective ns | (PBS only) | no |
|
||||
| SureFire `sfsrv-pve` | tenant VMs | ✅ `sfsrv-pve` ns | (PBS only) | no |
|
||||
|
||||
**Bare-metal hosts have NO PBS coverage** (PBS only backs up Proxmox
|
||||
guests). Their restic snapshot is the entire safety net — keep an eye on
|
||||
it. ana-ml2 → rest-server-ana; irv-ml1 → rest-server-nh3.
|
||||
|
||||
---
|
||||
|
||||
## Layer 1 — PBS (whole-VM/CT images)
|
||||
|
||||
- **PBS-ANA** (`pbs-ana`, 10.250.50.90) — fleet primary. Datastore is
|
||||
NFS-backed: `10.250.50.50:/mnt/backup/pbs-ana` mounted at
|
||||
`/mnt/pbs-datastore` (~20 TB). Backs up Proxmox guests via vzdump,
|
||||
organised by **namespace per source hypervisor**:
|
||||
- `ana-pve` — pfi-pve guests (ana-docker, pfi-postgres VM105, ana-nas
|
||||
CT109, webhost, filebot, pteradactyl, tacticalrmm, ana-wg, …)
|
||||
- `esh-pve` — esh-pve guests
|
||||
- `nh3-pve` — nh3-pve guests
|
||||
- `sfsrv-pve` — SureFire tenant
|
||||
- ⚠️ there is **no `esh-pve-nas` namespace** — guests on that
|
||||
hypervisor (vm-esh-nas, likely esh-vm-db) are **not** PBS-covered.
|
||||
- **PBS-NH3** (`pbs-nh3`, 10.100.50.90) — DR mirror; syncs from PBS-ANA
|
||||
(datastore on nh3-nas).
|
||||
- Schedule: vzdump jobs defined in Proxmox (Datacenter → Backup),
|
||||
staggered through the early morning.
|
||||
|
||||
## Layer 2 — restic (granular file + DB)
|
||||
|
||||
restic clients push to one of **two rest-server endpoints** (HTTP, basic
|
||||
auth, append-only, private repos). The split is by site:
|
||||
|
||||
| rest-server | Endpoint | Backing store | Clients |
|
||||
|---|---|---|---|
|
||||
| **rest-server-ana** | `http://10.250.50.70:8000` (container `rest-server` on ana-docker) | `ana-nas:/mnt/backup/restic/repo/ana` (NFS bind → `/data`) | ana-docker, **ana-ml2**, esh-docker-vm, esh-vm-db, vm-esh-nas |
|
||||
| **rest-server-nh3** | `http://10.100.50.50:8000` (on nh3-nas) | `nh3-nas:/volume1/Backup/restic/<client>` | **irv-ml1**, nh3-docker |
|
||||
|
||||
- Per-client repos live as subdirs of the rest-server data dir
|
||||
(`.../repo/ana/<client>/` for the ana side); the shared `.htpasswd`
|
||||
for ana sits at `.../repo/ana/.htpasswd`.
|
||||
- **Scheduler = `resticprofile` systemd timers on each client host**, NOT
|
||||
Backrest:
|
||||
- `resticprofile-backup@profile-default.timer` — daily **01:00** PDT
|
||||
- `resticprofile-check@profile-default.timer` — weekly (Sun **05:00**)
|
||||
- **Backrest** (container on ana-docker, UI) is only a **repo viewer here
|
||||
— it has 0 plans.** Do not assume "Backrest healthy" means "backups
|
||||
running." The timers are the source of truth.
|
||||
- ⚠️ **Failures are silent** — a timer fires, restic errors against a
|
||||
down endpoint, and nothing alerts. (See Known gaps.)
|
||||
|
||||
---
|
||||
|
||||
## The 2-minute freshness check
|
||||
|
||||
Run these any time you need to answer "are we backed up?"
|
||||
|
||||
```bash
|
||||
# --- restic ANA side: newest snapshot per client (want: today/yesterday) ---
|
||||
ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do
|
||||
echo -n "$c: "; ls -t /mnt/backup/restic/repo/ana/$c/snapshots/ 2>/dev/null | head -1 \
|
||||
| xargs -I{} stat -c "%y" /mnt/backup/restic/repo/ana/$c/snapshots/{} 2>/dev/null || echo MISSING
|
||||
done'
|
||||
|
||||
# --- restic NH3 side ---
|
||||
ssh nh3-nas 'for c in irv-ml1 nh3-docker; do
|
||||
echo -n "$c: "; ls -lt /volume1/Backup/restic/$c/snapshots/ 2>/dev/null | sed -n 2p
|
||||
done'
|
||||
|
||||
# --- rest-server endpoints healthy? (401 = up & serving; Restarting = broken) ---
|
||||
ssh infra-ops@ana-docker 'sudo docker ps --format "{{.Names}}\t{{.Status}}" | grep rest-server'
|
||||
curl -s -o /dev/null -w 'rest-server-ana: %{http_code}\n' http://10.250.50.70:8000/
|
||||
curl -s -o /dev/null -w 'rest-server-nh3: %{http_code}\n' http://10.100.50.50:8000/
|
||||
|
||||
# --- PBS: newest snapshot per guest, all namespaces ---
|
||||
ssh pbs-ana 'for ns in /mnt/pbs-datastore/ns/*/; do nsn=$(basename "$ns")
|
||||
for d in vm ct; do for g in "$ns$d"/*/; do [ -d "$g" ] || continue
|
||||
echo "$nsn/$d/$(basename "$g") -> $(ls "$g" 2>/dev/null | grep ^20 | sort | tail -1)"
|
||||
done; done; done'
|
||||
```
|
||||
|
||||
**Force a backup now (don't wait for 01:00):** on the client host,
|
||||
`ssh infra-ops@<host> 'sudo systemctl start resticprofile-backup@profile-default.service'`
|
||||
(it's an incremental against the existing repo — bounded even if stale).
|
||||
|
||||
---
|
||||
|
||||
## Known failure mode: rest-server-ana crash-loop (the 2026-05-06 → 2026-06-20 outage)
|
||||
|
||||
**Symptom:** `rest-server` container on ana-docker stuck `Restarting`;
|
||||
logs show `cannot load /data/.htpasswd: permission denied`. All ana-side
|
||||
restic backups silently fail.
|
||||
|
||||
**Root cause:** ana-nas's NFS mount on ana-docker uses bare `defaults` in
|
||||
`/etc/fstab` (no `_netdev`, no retry). When the mount drops,
|
||||
`mnt-backup.mount` gets stuck `failed`, so `/mnt/backup/restic/repo/ana`
|
||||
resolves to an **empty local ghost dir** (no `.htpasswd`) and rest-server
|
||||
binds *that*. ana-nas itself is fine — the real repos are intact.
|
||||
|
||||
**Recovery** (needs root on ana-docker — use **`ssh infra-ops@ana-docker`**,
|
||||
which has NOPASSWD sudo; the default `ssh ana-docker` lands as `lkraven`
|
||||
*without* sudo):
|
||||
|
||||
```bash
|
||||
ssh infra-ops@ana-docker '
|
||||
sudo mount -a # re-attach the NFS (bypasses the failed unit)
|
||||
sudo systemctl reset-failed mnt-backup.mount # clear the stuck unit state
|
||||
mount | grep /mnt/backup # confirm nfs4 attached
|
||||
sudo ls /mnt/backup/restic/repo/ana/.htpasswd # real htpasswd now present
|
||||
cd /opt/docker/compose/rest-server-ana && sudo docker compose up -d --force-recreate
|
||||
' # recreate so the bind re-resolves onto NFS
|
||||
# verify: docker ps shows Up (healthy); curl :8000 -> 401; logs say "Loaded htpasswd file"
|
||||
```
|
||||
|
||||
If the ghost dir blocks the mount, see `disaster-recovery.md` Tier-0 for
|
||||
the stop→umount→rm-ghost→remount→start variant.
|
||||
|
||||
---
|
||||
|
||||
## Known gaps / TODO
|
||||
|
||||
- [ ] **No backup-freshness alerting** — the systemic miss; the ana
|
||||
outage went unnoticed ~6.5 weeks. Add a check (e.g. a daily job that
|
||||
alerts if any repo's newest snapshot is > 48 h old, or wire
|
||||
resticprofile's check/`--json` output to a notifier).
|
||||
- [ ] **fstab hardening** — change ana-docker's `/mnt/backup` entry from
|
||||
`defaults` to `_netdev,nofail,x-systemd.automount,x-systemd.mount-timeout=30,retry`
|
||||
so it self-heals on reboot/NAS-blip instead of getting stuck.
|
||||
- [ ] **ana-ml2 has no PBS net** (bare metal) — restic is its only layer.
|
||||
Acceptable IF restic stays healthy + alerted; bulk `/tank` models are
|
||||
re-downloadable, but bespoke quants/configs/scripts are not.
|
||||
- [ ] **Verify esh-pve-nas coverage** — vm-esh-nas + **esh-vm-db (a
|
||||
postgres+mongo DB VM)** have no PBS namespace. Confirm whether they
|
||||
get a PBS image anywhere; if not, restic is their sole net.
|
||||
- [ ] **Rotate rest-server repo passwords** — the 5 per-repo basic-auth
|
||||
creds were exposed during the 2026-06-20 diagnosis (Backrest config
|
||||
dump). Add to the disclosed-keys rotation queue.
|
||||
+25
-11
@@ -98,12 +98,14 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
playbooks run fully non-interactive over key SSH.
|
||||
|
||||
- **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo
|
||||
on most PFI Linux boxes (corviduo-dev included since 2026-06-15) — BUT
|
||||
**on ana-docker the session lands as `lkraven` (docker-group, NO
|
||||
passwordless sudo)**. So on ana-docker: docker commands work directly;
|
||||
root-owned-file edits need the docker-as-root trick or infra-ops; but
|
||||
the litellm config + most stack compose/conf are `lkraven`-owned →
|
||||
plain `cp`/edit, no sudo. (Learned 2026-06-20 — see Tried-and-abandoned.)
|
||||
on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On
|
||||
ana-docker there are TWO identities: the **default `ssh ana-docker` =
|
||||
`lkraven`** (docker-group, NO passwordless sudo — docker works, root-file
|
||||
edits don't); **but `ssh infra-ops@ana-docker` HAS NOPASSWD root** (verified
|
||||
2026-06-20). **→ For any sudo op on ana-docker (mount, root-owned files,
|
||||
service control), use `ssh infra-ops@ana-docker`, NOT the default session.**
|
||||
lkraven-owned files (litellm config, most stack compose/conf) still take
|
||||
plain `cp`/edit under either identity.
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
@@ -190,6 +192,16 @@ _As of 2026-06-20:_
|
||||
**⚠️ Disclosed creds:** diagnosing dumped the Backrest config, exposing the 5
|
||||
per-repo rest-server basic-auth passwords into the session transcript →
|
||||
rotate after recovery (add to the disclosed-keys rotation queue).
|
||||
**✅ RECOVERED (2026-06-20):** via `ssh infra-ops@ana-docker` (NOT the default
|
||||
lkraven session) — `sudo mount -a` + `reset-failed mnt-backup.mount` re-attached
|
||||
the NFS, then `docker compose up -d --force-recreate` rebound rest-server onto
|
||||
the real `/data` (now `Loaded htpasswd file`, 401, healthy). Verified end-to-end:
|
||||
triggered backups on the two no-PBS hosts' priority — **ana-ml2 snapshot landed
|
||||
22:53Z** (total 19→20); ana-docker's larger catch-up was still running at hand-off
|
||||
(will land). esh-docker-vm/esh-vm-db/vm-esh-nas auto-catch-up at their next 01:00
|
||||
timers (or trigger manually). **Full topology + 2-min freshness-check now documented
|
||||
in `docs/runbooks/backups.md`.** STILL OPEN (prevention): backup-freshness alert +
|
||||
fstab `_netdev`/retry hardening + verify esh-pve-nas PBS gap + rotate the 5 creds.
|
||||
|
||||
- **ana-docker disk incident RESOLVED — root cause was a 94 GB unrotated
|
||||
`langfuse-clickhouse` log, NOT (just) image bloat.** ClickHouse logs
|
||||
@@ -377,11 +389,13 @@ _101 older entries archived to archival-memory.md._
|
||||
state). Prune them with `docker buildx prune --builder <name> -af`, or
|
||||
`docker volume rm` orphaned ones (builder already removed).
|
||||
|
||||
- `[2026-06-20]` **On ana-docker I am `lkraven` (no NOPASSWD sudo), not
|
||||
infra-ops.** A `sudo cp` of the litellm config silently failed (password
|
||||
prompt) and I restarted the gateway on an unchanged config (one wasted
|
||||
bounce). The config + most stack files are `lkraven`-owned → plain `cp`,
|
||||
no sudo. Verify the SSH user/ownership before reaching for sudo.
|
||||
- `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) —
|
||||
but `ssh infra-ops@ana-docker` HAS NOPASSWD root** (corrected later same day).
|
||||
Early on a `sudo cp` as lkraven silently failed (password prompt) → one
|
||||
wasted gateway bounce; I then wrongly concluded "no sudo on ana-docker" and
|
||||
nearly punted the rest-server-ana recovery to the operator. The real rule:
|
||||
reach for `infra-ops@ana-docker` for sudo ops; lkraven-owned files (litellm
|
||||
config, most stack compose/conf) take plain `cp` under either identity.
|
||||
|
||||
- `[2026-06-19]` **litellm CONFIG-loaded models can't be hot-removed — `/model/delete` is DB-only.** Only a process restart drops a config model. The INVERSE works: `/model/new` live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user