179 lines
9.2 KiB
Markdown
179 lines
9.2 KiB
Markdown
# Fleet backup architecture & freshness runbook
|
|
|
|
The map that was missing: what backs up what, where it lands, and **how
|
|
to check in 2 minutes whether backups are actually fresh.** Companion to
|
|
[`disaster-recovery.md`](disaster-recovery.md) (which covers *recovery*
|
|
when a host/service is down). Read this one first when the question is
|
|
"are we backed up?"
|
|
|
|
> **Why this exists:** on 2026-06-20 diagnosing "are backups OK?" took a
|
|
> long exploration because the topology lived only in scattered memory.
|
|
> The ana-side restic layer had been failing **silently for ~6.5 weeks**
|
|
> (last good snapshot 2026-05-06) and nobody knew. This doc + a future
|
|
> freshness alert is the fix.
|
|
|
|
---
|
|
|
|
## TL;DR — coverage matrix
|
|
|
|
Two independent layers. **PBS = whole-VM images. restic = granular
|
|
file+DB.** A host is well-covered if it has *either* a current PBS image
|
|
*or* a current restic snapshot; the danger zone is a host whose **only**
|
|
layer has failed.
|
|
|
|
| Host | Kind | PBS (VM image) | restic (file+DB) | Sole net? |
|
|
|---|---|---|---|---|
|
|
| ana-docker | VM (pfi-pve) | ✅ `ana-pve` | ✅ → rest-server-**ana** | no |
|
|
| **ana-ml2** | **bare metal** | ❌ none (not a VM) | ✅ → rest-server-**ana** | ⚠️ **restic is the ONLY net** |
|
|
| **irv-ml1** | **bare metal** | ❌ none (not a VM) | ✅ → rest-server-**nh3** | ⚠️ **restic is the ONLY net** |
|
|
| nh3-docker | VM (nh3-pve) | ✅ `nh3-pve` | ✅ → rest-server-**nh3** | no |
|
|
| esh-docker-vm | VM (esh-pve) | ✅ `esh-pve` | ✅ → rest-server-**ana** | no |
|
|
| esh-vm-db | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only (a DB!)** |
|
|
| vm-esh-nas | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only** |
|
|
| other pfi-pve / nh3-pve VMs/CTs | VM/CT | ✅ respective ns | (PBS only) | no |
|
|
| SureFire `sfsrv-pve` | tenant VMs | ✅ `sfsrv-pve` ns | (PBS only) | no |
|
|
|
|
**Bare-metal hosts have NO PBS coverage** (PBS only backs up Proxmox
|
|
guests). Their restic snapshot is the entire safety net — keep an eye on
|
|
it. ana-ml2 → rest-server-ana; irv-ml1 → rest-server-nh3.
|
|
|
|
---
|
|
|
|
## Layer 1 — PBS (whole-VM/CT images)
|
|
|
|
- **PBS-ANA** (`pbs-ana`, 10.250.50.90) — fleet primary. Datastore is
|
|
NFS-backed: `10.250.50.50:/mnt/backup/pbs-ana` mounted at
|
|
`/mnt/pbs-datastore` (~20 TB). Backs up Proxmox guests via vzdump,
|
|
organised by **namespace per source hypervisor**:
|
|
- `ana-pve` — pfi-pve guests (ana-docker, pfi-postgres VM105, ana-nas
|
|
CT109, webhost, filebot, pteradactyl, tacticalrmm, ana-wg, …)
|
|
- `esh-pve` — esh-pve guests
|
|
- `nh3-pve` — nh3-pve guests
|
|
- `sfsrv-pve` — SureFire tenant
|
|
- ⚠️ there is **no `esh-pve-nas` namespace** — guests on that
|
|
hypervisor (vm-esh-nas, likely esh-vm-db) are **not** PBS-covered.
|
|
- **PBS-NH3** (`pbs-nh3`, 10.100.50.90) — DR mirror; syncs from PBS-ANA
|
|
(datastore on nh3-nas).
|
|
- Schedule: vzdump jobs defined in Proxmox (Datacenter → Backup),
|
|
staggered through the early morning.
|
|
|
|
## Layer 2 — restic (granular file + DB)
|
|
|
|
restic clients push to one of **two rest-server endpoints** (HTTP, basic
|
|
auth, append-only, private repos). The split is by site:
|
|
|
|
| rest-server | Endpoint | Backing store | Clients |
|
|
|---|---|---|---|
|
|
| **rest-server-ana** | `http://10.250.50.70:8000` (container `rest-server` on ana-docker) | `ana-nas:/mnt/backup/restic/repo/ana` (NFS bind → `/data`) | ana-docker, **ana-ml2**, esh-docker-vm, esh-vm-db, vm-esh-nas |
|
|
| **rest-server-nh3** | `http://10.100.50.50:8000` (on nh3-nas) | `nh3-nas:/volume1/Backup/restic/<client>` | **irv-ml1**, nh3-docker |
|
|
|
|
- Per-client repos live as subdirs of the rest-server data dir
|
|
(`.../repo/ana/<client>/` for the ana side); the shared `.htpasswd`
|
|
for ana sits at `.../repo/ana/.htpasswd`.
|
|
- **Scheduler = `resticprofile` systemd timers on each client host**, NOT
|
|
Backrest:
|
|
- `resticprofile-backup@profile-default.timer` — daily **01:00** PDT
|
|
- `resticprofile-check@profile-default.timer` — weekly (Sun **05:00**)
|
|
- **Backrest** (container on ana-docker, UI) is only a **repo viewer here
|
|
— it has 0 plans.** Do not assume "Backrest healthy" means "backups
|
|
running." The timers are the source of truth.
|
|
- ⚠️ **Failures are silent** — a timer fires, restic errors against a
|
|
down endpoint, and nothing alerts. (See Known gaps.)
|
|
|
|
---
|
|
|
|
## The 2-minute freshness check
|
|
|
|
Run these any time you need to answer "are we backed up?"
|
|
|
|
```bash
|
|
# --- restic ANA side: newest snapshot per client (want: today/yesterday) ---
|
|
ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do
|
|
echo -n "$c: "; ls -t /mnt/backup/restic/repo/ana/$c/snapshots/ 2>/dev/null | head -1 \
|
|
| xargs -I{} stat -c "%y" /mnt/backup/restic/repo/ana/$c/snapshots/{} 2>/dev/null || echo MISSING
|
|
done'
|
|
|
|
# --- restic NH3 side ---
|
|
ssh nh3-nas 'for c in irv-ml1 nh3-docker; do
|
|
echo -n "$c: "; ls -lt /volume1/Backup/restic/$c/snapshots/ 2>/dev/null | sed -n 2p
|
|
done'
|
|
|
|
# --- rest-server endpoints healthy? (401 = up & serving; Restarting = broken) ---
|
|
ssh infra-ops@ana-docker 'sudo docker ps --format "{{.Names}}\t{{.Status}}" | grep rest-server'
|
|
curl -s -o /dev/null -w 'rest-server-ana: %{http_code}\n' http://10.250.50.70:8000/
|
|
curl -s -o /dev/null -w 'rest-server-nh3: %{http_code}\n' http://10.100.50.50:8000/
|
|
|
|
# --- PBS: newest snapshot per guest, all namespaces ---
|
|
ssh pbs-ana 'for ns in /mnt/pbs-datastore/ns/*/; do nsn=$(basename "$ns")
|
|
for d in vm ct; do for g in "$ns$d"/*/; do [ -d "$g" ] || continue
|
|
echo "$nsn/$d/$(basename "$g") -> $(ls "$g" 2>/dev/null | grep ^20 | sort | tail -1)"
|
|
done; done; done'
|
|
```
|
|
|
|
**Force a backup now (don't wait for 01:00):** on the client host,
|
|
`ssh infra-ops@<host> 'sudo systemctl start resticprofile-backup@profile-default.service'`
|
|
(it's an incremental against the existing repo — bounded even if stale).
|
|
|
|
---
|
|
|
|
## Known failure mode: rest-server-ana crash-loop (the 2026-05-06 → 2026-06-20 outage)
|
|
|
|
**Symptom:** `rest-server` container on ana-docker stuck `Restarting`;
|
|
logs show `cannot load /data/.htpasswd: permission denied`. All ana-side
|
|
restic backups silently fail.
|
|
|
|
**Root cause:** ana-nas's NFS mount on ana-docker uses bare `defaults` in
|
|
`/etc/fstab` (no `_netdev`, no retry). When the mount drops,
|
|
`mnt-backup.mount` gets stuck `failed`, so `/mnt/backup/restic/repo/ana`
|
|
resolves to an **empty local ghost dir** (no `.htpasswd`) and rest-server
|
|
binds *that*. ana-nas itself is fine — the real repos are intact.
|
|
|
|
**Recovery** (needs root on ana-docker — use **`ssh infra-ops@ana-docker`**,
|
|
which has NOPASSWD sudo; the default `ssh ana-docker` lands as `lkraven`
|
|
*without* sudo):
|
|
|
|
```bash
|
|
ssh infra-ops@ana-docker '
|
|
sudo mount -a # re-attach the NFS (bypasses the failed unit)
|
|
sudo systemctl reset-failed mnt-backup.mount # clear the stuck unit state
|
|
mount | grep /mnt/backup # confirm nfs4 attached
|
|
sudo ls /mnt/backup/restic/repo/ana/.htpasswd # real htpasswd now present
|
|
cd /opt/docker/compose/rest-server-ana && sudo docker compose up -d --force-recreate
|
|
' # recreate so the bind re-resolves onto NFS
|
|
# verify: docker ps shows Up (healthy); curl :8000 -> 401; logs say "Loaded htpasswd file"
|
|
```
|
|
|
|
If the ghost dir blocks the mount, see `disaster-recovery.md` Tier-0 for
|
|
the stop→umount→rm-ghost→remount→start variant.
|
|
|
|
---
|
|
|
|
## Known gaps / TODO
|
|
|
|
- [x] **Backup-freshness alerting — DONE (2026-06-20).**
|
|
`scripts/check-backup-freshness.sh` (the 2-min check, exit 1 on
|
|
stale/down) + a daily **systemd user timer on nh3-dev** at 08:00
|
|
(`scripts/install-backup-freshness-timer.sh`) → `backup-freshness-alert.sh`
|
|
posts an **althing alert to infra-ops** on any stale/down layer. Run the
|
|
check by hand anytime. (Channel is althing for now — swap in email/ntfy if
|
|
you want a louder one.)
|
|
- [x] **fstab hardening — DONE (2026-06-20).** ana-docker `/mnt/backup` →
|
|
`noauto,x-systemd.automount,x-systemd.mount-timeout=30` (autofs self-heals
|
|
on a NAS blip instead of getting stuck `failed`; activates on next reboot).
|
|
`/etc/fstab.bak-pre-harden` saved. **`/mnt/compose` also hardened the same
|
|
way** and **activated live** (umount → `mnt-compose.automount` started →
|
|
autofs verified remounting on access) — it binds no container, so it was
|
|
safe to convert now; this also proved the autofs pattern works on ana-docker.
|
|
- [ ] **ana-ml2 has no PBS net** (bare metal) — restic is its only layer; now
|
|
healthy + alerted. Bulk `/tank` models are re-downloadable; bespoke
|
|
quants/configs/scripts are the real loss-risk.
|
|
- [x] **esh-pve-nas coverage — VERIFIED (2026-06-20): NOT PBS-covered.** No
|
|
`esh-pve-nas` namespace exists on PBS-ANA, so **esh-vm-db (postgres+mongo)
|
|
+ vm-esh-nas are restic-only.** For the DB VM, restic-with-dumps is the
|
|
*preferred* method (vs a VM image) **IF** the resticprofile includes
|
|
`pg_dump`/`mongodump` — confirm that. Optionally add esh-pve-nas as a PBS
|
|
source. ESH is home-lab (no SLA).
|
|
- [ ] **Rotate rest-server repo passwords** — the 5 per-repo basic-auth creds
|
|
were exposed during the 2026-06-20 diagnosis. **BELAYED** — operator
|
|
handling offline.
|