esh-pfi-infrastructure/docs/runbooks/backups.md

# Fleet backup architecture & freshness runbook

The map that was missing: what backs up what, where it lands, and **how
to check in 2 minutes whether backups are actually fresh.** Companion to
[`disaster-recovery.md`](disaster-recovery.md) (which covers *recovery*
when a host/service is down). Read this one first when the question is
"are we backed up?"

> **Why this exists:** on 2026-06-20 diagnosing "are backups OK?" took a
> long exploration because the topology lived only in scattered memory.
> The ana-side restic layer had been failing **silently for ~6.5 weeks**
> (last good snapshot 2026-05-06) and nobody knew. This doc + a future
> freshness alert is the fix.

---

## TL;DR — coverage matrix

Two independent layers. **PBS = whole-VM images. restic = granular
file+DB.** A host is well-covered if it has *either* a current PBS image
*or* a current restic snapshot; the danger zone is a host whose **only**
layer has failed.

| Host | Kind | PBS (VM image) | restic (file+DB) | Sole net? |
|---|---|---|---|---|
| ana-docker | VM (pfi-pve) | ✅ `ana-pve` | ✅ → rest-server-**ana** | no |
| **ana-ml2** | **bare metal** | ❌ none (not a VM) | ✅ → rest-server-**ana** | ⚠️ **restic is the ONLY net** |
| **irv-ml1** | **bare metal** | ❌ none (not a VM) | ✅ → rest-server-**nh3** | ⚠️ **restic is the ONLY net** |
| nh3-docker | VM (nh3-pve) | ✅ `nh3-pve` | ✅ → rest-server-**nh3** | no |
| esh-docker-vm | VM (esh-pve) | ✅ `esh-pve` | ✅ → rest-server-**ana** | no |
| esh-vm-db | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only (a DB!)** |
| vm-esh-nas | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only** |
| other pfi-pve / nh3-pve VMs/CTs | VM/CT | ✅ respective ns | (PBS only) | no |
| SureFire `sfsrv-pve` | tenant VMs | ✅ `sfsrv-pve` ns | (PBS only) | no |

**Bare-metal hosts have NO PBS coverage** (PBS only backs up Proxmox
guests). Their restic snapshot is the entire safety net — keep an eye on
it. ana-ml2 → rest-server-ana; irv-ml1 → rest-server-nh3.

---

## Layer 1 — PBS (whole-VM/CT images)

- **PBS-ANA** (`pbs-ana`, 10.250.50.90) — fleet primary. Datastore is
  NFS-backed: `10.250.50.50:/mnt/backup/pbs-ana` mounted at
  `/mnt/pbs-datastore` (~20 TB). Backs up Proxmox guests via vzdump,
  organised by **namespace per source hypervisor**:
  - `ana-pve` — pfi-pve guests (ana-docker, pfi-postgres VM105, ana-nas
    CT109, webhost, filebot, pteradactyl, tacticalrmm, ana-wg, …)
  - `esh-pve` — esh-pve guests
  - `nh3-pve` — nh3-pve guests
  - `sfsrv-pve` — SureFire tenant
  - ⚠️ there is **no `esh-pve-nas` namespace** — guests on that
    hypervisor (vm-esh-nas, likely esh-vm-db) are **not** PBS-covered.
- **PBS-NH3** (`pbs-nh3`, 10.100.50.90) — DR mirror; syncs from PBS-ANA
  (datastore on nh3-nas).
- Schedule: vzdump jobs defined in Proxmox (Datacenter → Backup),
  staggered through the early morning.

## Layer 2 — restic (granular file + DB)

restic clients push to one of **two rest-server endpoints** (HTTP, basic
auth, append-only, private repos). The split is by site:

| rest-server | Endpoint | Backing store | Clients |
|---|---|---|---|
| **rest-server-ana** | `http://10.250.50.70:8000` (container `rest-server` on ana-docker) | `ana-nas:/mnt/backup/restic/repo/ana` (NFS bind → `/data`) | ana-docker, **ana-ml2**, esh-docker-vm, esh-vm-db, vm-esh-nas |
| **rest-server-nh3** | `http://10.100.50.50:8000` (on nh3-nas) | `nh3-nas:/volume1/Backup/restic/<client>` | **irv-ml1**, nh3-docker |

- Per-client repos live as subdirs of the rest-server data dir
  (`.../repo/ana/<client>/` for the ana side); the shared `.htpasswd`
  for ana sits at `.../repo/ana/.htpasswd`.
- **Scheduler = `resticprofile` systemd timers on each client host**, NOT
  Backrest:
  - `resticprofile-backup@profile-default.timer` — daily **01:00** PDT
  - `resticprofile-check@profile-default.timer` — weekly (Sun **05:00**)
- **Backrest** (container on ana-docker, UI) is only a **repo viewer here
  — it has 0 plans.** Do not assume "Backrest healthy" means "backups
  running." The timers are the source of truth.
- ⚠️ **Failures are silent** — a timer fires, restic errors against a
  down endpoint, and nothing alerts. (See Known gaps.)

---

## ⚠ Answering "is X backed up?" — ask the repo, never the job

**Coverage is a property of the backup SYSTEM, not of any one job's configured
scope.** Establish it by querying the repository for the path and seeing it in a
real snapshot. Never by reading a job's `SRC=`.

```sh
# as infra-ops on the host, against its own profile
sudo -n bash -c 'set -a; . /etc/restic/restic.env; set +a
  export RESTIC_PASSWORD_FILE=/etc/restic/password
  id=$(restic snapshots --latest 1 --json | jq -r ".[0].short_id")
  restic ls "$id" /path/in/question | head'
```

**The failure this rule exists to prevent (2026-09-22).** A peer agent checked
`dev-backup.sh`, found `SRC="$HOME/development/"`, and reported — with
specifics, unhedged, to the operator — that five home-directory paths including
Miranda's entire conversation had *"never been backed up anywhere"*. All five
were in that night's restic snapshot. `dev-backup` is the **hourly** job for one
directory; **resticprofile** is the daily job covering all of `/home/lkraven`,
and it is what actually protects home. Checking one job and generalising to the
system produced a confident, false, escalated claim.

The true finding underneath was much smaller and is a **cadence** question, not
a coverage one: live conversation state has a 24-hour RPO because it rides the
daily job rather than the hourly one.

## The 2-minute freshness check

Run these any time you need to answer "are we backed up?"

```bash
# --- restic ANA side: newest snapshot per client (want: today/yesterday) ---
ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do
  echo -n "$c: "; ls -t /mnt/backup/restic/repo/ana/$c/snapshots/ 2>/dev/null | head -1 \
    | xargs -I{} stat -c "%y" /mnt/backup/restic/repo/ana/$c/snapshots/{} 2>/dev/null || echo MISSING
done'

# --- restic NH3 side ---
ssh nh3-nas 'for c in irv-ml1 nh3-docker; do
  echo -n "$c: "; ls -lt /volume1/Backup/restic/$c/snapshots/ 2>/dev/null | sed -n 2p
done'

# --- rest-server endpoints healthy? (401 = up & serving; Restarting = broken) ---
ssh infra-ops@ana-docker 'sudo docker ps --format "{{.Names}}\t{{.Status}}" | grep rest-server'
curl -s -o /dev/null -w 'rest-server-ana: %{http_code}\n' http://10.250.50.70:8000/
curl -s -o /dev/null -w 'rest-server-nh3: %{http_code}\n' http://10.100.50.50:8000/

# --- PBS: newest snapshot per guest, all namespaces ---
ssh pbs-ana 'for ns in /mnt/pbs-datastore/ns/*/; do nsn=$(basename "$ns")
  for d in vm ct; do for g in "$ns$d"/*/; do [ -d "$g" ] || continue
    echo "$nsn/$d/$(basename "$g") -> $(ls "$g" 2>/dev/null | grep ^20 | sort | tail -1)"
  done; done; done'
```

**Force a backup now (don't wait for 01:00):** on the client host,
`ssh infra-ops@<host> 'sudo systemctl start resticprofile-backup@profile-default.service'`
(it's an incremental against the existing repo — bounded even if stale).

---

## Known failure mode: rest-server-ana crash-loop (the 2026-05-06 → 2026-06-20 outage)

**Symptom:** `rest-server` container on ana-docker stuck `Restarting`;
logs show `cannot load /data/.htpasswd: permission denied`. All ana-side
restic backups silently fail.

**Root cause:** ana-nas's NFS mount on ana-docker uses bare `defaults` in
`/etc/fstab` (no `_netdev`, no retry). When the mount drops,
`mnt-backup.mount` gets stuck `failed`, so `/mnt/backup/restic/repo/ana`
resolves to an **empty local ghost dir** (no `.htpasswd`) and rest-server
binds *that*. ana-nas itself is fine — the real repos are intact.

**Recovery** (needs root on ana-docker — use **`ssh infra-ops@ana-docker`**,
which has NOPASSWD sudo; the default `ssh ana-docker` lands as `lkraven`
*without* sudo):

```bash
ssh infra-ops@ana-docker '
  sudo mount -a                                   # re-attach the NFS (bypasses the failed unit)
  sudo systemctl reset-failed mnt-backup.mount    # clear the stuck unit state
  mount | grep /mnt/backup                         # confirm nfs4 attached
  sudo ls /mnt/backup/restic/repo/ana/.htpasswd    # real htpasswd now present
  cd /opt/docker/compose/rest-server-ana && sudo docker compose up -d --force-recreate
'                                                  # recreate so the bind re-resolves onto NFS
# verify: docker ps shows Up (healthy); curl :8000 -> 401; logs say "Loaded htpasswd file"
```

If the ghost dir blocks the mount, see `disaster-recovery.md` Tier-0 for
the stop→umount→rm-ghost→remount→start variant.

---

## Known gaps / TODO

### ⚠ NOTHING VERIFIES THAT RESTIC EVER RUNS (found 2026-09-22, open)

`scripts/backup-freshness-alert.sh` contains **zero** references to restic:

```sh
grep -ic restic scripts/backup-freshness-alert.sh   # -> 0
```

It checks PBS guest snapshot ages, and it pings the rest-servers for liveness
(`rest-server-nh3: up (401)`). **That confirms the server answers. It says
nothing about whether a snapshot was ever written to it.**

So if resticprofile stopped, the password file broke, or every run failed, the
checker would go on printing `RESULT: all backups fresh` — correctly by its own
definition — indefinitely. Restic holds the entire home directory on nh3-dev;
PBS holds the VM images. The layer with the granular data is the unwatched one.

This is the same shape as two other instruments found the same day: a tool that
enumerates what is fine has selected against its own subject. Here the light is
not merely blind — it is **actively reassuring** about a system it cannot see.

Fix when taken: check restic snapshot **age per host**, not endpoint liveness,
and carry an expected-paths list so "covered by a job that has not run in a
month" stops rendering identically to "covered".



- [x] **Backup-freshness alerting — DONE (2026-06-20).**
      `scripts/check-backup-freshness.sh` (the 2-min check, exit 1 on
      stale/down) + a daily **systemd user timer on nh3-dev** at 08:00
      (`scripts/install-backup-freshness-timer.sh`) → `backup-freshness-alert.sh`
      posts an **althing alert to infra-ops** on any stale/down layer. Run the
      check by hand anytime. (Channel is althing for now — swap in email/ntfy if
      you want a louder one.)
- [x] **fstab hardening — DONE (2026-06-20).** ana-docker `/mnt/backup` →
      `noauto,x-systemd.automount,x-systemd.mount-timeout=30` (autofs self-heals
      on a NAS blip instead of getting stuck `failed`; activates on next reboot).
      `/etc/fstab.bak-pre-harden` saved. **`/mnt/compose` also hardened the same
      way** and **activated live** (umount → `mnt-compose.automount` started →
      autofs verified remounting on access) — it binds no container, so it was
      safe to convert now; this also proved the autofs pattern works on ana-docker.
- [ ] **ana-ml2 has no PBS net** (bare metal) — restic is its only layer; now
      healthy + alerted. Bulk `/tank` models are re-downloadable; bespoke
      quants/configs/scripts are the real loss-risk.
- [x] **esh-pve-nas coverage — VERIFIED (2026-06-20): NOT PBS-covered.** No
      `esh-pve-nas` namespace exists on PBS-ANA, so **esh-vm-db (postgres+mongo)
      + vm-esh-nas are restic-only.** For the DB VM, restic-with-dumps is the
      *preferred* method (vs a VM image) **IF** the resticprofile includes
      `pg_dump`/`mongodump` — confirm that. Optionally add esh-pve-nas as a PBS
      source. ESH is home-lab (no SLA).
- [ ] **Rotate rest-server repo passwords** — the 5 per-repo basic-auth creds
      were exposed during the 2026-06-20 diagnosis. **BELAYED** — operator
      handling offline.