esh-pfi-infrastructure/docs/runbooks/backups.md

# Fleet backup architecture & freshness runbook

The map that was missing: what backs up what, where it lands, and **how
to check in 2 minutes whether backups are actually fresh.** Companion to
[`disaster-recovery.md`](disaster-recovery.md) (which covers *recovery*
when a host/service is down). Read this one first when the question is
"are we backed up?"

> **Why this exists:** on 2026-06-20 diagnosing "are backups OK?" took a
> long exploration because the topology lived only in scattered memory.
> The ana-side restic layer had been failing **silently for ~6.5 weeks**
> (last good snapshot 2026-05-06) and nobody knew. This doc + a future
> freshness alert is the fix.

---

## TL;DR — coverage matrix

Two independent layers. **PBS = whole-VM images. restic = granular
file+DB.** A host is well-covered if it has *either* a current PBS image
*or* a current restic snapshot; the danger zone is a host whose **only**
layer has failed.

| Host | Kind | PBS (VM image) | restic (file+DB) | Sole net? |
|---|---|---|---|---|
| ana-docker | VM (pfi-pve) | ✅ `ana-pve` | ✅ → rest-server-**ana** | no |
| **ana-ml2** | **bare metal** | ❌ none (not a VM) | ✅ → rest-server-**ana** | ⚠️ **restic is the ONLY net** |
| **irv-ml1** | **bare metal** | ❌ none (not a VM) | ✅ → rest-server-**nh3** | ⚠️ **restic is the ONLY net** |
| nh3-docker | VM (nh3-pve) | ✅ `nh3-pve` | ✅ → rest-server-**nh3** | no |
| esh-docker-vm | VM (esh-pve) | ✅ `esh-pve` | ✅ → rest-server-**ana** | no |
| esh-vm-db | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only (a DB!)** |
| vm-esh-nas | VM (esh-pve-nas) | ❌ **none** (esh-pve-nas not a PBS source) | ✅ → rest-server-**ana** | ⚠️ **restic-only** |
| other pfi-pve / nh3-pve VMs/CTs | VM/CT | ✅ respective ns | (PBS only) | no |
| SureFire `sfsrv-pve` | tenant VMs | ✅ `sfsrv-pve` ns | (PBS only) | no |

**Bare-metal hosts have NO PBS coverage** (PBS only backs up Proxmox
guests). Their restic snapshot is the entire safety net — keep an eye on
it. ana-ml2 → rest-server-ana; irv-ml1 → rest-server-nh3.

---

## Layer 1 — PBS (whole-VM/CT images)

- **PBS-ANA** (`pbs-ana`, 10.250.50.90) — fleet primary. Datastore is
  NFS-backed: `10.250.50.50:/mnt/backup/pbs-ana` mounted at
  `/mnt/pbs-datastore` (~20 TB). Backs up Proxmox guests via vzdump,
  organised by **namespace per source hypervisor**:
  - `ana-pve` — pfi-pve guests (ana-docker, pfi-postgres VM105, ana-nas
    CT109, webhost, filebot, pteradactyl, tacticalrmm, ana-wg, …)
  - `esh-pve` — esh-pve guests
  - `nh3-pve` — nh3-pve guests
  - `sfsrv-pve` — SureFire tenant
  - ⚠️ there is **no `esh-pve-nas` namespace** — guests on that
    hypervisor (vm-esh-nas, likely esh-vm-db) are **not** PBS-covered.
- **PBS-NH3** (`pbs-nh3`, 10.100.50.90) — DR mirror; syncs from PBS-ANA
  (datastore on nh3-nas).
- Schedule: vzdump jobs defined in Proxmox (Datacenter → Backup),
  staggered through the early morning.

## Layer 2 — restic (granular file + DB)

restic clients push to one of **two rest-server endpoints** (HTTP, basic
auth, append-only, private repos). The split is by site:

| rest-server | Endpoint | Backing store | Clients |
|---|---|---|---|
| **rest-server-ana** | `http://10.250.50.70:8000` (container `rest-server` on ana-docker) | `ana-nas:/mnt/backup/restic/repo/ana` (NFS bind → `/data`) | ana-docker, **ana-ml2**, esh-docker-vm, esh-vm-db, vm-esh-nas |
| **rest-server-nh3** | `http://10.100.50.50:8000` (on nh3-nas) | `nh3-nas:/volume1/Backup/restic/<client>` | **irv-ml1**, nh3-docker |

- Per-client repos live as subdirs of the rest-server data dir
  (`.../repo/ana/<client>/` for the ana side); the shared `.htpasswd`
  for ana sits at `.../repo/ana/.htpasswd`.
- **Scheduler = `resticprofile` systemd timers on each client host**, NOT
  Backrest:
  - `resticprofile-backup@profile-default.timer` — daily **01:00** PDT
  - `resticprofile-check@profile-default.timer` — weekly (Sun **05:00**)
- **Backrest** (container on ana-docker, UI) is only a **repo viewer here
  — it has 0 plans.** Do not assume "Backrest healthy" means "backups
  running." The timers are the source of truth.
- ⚠️ **Failures are silent** — a timer fires, restic errors against a
  down endpoint, and nothing alerts. (See Known gaps.)

---

## ⚠ Answering "is X backed up?" — ask the repo, never the job

**Coverage is a property of the backup SYSTEM, not of any one job's configured
scope.** Establish it by querying the repository for the path and seeing it in a
real snapshot. Never by reading a job's `SRC=`.

```sh
# as infra-ops on the host, against its own profile
sudo -n bash -c 'set -a; . /etc/restic/restic.env; set +a
  export RESTIC_PASSWORD_FILE=/etc/restic/password
  id=$(restic snapshots --latest 1 --json | jq -r ".[0].short_id")
  restic ls "$id" /path/in/question | head'
```

**The failure this rule exists to prevent (2026-09-22).** A peer agent checked
`dev-backup.sh`, found `SRC="$HOME/development/"`, and reported — with
specifics, unhedged, to the operator — that five home-directory paths including
Miranda's entire conversation had *"never been backed up anywhere"*. All five
were in that night's restic snapshot. `dev-backup` is the **hourly** job for one
directory; **resticprofile** is the daily job covering all of `/home/lkraven`,
and it is what actually protects home. Checking one job and generalising to the
system produced a confident, false, escalated claim.

The true finding underneath was much smaller and is a **cadence** question, not
a coverage one: live conversation state has a 24-hour RPO because it rides the
daily job rather than the hourly one.

## The 2-minute freshness check

Run these any time you need to answer "are we backed up?"

```bash
# --- restic ANA side: newest snapshot per client (want: today/yesterday) ---
ssh ana-nas 'for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do
  echo -n "$c: "; ls -t /mnt/backup/restic/repo/ana/$c/snapshots/ 2>/dev/null | head -1 \
    | xargs -I{} stat -c "%y" /mnt/backup/restic/repo/ana/$c/snapshots/{} 2>/dev/null || echo MISSING
done'

# --- restic NH3 side ---
ssh nh3-nas 'for c in irv-ml1 nh3-docker; do
  echo -n "$c: "; ls -lt /volume1/Backup/restic/$c/snapshots/ 2>/dev/null | sed -n 2p
done'

# --- rest-server endpoints healthy? (401 = up & serving; Restarting = broken) ---
ssh infra-ops@ana-docker 'sudo docker ps --format "{{.Names}}\t{{.Status}}" | grep rest-server'
curl -s -o /dev/null -w 'rest-server-ana: %{http_code}\n' http://10.250.50.70:8000/
curl -s -o /dev/null -w 'rest-server-nh3: %{http_code}\n' http://10.100.50.50:8000/

# --- PBS: newest snapshot per guest, all namespaces ---
ssh pbs-ana 'for ns in /mnt/pbs-datastore/ns/*/; do nsn=$(basename "$ns")
  for d in vm ct; do for g in "$ns$d"/*/; do [ -d "$g" ] || continue
    echo "$nsn/$d/$(basename "$g") -> $(ls "$g" 2>/dev/null | grep ^20 | sort | tail -1)"
  done; done; done'
```

**Force a backup now (don't wait for 01:00):** on the client host,
`ssh infra-ops@<host> 'sudo systemctl start resticprofile-backup@profile-default.service'`
(it's an incremental against the existing repo — bounded even if stale).

---

## Known failure mode: rest-server-ana crash-loop (the 2026-05-06 → 2026-06-20 outage)

**Symptom:** `rest-server` container on ana-docker stuck `Restarting`;
logs show `cannot load /data/.htpasswd: permission denied`. All ana-side
restic backups silently fail.

**Root cause:** ana-nas's NFS mount on ana-docker uses bare `defaults` in
`/etc/fstab` (no `_netdev`, no retry). When the mount drops,
`mnt-backup.mount` gets stuck `failed`, so `/mnt/backup/restic/repo/ana`
resolves to an **empty local ghost dir** (no `.htpasswd`) and rest-server
binds *that*. ana-nas itself is fine — the real repos are intact.

**Recovery** (needs root on ana-docker — use **`ssh infra-ops@ana-docker`**,
which has NOPASSWD sudo; the default `ssh ana-docker` lands as `lkraven`
*without* sudo):

```bash
ssh infra-ops@ana-docker '
  sudo mount -a                                   # re-attach the NFS (bypasses the failed unit)
  sudo systemctl reset-failed mnt-backup.mount    # clear the stuck unit state
  mount | grep /mnt/backup                         # confirm nfs4 attached
  sudo ls /mnt/backup/restic/repo/ana/.htpasswd    # real htpasswd now present
  cd /opt/docker/compose/rest-server-ana && sudo docker compose up -d --force-recreate
'                                                  # recreate so the bind re-resolves onto NFS
# verify: docker ps shows Up (healthy); curl :8000 -> 401; logs say "Loaded htpasswd file"
```

If the ghost dir blocks the mount, see `disaster-recovery.md` Tier-0 for
the stop→umount→rm-ghost→remount→start variant.

---

## Known gaps / TODO

### ✅ restic content assertion (2026-09-22, CLOSED)

⚠ **I reported this gap wrongly first.** I ran `grep -ic restic` against
`backup-freshness-alert.sh` — the **wrapper** — got 0, and told the operator
nothing on the fleet verified restic. The actual checking lives in
`check-backup-freshness.sh`, which had always checked restic snapshot ages for
seven hosts. Grepping one file of a two-file tool and generalising is the same
error a peer had just made with `dev-backup`, made while correcting them.

**The real gap was narrower and still real:** the host list read
`irv-ml1 nh3-docker` and **omitted nh3-dev** — the repo holding every Claude
Code transcript, althing routes, hermes history and Miranda's conversation.
`/volume1/Backup/restic/nh3-dev/` had always existed and always been written;
it was simply never enumerated, so a total failure of the one repo nobody could
reconstruct would have gone unreported indefinitely.

Both are fixed. nh3-dev is in the list, and a **content assertion** now runs per
the operator ruling of 2026-09-22 (assert snapshot content, not REST-server
reachability):

- **Conjunctive**: latest snapshot newer than `MAX_AGE_H` **AND** the probe path
  present **AND** a restore returns non-zero bytes. Age alone was the old
  check's problem; content alone is the same problem rotated.
- **It restores.** Listing proves the *index*; only a restore proves the bytes
  come back, and metadata is exactly what survives a pruned or partially-written
  repo. ~493 KB from `~/.local/state/svos`, sub-second. The large paths are not
  restored: a repo that returns one file will return others, one that cannot is
  broken for everything.
- **Identity**: the timer runs as `lkraven` (no NOPASSWD sudo here), so the probe
  hops through `infra-ops@localhost`, which has it. The credentials under
  `/etc/restic` are deliberately root-only.

### Repos are DISCOVERED, not enumerated

The host list is no longer hand-written. `check-backup-freshness.sh` asks each
NAS which repositories exist — a directory counts as one when it has a
`snapshots/` child, which distinguishes a real repo from a container dir
(`/volume1/Backup/restic/repo/` holds `ana|esh|nh3` namespaces and no snapshots
of its own).

**Why (svos-dev's framing, 2026-09-22):** adding nh3-dev fixed the *instance*;
the *class* was that a hand-maintained list of things to watch, sitting beside a
NAS that already knows which repos exist, means **the next repo added is
unwatched by default and nothing says so.** The list of what to check can
silently disagree with the set of what exists.

The hand-written list survives **demoted**, as `EXPECTED_RESTIC_ANA` /
`EXPECTED_RESTIC_NH3`, used only to report a repo that has *vanished*. That
keeps two facts distinct that would otherwise both read as silence:

| fact | now |
|---|---|
| a repo exists that nobody watches | **impossible** — it is discovered |
| a repo we expected is gone | reported as `EXPECTED REPO NOT FOUND` |

Currently 8 repos: 5 on ana-nas (`repo/ana/*`), 3 on nh3-nas. The `esh` and
`nh3` namespaces under `repo/` on ana-nas are empty.

⚠ **`restic ls` always prints a header line**, matched or not — a path absent
from the repo returns 1 line, a real one returned 6. Counting with `grep -c .`
read the header as a hit and made the ABSENT case fall through to *"blobs
gone"*, telling an operator the repository was corrupt when the truth was a
mistyped path. Count `grep -c '^/'`.

**Tested by making it fail**, because a check only ever seen passing is
untested: a bogus probe path reports *absent*; `BACKUP_MAX_AGE_HOURS=1` reports
*too old*; the healthy path reports the snapshot id, age, entry count and bytes
restored.



- [x] **Backup-freshness alerting — DONE (2026-06-20).**
      `scripts/check-backup-freshness.sh` (the 2-min check, exit 1 on
      stale/down) + a daily **systemd user timer on nh3-dev** at 08:00
      (`scripts/install-backup-freshness-timer.sh`) → `backup-freshness-alert.sh`
      posts an **althing alert to infra-ops** on any stale/down layer. Run the
      check by hand anytime. (Channel is althing for now — swap in email/ntfy if
      you want a louder one.)
- [x] **fstab hardening — DONE (2026-06-20).** ana-docker `/mnt/backup` →
      `noauto,x-systemd.automount,x-systemd.mount-timeout=30` (autofs self-heals
      on a NAS blip instead of getting stuck `failed`; activates on next reboot).
      `/etc/fstab.bak-pre-harden` saved. **`/mnt/compose` also hardened the same
      way** and **activated live** (umount → `mnt-compose.automount` started →
      autofs verified remounting on access) — it binds no container, so it was
      safe to convert now; this also proved the autofs pattern works on ana-docker.
- [ ] **ana-ml2 has no PBS net** (bare metal) — restic is its only layer; now
      healthy + alerted. Bulk `/tank` models are re-downloadable; bespoke
      quants/configs/scripts are the real loss-risk.
- [x] **esh-pve-nas coverage — VERIFIED (2026-06-20): NOT PBS-covered.** No
      `esh-pve-nas` namespace exists on PBS-ANA, so **esh-vm-db (postgres+mongo)
      + vm-esh-nas are restic-only.** For the DB VM, restic-with-dumps is the
      *preferred* method (vs a VM image) **IF** the resticprofile includes
      `pg_dump`/`mongodump` — confirm that. Optionally add esh-pve-nas as a PBS
      source. ESH is home-lab (no SLA).
- [ ] **Rotate rest-server repo passwords** — the 5 per-repo basic-auth creds
      were exposed during the 2026-06-20 diagnosis. **BELAYED** — operator
      handling offline.