feat(backups): assert restic CONTENT, and add the host that was never checked

Operator ruling 2026-09-22 (relayed via svos-dev): have the freshness check
assert snapshot content rather than REST-server reachability, so the green
light is a statement about DATA rather than about a daemon.

⚠ FIRST, A CORRECTION TO MY OWN REPORT. I ran `grep -ic restic` against
backup-freshness-alert.sh -- the WRAPPER -- got 0, and told the operator that
nothing on the fleet verified restic. Wrong. check-backup-freshness.sh has
always checked restic snapshot ages, for seven hosts. I grepped one file of a
two-file tool and generalised, which is the identical error a peer had just
made with dev-backup, made while correcting them.

THE REAL GAP was narrower and still real: the NH3 host list read
`irv-ml1 nh3-docker` and omitted nh3-dev -- the repo holding every Claude Code
session transcript, althing routes, hermes run history and Miranda's
conversation. /volume1/Backup/restic/nh3-dev/ has always existed and always
been written nightly; it was simply never enumerated, so a total failure of the
one repo nobody could reconstruct would have gone unreported indefinitely.

THE CONTENT ASSERTION, built on svos-dev's ladder (list proves the repo
answers; ls proves the index; check proves structure; RESTORE proves the bytes
come back):
  - CONJUNCTIVE: latest snapshot newer than MAX_AGE_H AND the probe path
    present AND a restore returns non-zero bytes. "A snapshot exists containing
    X" is satisfied by a three-month-old one; "the latest is recent" is
    satisfied by an empty one. Age alone was the old problem; content alone is
    the same problem rotated.
  - IT RESTORES. Metadata is what survives the failures worth fearing -- a
    pruned or partially-written repo can list a path whose blobs are gone.
    ~493KB from ~/.local/state/svos, sub-second. Large paths are not restored:
    a repo that returns one file will return others.
  - IDENTITY: the timer runs as lkraven, which has no NOPASSWD sudo on nh3-dev,
    so the probe hops through infra-ops@localhost. /etc/restic is root-only by
    design.

⚠ restic ls ALWAYS PRINTS A HEADER LINE, matched or not. A path absent from the
repo returns 1 line; a real one returned 6. Counting with `grep -c .` read the
header as a hit, so the ABSENT case fell through and reported "blobs gone" --
telling an operator the repository was corrupt when the truth was a mistyped
path. Now `grep -c '^/'`.

TESTED BY MAKING IT FAIL, because a check only ever seen passing is untested:
bogus probe path -> "absent from snapshot"; BACKUP_MAX_AGE_HOURS=1 -> "12h old
(>1h)"; healthy -> snapshot id, age, entries, bytes restored.

docs/runbooks/backups.md

@@ -177,30 +177,48 @@ the stop→umount→rm-ghost→remount→start variant.
 
 ## Known gaps / TODO
 
-### ⚠ NOTHING VERIFIES THAT RESTIC EVER RUNS (found 2026-09-22, open)
+### ✅ restic content assertion (2026-09-22, CLOSED)
 
-`scripts/backup-freshness-alert.sh` contains **zero** references to restic:
+⚠ **I reported this gap wrongly first.** I ran `grep -ic restic` against
+`backup-freshness-alert.sh` — the **wrapper** — got 0, and told the operator
+nothing on the fleet verified restic. The actual checking lives in
+`check-backup-freshness.sh`, which had always checked restic snapshot ages for
+seven hosts. Grepping one file of a two-file tool and generalising is the same
+error a peer had just made with `dev-backup`, made while correcting them.
 
-```sh
+**The real gap was narrower and still real:** the host list read
-grep -ic restic scripts/backup-freshness-alert.sh   # -> 0
+`irv-ml1 nh3-docker` and **omitted nh3-dev** — the repo holding every Claude
-```
+Code transcript, althing routes, hermes history and Miranda's conversation.
+`/volume1/Backup/restic/nh3-dev/` had always existed and always been written;
+it was simply never enumerated, so a total failure of the one repo nobody could
+reconstruct would have gone unreported indefinitely.
 
-It checks PBS guest snapshot ages, and it pings the rest-servers for liveness
+Both are fixed. nh3-dev is in the list, and a **content assertion** now runs per
-(`rest-server-nh3: up (401)`). **That confirms the server answers. It says
+the operator ruling of 2026-09-22 (assert snapshot content, not REST-server
-nothing about whether a snapshot was ever written to it.**
+reachability):
 
-So if resticprofile stopped, the password file broke, or every run failed, the
+- **Conjunctive**: latest snapshot newer than `MAX_AGE_H` **AND** the probe path
-checker would go on printing `RESULT: all backups fresh` — correctly by its own
+  present **AND** a restore returns non-zero bytes. Age alone was the old
-definition — indefinitely. Restic holds the entire home directory on nh3-dev;
+  check's problem; content alone is the same problem rotated.
-PBS holds the VM images. The layer with the granular data is the unwatched one.
+- **It restores.** Listing proves the *index*; only a restore proves the bytes
+  come back, and metadata is exactly what survives a pruned or partially-written
+  repo. ~493 KB from `~/.local/state/svos`, sub-second. The large paths are not
+  restored: a repo that returns one file will return others, one that cannot is
+  broken for everything.
+- **Identity**: the timer runs as `lkraven` (no NOPASSWD sudo here), so the probe
+  hops through `infra-ops@localhost`, which has it. The credentials under
+  `/etc/restic` are deliberately root-only.
 
-This is the same shape as two other instruments found the same day: a tool that
+⚠ **`restic ls` always prints a header line**, matched or not — a path absent
-enumerates what is fine has selected against its own subject. Here the light is
+from the repo returns 1 line, a real one returned 6. Counting with `grep -c .`
-not merely blind — it is **actively reassuring** about a system it cannot see.
+read the header as a hit and made the ABSENT case fall through to *"blobs
+gone"*, telling an operator the repository was corrupt when the truth was a
+mistyped path. Count `grep -c '^/'`.
 
-Fix when taken: check restic snapshot **age per host**, not endpoint liveness,
+**Tested by making it fail**, because a check only ever seen passing is
-and carry an expected-paths list so "covered by a job that has not run in a
+untested: a bogus probe path reports *absent*; `BACKUP_MAX_AGE_HOURS=1` reports
-month" stops rendering identically to "covered".
+*too old*; the healthy path reports the snapshot id, age, entry count and bytes
+restored.
 
 
 

scripts/check-backup-freshness.sh

@@ -133,7 +133,14 @@ for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do
   report "restic/ana/$c" "$(newest_epoch ana-nas "/mnt/backup/restic/repo/ana/$c/snapshots/*")"
 done
 # --- Layer: restic file+DB, NH3 side (rest-server-nh3) ---
-for c in irv-ml1 nh3-docker; do
+# ⚠ nh3-dev WAS MISSING FROM THIS LIST until 2026-09-22, and it is the host
+# whose restic repo holds the most irreplaceable data on the fleet: every
+# Claude Code session transcript, althing routes, hermes run history, and
+# Miranda's conversation. Its repo has always existed at
+# /volume1/Backup/restic/nh3-dev/ and has always been written nightly -- it was
+# simply never enumerated here, so a total failure of the one repo nobody could
+# reconstruct would have gone unreported indefinitely.
+for c in irv-ml1 nh3-docker nh3-dev; do
   report "restic/nh3/$c" "$(newest_epoch nh3-nas "/volume1/Backup/restic/$c/snapshots/*")"
 done
 # --- Layer: PBS VM images (newest per guest, all namespaces) ---
@@ -192,6 +199,76 @@ for entry in "${PVE_NODES[@]}"; do
   done <<<"$bad"
 done
 
+# --- restic CONTENT assertion (operator ruling 2026-09-22) -------------------
+#
+# The checks above read a snapshot directory's MTIME on the NAS. That proves a
+# file was written recently. It does not prove the repository holds the data,
+# and metadata is exactly what survives the failures worth fearing: a pruned or
+# partially-written repo can still list a path whose blobs are gone.
+#
+# ⚠ THE ASSERTION IS CONJUNCTIVE ON PURPOSE. "a snapshot exists containing X"
+# is satisfied by a three-month-old snapshot; "the latest snapshot is recent" is
+# satisfied by an empty one. Age alone was the old check's problem; content
+# alone is the same problem rotated. Both, or neither is worth printing.
+#
+# The rungs, and what each actually proves (svos-dev, 2026-09-22):
+#     snapshots list    the repo answers
+#     ls a path in it   the path is in the INDEX  <- metadata only
+#     restic check      the repo's structure is intact
+#     RESTORE a file    the bytes come back       <- the only proof of recovery
+# We take the top rung for the small irreplaceable set, because it is the only
+# one that answers the question a green light is taken to mean. Restoring from
+# the multi-gigabyte paths is not worth it: a repo that returns one file is
+# overwhelmingly likely to return others, and one that cannot is broken for
+# everything.
+#
+# ⚠ IDENTITY: this runs as a --user timer (lkraven), which has no NOPASSWD sudo
+# on nh3-dev, and the repo credentials live under /etc/restic. So the probe hops
+# through infra-ops@localhost, which does. Not a workaround -- infra-ops is the
+# documented ops identity and the credentials are deliberately root-only.
+RESTIC_PROBE_PATHS="${RESTIC_PROBE_PATHS:-/home/lkraven/.local/state/svos}"
+probe=$($SSH infra-ops@10.100.10.50 "sudo -n bash -s" <<PROBE 2>/dev/null
+set -a; . /etc/restic/restic.env 2>/dev/null; set +a
+export RESTIC_PASSWORD_FILE=/etc/restic/password
+id=\$(restic snapshots --latest 1 --json 2>/dev/null | python3 -c 'import json,sys
+d=json.load(sys.stdin)
+print(d[0]["short_id"], int(__import__("datetime").datetime.fromisoformat(d[0]["time"][:19]).timestamp())) if d else print("NONE 0")' 2>/dev/null)
+sid=\$(echo "\$id" | cut -d" " -f1); ep=\$(echo "\$id" | cut -d" " -f2)
+[ "\$sid" = "NONE" ] && { echo "NOSNAP 0 0 0"; exit 0; }
+# ⚠ grep -c '^/' NOT grep -c '.' — \`restic ls\` always prints a HEADER line
+# ("snapshot <id> of [...] filtered by [...]") whether or not anything matched,
+# so a path absent from the repo returns 1 line and a naive count reads it as
+# found. Measured 2026-09-22: real path 6 lines, bogus path 1. That made the
+# ABSENT case fall through and report "blobs gone" — telling an operator the
+# repository was corrupt when the truth was a mistyped path. Only lines that
+# begin with / are entries.
+n=\$(restic ls "\$sid" $RESTIC_PROBE_PATHS 2>/dev/null | grep -c '^/')
+tmp=\$(mktemp -d); restic restore "\$sid" --target "\$tmp" --include $RESTIC_PROBE_PATHS >/dev/null 2>&1
+bytes=\$(find "\$tmp" -type f -printf '%s\n' 2>/dev/null | awk '{s+=\$1} END {print s+0}')
+rm -rf -- "\$tmp"
+echo "\$sid \$ep \$n \$bytes"
+PROBE
+)
+set -- ${probe:-ERR 0 0 0}
+r_sid="$1"; r_ep="$2"; r_entries="$3"; r_bytes="$4"
+if [ "$r_sid" = "ERR" ] || [ -z "$probe" ]; then
+  stale+=("restic/nh3-dev CONTENT: probe could not run (repo unreachable or creds unreadable)")
+elif [ "$r_sid" = "NOSNAP" ]; then
+  stale+=("restic/nh3-dev CONTENT: repository holds NO SNAPSHOTS")
+else
+  r_age=$(( (now - r_ep) / 3600 ))
+  # conjunctive: recent AND indexed AND restorable
+  if [ "$r_age" -gt "$MAX_AGE_H" ]; then
+    stale+=("restic/nh3-dev CONTENT: latest snapshot $r_sid is ${r_age}h old (>${MAX_AGE_H}h)")
+  elif [ "${r_entries:-0}" -lt 1 ]; then
+    stale+=("restic/nh3-dev CONTENT: $RESTIC_PROBE_PATHS absent from snapshot $r_sid")
+  elif [ "${r_bytes:-0}" -lt 1 ]; then
+    stale+=("restic/nh3-dev CONTENT: $RESTIC_PROBE_PATHS listed in $r_sid but RESTORED ZERO BYTES — index intact, blobs gone")
+  else
+    fresh+=("restic/nh3-dev CONTENT: $r_sid ${r_age}h old, $r_entries entries, restored ${r_bytes}B")
+  fi
+fi
+
 # --- rest-server endpoint health (401 = up & serving) ---
 for ep in "rest-server-ana http://10.250.50.70:8000/" "rest-server-nh3 http://10.100.50.50:8000/"; do
   set -- $ep