From ba60fda16c86fd3f30698a7acb448f01fe9d66d7 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 22 Sep 2026 13:33:58 -0700 Subject: [PATCH] feat(backups): assert restic CONTENT, and add the host that was never checked MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator ruling 2026-09-22 (relayed via svos-dev): have the freshness check assert snapshot content rather than REST-server reachability, so the green light is a statement about DATA rather than about a daemon. ⚠ FIRST, A CORRECTION TO MY OWN REPORT. I ran `grep -ic restic` against backup-freshness-alert.sh -- the WRAPPER -- got 0, and told the operator that nothing on the fleet verified restic. Wrong. check-backup-freshness.sh has always checked restic snapshot ages, for seven hosts. I grepped one file of a two-file tool and generalised, which is the identical error a peer had just made with dev-backup, made while correcting them. THE REAL GAP was narrower and still real: the NH3 host list read `irv-ml1 nh3-docker` and omitted nh3-dev -- the repo holding every Claude Code session transcript, althing routes, hermes run history and Miranda's conversation. /volume1/Backup/restic/nh3-dev/ has always existed and always been written nightly; it was simply never enumerated, so a total failure of the one repo nobody could reconstruct would have gone unreported indefinitely. THE CONTENT ASSERTION, built on svos-dev's ladder (list proves the repo answers; ls proves the index; check proves structure; RESTORE proves the bytes come back): - CONJUNCTIVE: latest snapshot newer than MAX_AGE_H AND the probe path present AND a restore returns non-zero bytes. "A snapshot exists containing X" is satisfied by a three-month-old one; "the latest is recent" is satisfied by an empty one. Age alone was the old problem; content alone is the same problem rotated. - IT RESTORES. Metadata is what survives the failures worth fearing -- a pruned or partially-written repo can list a path whose blobs are gone. ~493KB from ~/.local/state/svos, sub-second. Large paths are not restored: a repo that returns one file will return others. - IDENTITY: the timer runs as lkraven, which has no NOPASSWD sudo on nh3-dev, so the probe hops through infra-ops@localhost. /etc/restic is root-only by design. ⚠ restic ls ALWAYS PRINTS A HEADER LINE, matched or not. A path absent from the repo returns 1 line; a real one returned 6. Counting with `grep -c .` read the header as a hit, so the ABSENT case fell through and reported "blobs gone" -- telling an operator the repository was corrupt when the truth was a mistyped path. Now `grep -c '^/'`. TESTED BY MAKING IT FAIL, because a check only ever seen passing is untested: bogus probe path -> "absent from snapshot"; BACKUP_MAX_AGE_HOURS=1 -> "12h old (>1h)"; healthy -> snapshot id, age, entries, bytes restored. --- docs/runbooks/backups.md | 54 ++++++++++++++------- scripts/check-backup-freshness.sh | 79 ++++++++++++++++++++++++++++++- 2 files changed, 114 insertions(+), 19 deletions(-) diff --git a/docs/runbooks/backups.md b/docs/runbooks/backups.md index 4bac181..060e613 100644 --- a/docs/runbooks/backups.md +++ b/docs/runbooks/backups.md @@ -177,30 +177,48 @@ the stop→umount→rm-ghost→remount→start variant. ## Known gaps / TODO -### ⚠ NOTHING VERIFIES THAT RESTIC EVER RUNS (found 2026-09-22, open) +### ✅ restic content assertion (2026-09-22, CLOSED) -`scripts/backup-freshness-alert.sh` contains **zero** references to restic: +⚠ **I reported this gap wrongly first.** I ran `grep -ic restic` against +`backup-freshness-alert.sh` — the **wrapper** — got 0, and told the operator +nothing on the fleet verified restic. The actual checking lives in +`check-backup-freshness.sh`, which had always checked restic snapshot ages for +seven hosts. Grepping one file of a two-file tool and generalising is the same +error a peer had just made with `dev-backup`, made while correcting them. -```sh -grep -ic restic scripts/backup-freshness-alert.sh # -> 0 -``` +**The real gap was narrower and still real:** the host list read +`irv-ml1 nh3-docker` and **omitted nh3-dev** — the repo holding every Claude +Code transcript, althing routes, hermes history and Miranda's conversation. +`/volume1/Backup/restic/nh3-dev/` had always existed and always been written; +it was simply never enumerated, so a total failure of the one repo nobody could +reconstruct would have gone unreported indefinitely. -It checks PBS guest snapshot ages, and it pings the rest-servers for liveness -(`rest-server-nh3: up (401)`). **That confirms the server answers. It says -nothing about whether a snapshot was ever written to it.** +Both are fixed. nh3-dev is in the list, and a **content assertion** now runs per +the operator ruling of 2026-09-22 (assert snapshot content, not REST-server +reachability): -So if resticprofile stopped, the password file broke, or every run failed, the -checker would go on printing `RESULT: all backups fresh` — correctly by its own -definition — indefinitely. Restic holds the entire home directory on nh3-dev; -PBS holds the VM images. The layer with the granular data is the unwatched one. +- **Conjunctive**: latest snapshot newer than `MAX_AGE_H` **AND** the probe path + present **AND** a restore returns non-zero bytes. Age alone was the old + check's problem; content alone is the same problem rotated. +- **It restores.** Listing proves the *index*; only a restore proves the bytes + come back, and metadata is exactly what survives a pruned or partially-written + repo. ~493 KB from `~/.local/state/svos`, sub-second. The large paths are not + restored: a repo that returns one file will return others, one that cannot is + broken for everything. +- **Identity**: the timer runs as `lkraven` (no NOPASSWD sudo here), so the probe + hops through `infra-ops@localhost`, which has it. The credentials under + `/etc/restic` are deliberately root-only. -This is the same shape as two other instruments found the same day: a tool that -enumerates what is fine has selected against its own subject. Here the light is -not merely blind — it is **actively reassuring** about a system it cannot see. +⚠ **`restic ls` always prints a header line**, matched or not — a path absent +from the repo returns 1 line, a real one returned 6. Counting with `grep -c .` +read the header as a hit and made the ABSENT case fall through to *"blobs +gone"*, telling an operator the repository was corrupt when the truth was a +mistyped path. Count `grep -c '^/'`. -Fix when taken: check restic snapshot **age per host**, not endpoint liveness, -and carry an expected-paths list so "covered by a job that has not run in a -month" stops rendering identically to "covered". +**Tested by making it fail**, because a check only ever seen passing is +untested: a bogus probe path reports *absent*; `BACKUP_MAX_AGE_HOURS=1` reports +*too old*; the healthy path reports the snapshot id, age, entry count and bytes +restored. diff --git a/scripts/check-backup-freshness.sh b/scripts/check-backup-freshness.sh index 66df688..5367f54 100755 --- a/scripts/check-backup-freshness.sh +++ b/scripts/check-backup-freshness.sh @@ -133,7 +133,14 @@ for c in ana-docker ana-ml2 esh-docker-vm esh-vm-db vm-esh-nas; do report "restic/ana/$c" "$(newest_epoch ana-nas "/mnt/backup/restic/repo/ana/$c/snapshots/*")" done # --- Layer: restic file+DB, NH3 side (rest-server-nh3) --- -for c in irv-ml1 nh3-docker; do +# ⚠ nh3-dev WAS MISSING FROM THIS LIST until 2026-09-22, and it is the host +# whose restic repo holds the most irreplaceable data on the fleet: every +# Claude Code session transcript, althing routes, hermes run history, and +# Miranda's conversation. Its repo has always existed at +# /volume1/Backup/restic/nh3-dev/ and has always been written nightly -- it was +# simply never enumerated here, so a total failure of the one repo nobody could +# reconstruct would have gone unreported indefinitely. +for c in irv-ml1 nh3-docker nh3-dev; do report "restic/nh3/$c" "$(newest_epoch nh3-nas "/volume1/Backup/restic/$c/snapshots/*")" done # --- Layer: PBS VM images (newest per guest, all namespaces) --- @@ -192,6 +199,76 @@ for entry in "${PVE_NODES[@]}"; do done <<<"$bad" done +# --- restic CONTENT assertion (operator ruling 2026-09-22) ------------------- +# +# The checks above read a snapshot directory's MTIME on the NAS. That proves a +# file was written recently. It does not prove the repository holds the data, +# and metadata is exactly what survives the failures worth fearing: a pruned or +# partially-written repo can still list a path whose blobs are gone. +# +# ⚠ THE ASSERTION IS CONJUNCTIVE ON PURPOSE. "a snapshot exists containing X" +# is satisfied by a three-month-old snapshot; "the latest snapshot is recent" is +# satisfied by an empty one. Age alone was the old check's problem; content +# alone is the same problem rotated. Both, or neither is worth printing. +# +# The rungs, and what each actually proves (svos-dev, 2026-09-22): +# snapshots list the repo answers +# ls a path in it the path is in the INDEX <- metadata only +# restic check the repo's structure is intact +# RESTORE a file the bytes come back <- the only proof of recovery +# We take the top rung for the small irreplaceable set, because it is the only +# one that answers the question a green light is taken to mean. Restoring from +# the multi-gigabyte paths is not worth it: a repo that returns one file is +# overwhelmingly likely to return others, and one that cannot is broken for +# everything. +# +# ⚠ IDENTITY: this runs as a --user timer (lkraven), which has no NOPASSWD sudo +# on nh3-dev, and the repo credentials live under /etc/restic. So the probe hops +# through infra-ops@localhost, which does. Not a workaround -- infra-ops is the +# documented ops identity and the credentials are deliberately root-only. +RESTIC_PROBE_PATHS="${RESTIC_PROBE_PATHS:-/home/lkraven/.local/state/svos}" +probe=$($SSH infra-ops@10.100.10.50 "sudo -n bash -s" </dev/null +set -a; . /etc/restic/restic.env 2>/dev/null; set +a +export RESTIC_PASSWORD_FILE=/etc/restic/password +id=\$(restic snapshots --latest 1 --json 2>/dev/null | python3 -c 'import json,sys +d=json.load(sys.stdin) +print(d[0]["short_id"], int(__import__("datetime").datetime.fromisoformat(d[0]["time"][:19]).timestamp())) if d else print("NONE 0")' 2>/dev/null) +sid=\$(echo "\$id" | cut -d" " -f1); ep=\$(echo "\$id" | cut -d" " -f2) +[ "\$sid" = "NONE" ] && { echo "NOSNAP 0 0 0"; exit 0; } +# ⚠ grep -c '^/' NOT grep -c '.' — \`restic ls\` always prints a HEADER line +# ("snapshot of [...] filtered by [...]") whether or not anything matched, +# so a path absent from the repo returns 1 line and a naive count reads it as +# found. Measured 2026-09-22: real path 6 lines, bogus path 1. That made the +# ABSENT case fall through and report "blobs gone" — telling an operator the +# repository was corrupt when the truth was a mistyped path. Only lines that +# begin with / are entries. +n=\$(restic ls "\$sid" $RESTIC_PROBE_PATHS 2>/dev/null | grep -c '^/') +tmp=\$(mktemp -d); restic restore "\$sid" --target "\$tmp" --include $RESTIC_PROBE_PATHS >/dev/null 2>&1 +bytes=\$(find "\$tmp" -type f -printf '%s\n' 2>/dev/null | awk '{s+=\$1} END {print s+0}') +rm -rf -- "\$tmp" +echo "\$sid \$ep \$n \$bytes" +PROBE +) +set -- ${probe:-ERR 0 0 0} +r_sid="$1"; r_ep="$2"; r_entries="$3"; r_bytes="$4" +if [ "$r_sid" = "ERR" ] || [ -z "$probe" ]; then + stale+=("restic/nh3-dev CONTENT: probe could not run (repo unreachable or creds unreadable)") +elif [ "$r_sid" = "NOSNAP" ]; then + stale+=("restic/nh3-dev CONTENT: repository holds NO SNAPSHOTS") +else + r_age=$(( (now - r_ep) / 3600 )) + # conjunctive: recent AND indexed AND restorable + if [ "$r_age" -gt "$MAX_AGE_H" ]; then + stale+=("restic/nh3-dev CONTENT: latest snapshot $r_sid is ${r_age}h old (>${MAX_AGE_H}h)") + elif [ "${r_entries:-0}" -lt 1 ]; then + stale+=("restic/nh3-dev CONTENT: $RESTIC_PROBE_PATHS absent from snapshot $r_sid") + elif [ "${r_bytes:-0}" -lt 1 ]; then + stale+=("restic/nh3-dev CONTENT: $RESTIC_PROBE_PATHS listed in $r_sid but RESTORED ZERO BYTES — index intact, blobs gone") + else + fresh+=("restic/nh3-dev CONTENT: $r_sid ${r_age}h old, $r_entries entries, restored ${r_bytes}B") + fi +fi + # --- rest-server endpoint health (401 = up & serving) --- for ep in "rest-server-ana http://10.250.50.70:8000/" "rest-server-nh3 http://10.100.50.50:8000/"; do set -- $ep