diff --git a/docs/runbooks/backups.md b/docs/runbooks/backups.md index 27e2f42..4bac181 100644 --- a/docs/runbooks/backups.md +++ b/docs/runbooks/backups.md @@ -82,6 +82,33 @@ auth, append-only, private repos). The split is by site: --- +## ⚠ Answering "is X backed up?" — ask the repo, never the job + +**Coverage is a property of the backup SYSTEM, not of any one job's configured +scope.** Establish it by querying the repository for the path and seeing it in a +real snapshot. Never by reading a job's `SRC=`. + +```sh +# as infra-ops on the host, against its own profile +sudo -n bash -c 'set -a; . /etc/restic/restic.env; set +a + export RESTIC_PASSWORD_FILE=/etc/restic/password + id=$(restic snapshots --latest 1 --json | jq -r ".[0].short_id") + restic ls "$id" /path/in/question | head' +``` + +**The failure this rule exists to prevent (2026-09-22).** A peer agent checked +`dev-backup.sh`, found `SRC="$HOME/development/"`, and reported — with +specifics, unhedged, to the operator — that five home-directory paths including +Miranda's entire conversation had *"never been backed up anywhere"*. All five +were in that night's restic snapshot. `dev-backup` is the **hourly** job for one +directory; **resticprofile** is the daily job covering all of `/home/lkraven`, +and it is what actually protects home. Checking one job and generalising to the +system produced a confident, false, escalated claim. + +The true finding underneath was much smaller and is a **cadence** question, not +a coverage one: live conversation state has a 24-hour RPO because it rides the +daily job rather than the hourly one. + ## The 2-minute freshness check Run these any time you need to answer "are we backed up?" @@ -150,6 +177,33 @@ the stop→umount→rm-ghost→remount→start variant. ## Known gaps / TODO +### ⚠ NOTHING VERIFIES THAT RESTIC EVER RUNS (found 2026-09-22, open) + +`scripts/backup-freshness-alert.sh` contains **zero** references to restic: + +```sh +grep -ic restic scripts/backup-freshness-alert.sh # -> 0 +``` + +It checks PBS guest snapshot ages, and it pings the rest-servers for liveness +(`rest-server-nh3: up (401)`). **That confirms the server answers. It says +nothing about whether a snapshot was ever written to it.** + +So if resticprofile stopped, the password file broke, or every run failed, the +checker would go on printing `RESULT: all backups fresh` — correctly by its own +definition — indefinitely. Restic holds the entire home directory on nh3-dev; +PBS holds the VM images. The layer with the granular data is the unwatched one. + +This is the same shape as two other instruments found the same day: a tool that +enumerates what is fine has selected against its own subject. Here the light is +not merely blind — it is **actively reassuring** about a system it cannot see. + +Fix when taken: check restic snapshot **age per host**, not endpoint liveness, +and carry an expected-paths list so "covered by a job that has not run in a +month" stops rendering identically to "covered". + + + - [x] **Backup-freshness alerting — DONE (2026-06-20).** `scripts/check-backup-freshness.sh` (the 2-min check, exit 1 on stale/down) + a daily **systemd user timer on nh3-dev** at 08:00