docs(backups): ask the repo not the job, and record that nothing watches restic

Two additions, both from a 2026-09-22 exchange with svos-dev.

THE RULE. Coverage is a property of the backup SYSTEM, not of one job's
configured scope. A peer checked dev-backup.sh, found SRC=$HOME/development,
and reported to the operator -- with specifics and unhedged -- that five
home-directory paths including Miranda's entire conversation had never been
backed up anywhere. All five were in that night's restic snapshot. dev-backup
is the hourly job for one directory; resticprofile is the daily job covering
all of /home/lkraven. Checking one job and generalising to the system produced
a confident, false, escalated claim. The runbook now carries the query that
answers the question properly.

THE GAP THAT VERIFYING IT EXPOSED, and it is worse. grep -ic restic against
scripts/backup-freshness-alert.sh returns 0. The checker inspects PBS guest
ages and pings the rest-servers for liveness -- which confirms the server
answers, not that a snapshot was written. If resticprofile stopped entirely the
light would stay green, correctly by its own definition, forever. Restic holds
the whole home directory; PBS holds VM images. The layer with the granular data
is the unwatched one, and the light is not merely blind but actively reassuring
about a system it cannot see.

Recorded as an open gap rather than patched, because fixing it changes what an
existing green light means and people have been reading that light for months.
This commit is contained in:
vh
2026-09-22 13:28:04 -07:00
parent 00a03921ff
commit a4cf2ba0dc
+54
View File
@@ -82,6 +82,33 @@ auth, append-only, private repos). The split is by site:
---
## ⚠ Answering "is X backed up?" — ask the repo, never the job
**Coverage is a property of the backup SYSTEM, not of any one job's configured
scope.** Establish it by querying the repository for the path and seeing it in a
real snapshot. Never by reading a job's `SRC=`.
```sh
# as infra-ops on the host, against its own profile
sudo -n bash -c 'set -a; . /etc/restic/restic.env; set +a
export RESTIC_PASSWORD_FILE=/etc/restic/password
id=$(restic snapshots --latest 1 --json | jq -r ".[0].short_id")
restic ls "$id" /path/in/question | head'
```
**The failure this rule exists to prevent (2026-09-22).** A peer agent checked
`dev-backup.sh`, found `SRC="$HOME/development/"`, and reported — with
specifics, unhedged, to the operator — that five home-directory paths including
Miranda's entire conversation had *"never been backed up anywhere"*. All five
were in that night's restic snapshot. `dev-backup` is the **hourly** job for one
directory; **resticprofile** is the daily job covering all of `/home/lkraven`,
and it is what actually protects home. Checking one job and generalising to the
system produced a confident, false, escalated claim.
The true finding underneath was much smaller and is a **cadence** question, not
a coverage one: live conversation state has a 24-hour RPO because it rides the
daily job rather than the hourly one.
## The 2-minute freshness check
Run these any time you need to answer "are we backed up?"
@@ -150,6 +177,33 @@ the stop→umount→rm-ghost→remount→start variant.
## Known gaps / TODO
### ⚠ NOTHING VERIFIES THAT RESTIC EVER RUNS (found 2026-09-22, open)
`scripts/backup-freshness-alert.sh` contains **zero** references to restic:
```sh
grep -ic restic scripts/backup-freshness-alert.sh # -> 0
```
It checks PBS guest snapshot ages, and it pings the rest-servers for liveness
(`rest-server-nh3: up (401)`). **That confirms the server answers. It says
nothing about whether a snapshot was ever written to it.**
So if resticprofile stopped, the password file broke, or every run failed, the
checker would go on printing `RESULT: all backups fresh` — correctly by its own
definition — indefinitely. Restic holds the entire home directory on nh3-dev;
PBS holds the VM images. The layer with the granular data is the unwatched one.
This is the same shape as two other instruments found the same day: a tool that
enumerates what is fine has selected against its own subject. Here the light is
not merely blind — it is **actively reassuring** about a system it cannot see.
Fix when taken: check restic snapshot **age per host**, not endpoint liveness,
and carry an expected-paths list so "covered by a job that has not run in a
month" stops rendering identically to "covered".
- [x] **Backup-freshness alerting — DONE (2026-06-20).**
`scripts/check-backup-freshness.sh` (the 2-min check, exit 1 on
stale/down) + a daily **systemd user timer on nh3-dev** at 08:00