docs(backups): ask the repo not the job, and record that nothing watches restic
Two additions, both from a 2026-09-22 exchange with svos-dev. THE RULE. Coverage is a property of the backup SYSTEM, not of one job's configured scope. A peer checked dev-backup.sh, found SRC=$HOME/development, and reported to the operator -- with specifics and unhedged -- that five home-directory paths including Miranda's entire conversation had never been backed up anywhere. All five were in that night's restic snapshot. dev-backup is the hourly job for one directory; resticprofile is the daily job covering all of /home/lkraven. Checking one job and generalising to the system produced a confident, false, escalated claim. The runbook now carries the query that answers the question properly. THE GAP THAT VERIFYING IT EXPOSED, and it is worse. grep -ic restic against scripts/backup-freshness-alert.sh returns 0. The checker inspects PBS guest ages and pings the rest-servers for liveness -- which confirms the server answers, not that a snapshot was written. If resticprofile stopped entirely the light would stay green, correctly by its own definition, forever. Restic holds the whole home directory; PBS holds VM images. The layer with the granular data is the unwatched one, and the light is not merely blind but actively reassuring about a system it cannot see. Recorded as an open gap rather than patched, because fixing it changes what an existing green light means and people have been reading that light for months.
This commit is contained in:
@@ -82,6 +82,33 @@ auth, append-only, private repos). The split is by site:
|
||||
|
||||
---
|
||||
|
||||
## ⚠ Answering "is X backed up?" — ask the repo, never the job
|
||||
|
||||
**Coverage is a property of the backup SYSTEM, not of any one job's configured
|
||||
scope.** Establish it by querying the repository for the path and seeing it in a
|
||||
real snapshot. Never by reading a job's `SRC=`.
|
||||
|
||||
```sh
|
||||
# as infra-ops on the host, against its own profile
|
||||
sudo -n bash -c 'set -a; . /etc/restic/restic.env; set +a
|
||||
export RESTIC_PASSWORD_FILE=/etc/restic/password
|
||||
id=$(restic snapshots --latest 1 --json | jq -r ".[0].short_id")
|
||||
restic ls "$id" /path/in/question | head'
|
||||
```
|
||||
|
||||
**The failure this rule exists to prevent (2026-09-22).** A peer agent checked
|
||||
`dev-backup.sh`, found `SRC="$HOME/development/"`, and reported — with
|
||||
specifics, unhedged, to the operator — that five home-directory paths including
|
||||
Miranda's entire conversation had *"never been backed up anywhere"*. All five
|
||||
were in that night's restic snapshot. `dev-backup` is the **hourly** job for one
|
||||
directory; **resticprofile** is the daily job covering all of `/home/lkraven`,
|
||||
and it is what actually protects home. Checking one job and generalising to the
|
||||
system produced a confident, false, escalated claim.
|
||||
|
||||
The true finding underneath was much smaller and is a **cadence** question, not
|
||||
a coverage one: live conversation state has a 24-hour RPO because it rides the
|
||||
daily job rather than the hourly one.
|
||||
|
||||
## The 2-minute freshness check
|
||||
|
||||
Run these any time you need to answer "are we backed up?"
|
||||
@@ -150,6 +177,33 @@ the stop→umount→rm-ghost→remount→start variant.
|
||||
|
||||
## Known gaps / TODO
|
||||
|
||||
### ⚠ NOTHING VERIFIES THAT RESTIC EVER RUNS (found 2026-09-22, open)
|
||||
|
||||
`scripts/backup-freshness-alert.sh` contains **zero** references to restic:
|
||||
|
||||
```sh
|
||||
grep -ic restic scripts/backup-freshness-alert.sh # -> 0
|
||||
```
|
||||
|
||||
It checks PBS guest snapshot ages, and it pings the rest-servers for liveness
|
||||
(`rest-server-nh3: up (401)`). **That confirms the server answers. It says
|
||||
nothing about whether a snapshot was ever written to it.**
|
||||
|
||||
So if resticprofile stopped, the password file broke, or every run failed, the
|
||||
checker would go on printing `RESULT: all backups fresh` — correctly by its own
|
||||
definition — indefinitely. Restic holds the entire home directory on nh3-dev;
|
||||
PBS holds the VM images. The layer with the granular data is the unwatched one.
|
||||
|
||||
This is the same shape as two other instruments found the same day: a tool that
|
||||
enumerates what is fine has selected against its own subject. Here the light is
|
||||
not merely blind — it is **actively reassuring** about a system it cannot see.
|
||||
|
||||
Fix when taken: check restic snapshot **age per host**, not endpoint liveness,
|
||||
and carry an expected-paths list so "covered by a job that has not run in a
|
||||
month" stops rendering identically to "covered".
|
||||
|
||||
|
||||
|
||||
- [x] **Backup-freshness alerting — DONE (2026-06-20).**
|
||||
`scripts/check-backup-freshness.sh` (the 2-min check, exit 1 on
|
||||
stale/down) + a daily **systemd user timer on nh3-dev** at 08:00
|
||||
|
||||
Reference in New Issue
Block a user