diff --git a/docs/runbooks/backups.md b/docs/runbooks/backups.md index 327dabd..d2c6ab2 100644 --- a/docs/runbooks/backups.md +++ b/docs/runbooks/backups.md @@ -277,6 +277,20 @@ untested: a bogus probe path reports *absent*; `BACKUP_MAX_AGE_HOURS=1` reports *too old*; the healthy path reports the snapshot id, age, entry count and bytes restored. +⚠ **The PBS layer has exactly one test (the `MAX_AGE_HOURS` positive control); +everything else in it is trust-the-instrument.** On 2026-10-05/06 three +independent defects landed in the same PBS loop in ninety minutes — a `case` +that never matched a namespaced path (crash under `set -u`), an `ssh` without +`-n` inside a `while read` loop (swallowed the loop's stdin; the whole PBS +layer vanished from the report while the verdict stayed green), and snapshot +age read from directory mtime (PBS rewrites it at each verify, delaying a +stopped-backup alarm ~1 day then silencing it for a month). None were visible +by reading the file; all were caught only by running the checker and +interrogating the numbers against known truth. Lesson for any edit to this +script: run it, and have a second seat read the diff before an alarm-script +change ships — an alarm script with a silent hole is worse than no alarm, +because it is trusted. + - [x] **Backup-freshness alerting — DONE (2026-06-20).**