docs(backups): the PBS loop's defect triple of 2026-10-05/06 and the review rule

Three defects in one loop in ninety minutes, none visible by reading the file, all caught by running it against known truth. The runbook's tested-note now says what is actually tested (one positive control) and what that implies for future edits.
2026-10-06 08:07:52 -07:00
parent e7bc70bc48
commit 24b7195bcd
1 changed files with 14 additions and 0 deletions
@@ -277,6 +277,20 @@ untested: a bogus probe path reports *absent*; `BACKUP_MAX_AGE_HOURS=1` reports
 *too old*; the healthy path reports the snapshot id, age, entry count and bytes
 restored.
 ⚠ **The PBS layer has exactly one test (the `MAX_AGE_HOURS` positive control);
 everything else in it is trust-the-instrument.** On 2026-10-05/06 three
 independent defects landed in the same PBS loop in ninety minutes — a `case`
 that never matched a namespaced path (crash under `set -u`), an `ssh` without
 `-n` inside a `while read` loop (swallowed the loop's stdin; the whole PBS
 layer vanished from the report while the verdict stayed green), and snapshot
 age read from directory mtime (PBS rewrites it at each verify, delaying a
 stopped-backup alarm ~1 day then silencing it for a month). None were visible
 by reading the file; all were caught only by running the checker and
 interrogating the numbers against known truth. Lesson for any edit to this
 script: run it, and have a second seat read the diff before an alarm-script
 change ships — an alarm script with a silent hole is worse than no alarm,
 because it is trusted.
 - [x] **Backup-freshness alerting — DONE (2026-06-20).**