docs(backups): the PBS loop's defect triple of 2026-10-05/06 and the review rule

Three defects in one loop in ninety minutes, none visible by reading the
file, all caught by running it against known truth. The runbook's
tested-note now says what is actually tested (one positive control) and
what that implies for future edits.
This commit is contained in:
vh
2026-10-06 08:07:52 -07:00
parent e7bc70bc48
commit 24b7195bcd
+14
View File
@@ -277,6 +277,20 @@ untested: a bogus probe path reports *absent*; `BACKUP_MAX_AGE_HOURS=1` reports
*too old*; the healthy path reports the snapshot id, age, entry count and bytes
restored.
⚠ **The PBS layer has exactly one test (the `MAX_AGE_HOURS` positive control);
everything else in it is trust-the-instrument.** On 2026-10-05/06 three
independent defects landed in the same PBS loop in ninety minutes — a `case`
that never matched a namespaced path (crash under `set -u`), an `ssh` without
`-n` inside a `while read` loop (swallowed the loop's stdin; the whole PBS
layer vanished from the report while the verdict stayed green), and snapshot
age read from directory mtime (PBS rewrites it at each verify, delaying a
stopped-backup alarm ~1 day then silencing it for a month). None were visible
by reading the file; all were caught only by running the checker and
interrogating the numbers against known truth. Lesson for any edit to this
script: run it, and have a second seat read the diff before an alarm-script
change ships — an alarm script with a silent hole is worse than no alarm,
because it is trusted.
- [x] **Backup-freshness alerting — DONE (2026-06-20).**