docs(backups): the PBS loop's defect triple of 2026-10-05/06 and the review rule
Three defects in one loop in ninety minutes, none visible by reading the file, all caught by running it against known truth. The runbook's tested-note now says what is actually tested (one positive control) and what that implies for future edits.
This commit is contained in:
@@ -277,6 +277,20 @@ untested: a bogus probe path reports *absent*; `BACKUP_MAX_AGE_HOURS=1` reports
|
||||
*too old*; the healthy path reports the snapshot id, age, entry count and bytes
|
||||
restored.
|
||||
|
||||
⚠ **The PBS layer has exactly one test (the `MAX_AGE_HOURS` positive control);
|
||||
everything else in it is trust-the-instrument.** On 2026-10-05/06 three
|
||||
independent defects landed in the same PBS loop in ninety minutes — a `case`
|
||||
that never matched a namespaced path (crash under `set -u`), an `ssh` without
|
||||
`-n` inside a `while read` loop (swallowed the loop's stdin; the whole PBS
|
||||
layer vanished from the report while the verdict stayed green), and snapshot
|
||||
age read from directory mtime (PBS rewrites it at each verify, delaying a
|
||||
stopped-backup alarm ~1 day then silencing it for a month). None were visible
|
||||
by reading the file; all were caught only by running the checker and
|
||||
interrogating the numbers against known truth. Lesson for any edit to this
|
||||
script: run it, and have a second seat read the diff before an alarm-script
|
||||
change ships — an alarm script with a silent hole is worse than no alarm,
|
||||
because it is trusted.
|
||||
|
||||
|
||||
|
||||
- [x] **Backup-freshness alerting — DONE (2026-06-20).**
|
||||
|
||||
Reference in New Issue
Block a user