From 0e47d2a741ba557df8d34df695fad449160e75dc Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 9 Sep 2026 17:00:57 -0700 Subject: [PATCH] =?UTF-8?q?docs(ops-lessons):=20=C2=A711=20check=20the=20w?= =?UTF-8?q?riter=20and=20the=20reader=20together,=20with=20the=20three=20f?= =?UTF-8?q?ailure=20directions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The empty-snapshot promote and brokkr-smithy-dev's -s-on-a-touched-sentinel are the same defect from opposite sides. Records the direction split (false reassurance / false refusal / false alarm) because the mitigations differ and a post-mortem that lumps them loses the actionable half. --- docs/pfi/ops-lessons-playbook.md | 52 ++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/docs/pfi/ops-lessons-playbook.md b/docs/pfi/ops-lessons-playbook.md index e8e38d3..124459f 100644 --- a/docs/pfi/ops-lessons-playbook.md +++ b/docs/pfi/ops-lessons-playbook.md @@ -313,6 +313,58 @@ The policy work is writing and proving default-deny, not deciding what to expose --- +## 11. Check the writer and the reader together — and name the failure's DIRECTION + +A guard's predicate has to be read against what its writer actually produces. +Each line is individually reasonable; the mismatch only exists when you hold +both. **A guard whose test disagrees with its writer's contract has quietly +stopped guarding, in whichever direction the mismatch runs.** + +Two of these surfaced within an hour on 2026-09-09, on opposite sides of the same +gate. + +**The dangerous half (ours).** `scripts/refresh-server-info.sh` and its Proxmox +sibling promoted the SSH capture with an unconditional `mv` whenever ssh exited +0. Every reader downstream tests that snapshot with `-s`. So a host that +connected fine and emitted nothing — inspect script dying before its first write, +output swallowed by a remote wrapper — would **replace a good snapshot with an +empty file and report `ok (0 bytes)`.** The header claimed "a failed run never +clobbers the previous good snapshot", which was true only for a failed +*connection*; the succeeded-but-empty case is the half nobody inspects, because +the line reads reassuring. Fixed `9b9f062`: empty capture refused, previous +snapshot kept, host counted as failed and the exit code carries it. A capture +that collapses to under a quarter of the previous one still **promotes but is +flagged** — a host really can shed services, and a script that guesses there will +eventually guess wrong on a real one. + +**The annoying half (brokkr-smithy-dev's, same day).** A launch guard tested a +completion sentinel with `-s`; the writer creates it with `touch`, so it is zero +bytes. The precondition could never pass, whatever the upstream job did. Worse +*shape* than a regression: from the outside it is indistinguishable from a +legitimate refusal, so it sends you hunting a problem that does not exist. + +### The three directions, because the mitigations differ + +Same root — the instrument observed something *adjacent* to what it was named +after — but these do not collapse into "the instrument was wrong": + +| direction | specimen (all 2026-09-09) | what it wants | +|---|---|---| +| **False reassurance** | the empty-snapshot promote; `pgrep -f base_window_r7` over ssh matching its own argv, reporting a peer's job "alive" for 2.5 h while blind to it | an **independent observation of the object** — the seat's own request log (`Running: N reqs`), the artifact itself | +| **False refusal** | `-s` on a `touch`ed sentinel | a **predicate that matches its writer's contract** | +| **False alarm** | an error scan reporting 2 hits by matching the word "refusal" in a log | a **pattern that matches the thing**, not a word appearing near it | + +**False reassurance is the one that kills you quietly.** The other two announce +themselves: they waste attention and misdirect, but they cannot silently destroy +a good artifact. A post-mortem that lumps all three together loses the half that +decides what to do about it. + +**How to apply.** When you write or review a guard, open its writer in the same +pass and state the contract out loud — `touch` → exists-but-empty; `mv` on +exit-0 → may be empty; `>` → may be truncated; `pgrep -f ` → matches +your own argv. Then ask which direction this predicate fails toward. Sibling of +§3 (identity, not resemblance) and §2 (observe the state, don't infer it). + ## Superseded claims | date | claim | correction |