From cebffae56d573f23abb8146d432239eecf515f24 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 19 Sep 2026 08:08:36 -0700 Subject: [PATCH] docs(ops-log): record how this instrument has failed, not just the fixes Four attribution failures shipped on the day it was built, and every one of them RECORDED SOMETHING -- just not something findable or attributable. The individual fixes are in the commit log; the PATTERN was not written down anywhere, so a future session would meet each one as an isolated bug rather than as the fifth instance of a known shape. infra-hermes asked for this after the fourth one: 'worth an audit line item so the pattern is visible, not just the fixes.' He was right. The operative warning is at the bottom: absence of an entry is not evidence of absence of the event until you have checked this tool could have recorded it. --- docs/pfi/ops-log.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/docs/pfi/ops-log.md b/docs/pfi/ops-log.md index 29d029a..f35ca46 100644 --- a/docs/pfi/ops-log.md +++ b/docs/pfi/ops-log.md @@ -128,6 +128,38 @@ claim does. --- +## How this instrument has failed — the pattern, not just the fixes + +**All four failures below shipped on the day it was built, 2026-09-19, and +every one of them RECORDED SOMETHING.** None of them recorded something +*findable* or *attributable*. That is the shape of the failure class here: this +tool does not fall over loudly, it answers the wrong question quietly. Read +this table before assuming a missing entry means the event did not happen. + +| # | Failure | Symptom | Cause | Fixed | +|---|---|---|---|---| +| 1 | **Claim released by a sub-tool** | A 45-min operation claim vanished mid-rollout; nothing refused anything | `deploy-stack.sh` claims and releases around its own work, and released a claim it had only refreshed | `3e7d3a3` — `claim` exits 10 when already yours and leaves the holder file untouched | +| 2 | **Wrong order in the hook chain** | Some commits logged, some not; looked flaky | The commit hook was APPENDED behind graphify's eight `exit 0` paths, so a `graphify-out/`-only or empty commit could never reach it | `9141a41` — prepend after the shebang; attribution runs first, unconditionally | +| 3 | **No handle in the environment** | A peer's commit logged as `lkraven@nh3-dev` | `ALTHING_HANDLE` was exported only by `althing-infra-hermes-seat-run.sh`, not by `hermes-gateway.service` or the pump unit | Drop-in overrides (infra-hermes) + `4e778ae` — the fallback now says `unattributed(login)` instead of a bare login that reads like an answer | +| 4 | **Wrong host key on write** | Five records of a real `jobs.cfg` change on esh-pve were invisible to `--host esh-pve` | elway passed its ssh TARGET through as the host, so they landed under `infra-ops@esh-pve` | `f3b68e2` — `normalize_host` strips `user@` on write AND on query, so pre-fix records are findable too | + +**The rules that fell out of it, in descending order of how much they cost:** + +1. **A log you cannot query under the obvious name is not a log.** #4 was not + cosmetic — it produced a confident "this change is unattributed" report + about a change that had five records. +2. **Attribution must run FIRST**, never behind another tool's opinion about + whether an event is interesting. +3. **An instrument that cannot answer must say so.** A bare unix login where a + handle belongs is answer-shaped; `unattributed(...)` is not. +4. **A sub-tool must not drop the claim wrapping its caller.** +5. **Two samples that look flaky deserve a control-flow audit before a + concurrency hypothesis.** #2 was diagnosed as an `fcntl` race and was not; + it was deterministic and five lines up. + +⚠ **Absence of an entry is not evidence of absence of the event** until you +have checked that this tool could have recorded it in the first place. + ## What it deliberately does not do - **It does not cover raw `ssh`.** It cannot. `audit` is the backstop.