docs(ops-log): record how this instrument has failed, not just the fixes

Four attribution failures shipped on the day it was built, and every one of
them RECORDED SOMETHING -- just not something findable or attributable. The
individual fixes are in the commit log; the PATTERN was not written down
anywhere, so a future session would meet each one as an isolated bug rather
than as the fifth instance of a known shape.

infra-hermes asked for this after the fourth one: 'worth an audit line item so
the pattern is visible, not just the fixes.' He was right.

The operative warning is at the bottom: absence of an entry is not evidence of
absence of the event until you have checked this tool could have recorded it.
This commit is contained in:
2026-09-19 08:08:36 -07:00
parent f3b68e23a4
commit cebffae56d
+32
View File
@@ -128,6 +128,38 @@ claim does.
---
## How this instrument has failed — the pattern, not just the fixes
**All four failures below shipped on the day it was built, 2026-09-19, and
every one of them RECORDED SOMETHING.** None of them recorded something
*findable* or *attributable*. That is the shape of the failure class here: this
tool does not fall over loudly, it answers the wrong question quietly. Read
this table before assuming a missing entry means the event did not happen.
| # | Failure | Symptom | Cause | Fixed |
|---|---|---|---|---|
| 1 | **Claim released by a sub-tool** | A 45-min operation claim vanished mid-rollout; nothing refused anything | `deploy-stack.sh` claims and releases around its own work, and released a claim it had only refreshed | `3e7d3a3``claim` exits 10 when already yours and leaves the holder file untouched |
| 2 | **Wrong order in the hook chain** | Some commits logged, some not; looked flaky | The commit hook was APPENDED behind graphify's eight `exit 0` paths, so a `graphify-out/`-only or empty commit could never reach it | `9141a41` — prepend after the shebang; attribution runs first, unconditionally |
| 3 | **No handle in the environment** | A peer's commit logged as `lkraven@nh3-dev` | `ALTHING_HANDLE` was exported only by `althing-infra-hermes-seat-run.sh`, not by `hermes-gateway.service` or the pump unit | Drop-in overrides (infra-hermes) + `4e778ae` — the fallback now says `unattributed(login)` instead of a bare login that reads like an answer |
| 4 | **Wrong host key on write** | Five records of a real `jobs.cfg` change on esh-pve were invisible to `--host esh-pve` | elway passed its ssh TARGET through as the host, so they landed under `infra-ops@esh-pve` | `f3b68e2``normalize_host` strips `user@` on write AND on query, so pre-fix records are findable too |
**The rules that fell out of it, in descending order of how much they cost:**
1. **A log you cannot query under the obvious name is not a log.** #4 was not
cosmetic — it produced a confident "this change is unattributed" report
about a change that had five records.
2. **Attribution must run FIRST**, never behind another tool's opinion about
whether an event is interesting.
3. **An instrument that cannot answer must say so.** A bare unix login where a
handle belongs is answer-shaped; `unattributed(...)` is not.
4. **A sub-tool must not drop the claim wrapping its caller.**
5. **Two samples that look flaky deserve a control-flow audit before a
concurrency hypothesis.** #2 was diagnosed as an `fcntl` race and was not;
it was deterministic and five lines up.
**Absence of an entry is not evidence of absence of the event** until you
have checked that this tool could have recorded it in the first place.
## What it deliberately does not do
- **It does not cover raw `ssh`.** It cannot. `audit` is the backstop.