Files
esh-pfi-infrastructure/persistent-memory.d/2026-09-21-ops-log-and-the-instruments-that-lied.md
T
vh e52def115c memory: snapshot — the ops log, and a day spent on instruments that report without looking
Archived 15 entries (Recent decisions 14, Tried and abandoned 1) oldest-first
to archival-memory.md; 5 held back on the open-deferred-work guard and 164 on
the 14-day guard, so the index stays over the soft cap at 477 lines. An
over-cap file that keeps live decisions beats a scannable one that lost a
belayed item.

Four new detail files cover the day: the ops log and its four self-inflicted
failure modes, the Booth's two dead controls and the four-iteration layout
probe, the Gitea org grant plus the dead claude-bot token that had been
misreporting permissions, and the disk triage that rescued a LoRA adapter from
a directory this box sweeps at three days.

lv-mccarthy's run outcome remains unverified after two days and is the first
line of the in-flight section and step 1 of the handoff.
2026-09-21 14:26:55 -07:00

2.7 KiB

[2026-09-21] The ops log shipped, and the day's real subject was instruments that report without looking

Built scripts/ops-log (ffe7b24) to close the fingerprint-less-change gap: infra-ops and infra-hermes act as one OS identity, dockerd logs no per-caller exec, and every commit here is attributed to Vuong Hoang by convention. One appended line per host-changing action, a mkdir-atomic claim deploy-stack.sh refuses (exit 3), automatic writers in deploy-stack.sh + elway, and ops-log audit as the detector for the raw-ssh path the writers cannot see. 136-stack baseline laid so the detector starts from that day.

⚠ The instrument then failed FOUR ways in its first hours, and every one recorded something — just nothing findable. Documented as a table in docs/pfi/ops-log.md § "How this instrument has failed", which is the durable artifact:

  1. Claim released by a sub-tool (3e7d3a3) — a 45-min operation claim was refreshed then released by deploy-stack.sh's exit trap, mid-rollout. claim now exits 10 when already yours and leaves the holder file untouched, so a refresh cannot overwrite the reason and TTL the original claimant chose.
  2. Wrong order in the hook chain (9141a41) — the commit hook was APPENDED behind graphify's eight exit 0 paths, so a graphify-out/-only or empty commit could never be recorded. Prepend; attribution must never be a subordinate clause of another hook's interestingness filter.
  3. No handle in the environment (4e778ae) — ALTHING_HANDLE lived only in althing-infra-hermes-seat-run.sh, not the gateway unit. Fallback now says unattributed(login) rather than a bare login that reads like an answer.
  4. Wrong host key on write (f3b68e2) — elway passed its ssh TARGET through as the host, so five records of a real esh-pve change landed under infra-ops@esh-pve and were invisible to --host esh-pve. infra-hermes correctly reported the change as unattributed. A log you cannot query under the obvious name is not a log.

⚠ The general lesson, and it outlived the tool: twelve instruments reported confidently and wrongly across 2026-09-19→21, five of them mine. The recurring shape is configured ≠ effective — systemctl show -p Environment reporting a drop-in while /proc/<pid>/environ lacked it; a grep proving presence while evaluation proved absence; a green test suite over a control the browser never received. What broke the pattern every time was asking a different instrument the same question.

See 2026-09-21-booth-two-dead-controls for the same failure in a UI, and feedback_control_flow_before_concurrency in auto-memory for the triage rule that came out of it.