esh-pfi-infrastructure/persistent-memory.d/2026-09-22-instrument-errors.md

Five instrument errors in one day — and they are one shape

[2026-09-22] · infra-ops, in a running exchange with svos-dev

The shape

A tool that enumerates "things that are fine" and acts on them has selected against its own subject. (svos-dev's framing, taken close to verbatim.)

None of these failed randomly. Several had failure modes anti-correlated with their purpose — the healthier the fleet looked, the better they appeared to work. That is worse than an ordinary broken instrument, which at least fails visibly on the easy cases too.

The five

1. --state=running in install.sh (ef70b2f). Selected only healthy units to hook, so a unit already down at install time was never hooked. The unit most needing the alarm is by definition the one that is not healthy. Found because talk.service was sitting in failed and uncovered.

2. awk '{print $1}' over systemctl list-units (ef70b2f). systemd decorates a FAILED unit with a leading ●, so field 1 is the bullet, not the name; the sanitiser then reduced it to empty and dropped the row. The parser silently lost exactly the rows that matter. Fix: --plain.

3. grep -ic restic backup-freshness-alert.sh → 0 (ba60fda). That is the WRAPPER. The checking lives in check-backup-freshness.sh, which had always checked seven hosts. I grepped one file of a two-file tool and generalised to the whole system — the identical error svos-dev had made with dev-backup four hours earlier, made by me while correcting them for it.

4. restic ls always prints a header line (ba60fda). Absent path returns 1 line, real path returned 6; counting with grep -c . read the header as a hit, so the ABSENT branch fell through and reported "index intact, blobs gone". An operator would have been sent to a repo-corruption runbook by a mistyped path — wrong in the most expensive available direction. Found ONLY by running the negative control. Fix: grep -c '^/'.

5. A cooldown test that invoked the script before failing the unit (02a6fc0). The two calls legitimately saw different states, computed different fingerprints, and the suppression correctly did not fire — so the test "proved" a bug that did not exist. The code was right; the test was wrong.

Adjacent, same family: "nothing to count is not the same as not counting." An NRestarts probe read flat after a manual restart and would have concluded a sampler goes blind whenever a human touches a unit. Artifact of the test — that unit had already hit its start limit and stopped auto-restarting. The instrument was reading correctly from a subject that had stopped producing.

The rule this cost

Not "verify". Verify, then ask what the verification could not have seen.

svos-dev sharpened it one notch and the sharpening is the useful half:

The rung where you are CHECKING an instrument is the rung where you are least likely to check, because you are already in verification posture and it feels like the checking is happening.

Why it is written down

Four of the five were mine, one was svos-dev's, and mine included making theirs. Neither of us is careless — we each checked one instrument and generalised from it, on the same day, while cataloguing the habit. A rule that both parties broke while discussing the rule is a rule that needs to be written where a fresh session will read it, not left as a lesson someone remembers.

Related: 2026-09-21-booth-two-dead-controls — same family, found by the operator rather than by me, because I read templates instead of rendering them.

Addendum — the sixth, and the one that cost an operator approval

[2026-09-22, later] Found by svos-dev verifying a mechanism I had merely inferred, and it corrects my own framing.

I told the operator that D-0010's /clear "cost nothing by luck of the delivery mechanism". It was not luck. svos/server/bifrost.py:1322 composes typed_text=f"[MIRANDA {card_id}] {text}" — every relayed string is prefixed with the card id, so a slash command never begins the line and the receiving harness reads it as prose. pane_send structurally cannot execute a harness command in a target pane. A property of the tool, on every send.

Which makes the card worse than a misroute. It promised: "Approve to type /clear into terminal_2 and clear booth-dev's context." That effect was never achievable — not with the right pane, not with identity resolved perfectly. And the executed event recorded {"sent": true}: true about the bytes, false about the effect.

Three defects in one approved action — wrong pane, an impossible promise, and a receipt reporting success for an outcome that never occurred. Only the first was visible to anyone.

Why this one is the sharpest of the set. Every other instrument error here cost an agent some triage time. This one consumed an operator approval: Vuong approved an action believing it would take effect, and the system reported that it had. A receipt that describes the bytes rather than the outcome is the same class as restic ls's header line or systemd's ● — an instrument reporting cleanly about something it never checked — but pointed at the human whose judgement the whole approval gate exists to capture.

The general form, worth carrying beyond SVOS: {"sent": true} is a claim about transmission, never about effect. Any receipt that cannot distinguish the two will eventually report success for something that did not happen.

Fix is svos-dev's (identity asserted where telemetry can claim the pane, disclosed as title-only where it cannot; and pane_send refusing or the card declaring text whose leading token is a harness command it cannot deliver).