esh-pfi-infrastructure/persistent-memory.d/2026-09-22-instrument-errors.md

# Five instrument errors in one day — and they are one shape

`[2026-09-22]` · infra-ops, in a running exchange with svos-dev

## The shape

> **A tool that enumerates "things that are fine" and acts on them has selected
> against its own subject.** (svos-dev's framing, taken close to verbatim.)

None of these failed randomly. Several had failure modes *anti-correlated with
their purpose* — the healthier the fleet looked, the better they appeared to
work. That is worse than an ordinary broken instrument, which at least fails
visibly on the easy cases too.

## The five

1. **`--state=running` in `install.sh`** (`ef70b2f`). Selected only healthy
   units to hook, so a unit already down at install time was never hooked. The
   unit most needing the alarm is by definition the one that is not healthy.
   Found because `talk.service` was sitting in `failed` and uncovered.

2. **`awk '{print $1}'` over `systemctl list-units`** (`ef70b2f`). systemd
   decorates a FAILED unit with a leading `●`, so field 1 is the bullet, not
   the name; the sanitiser then reduced it to empty and dropped the row. The
   parser silently lost *exactly the rows that matter*. Fix: `--plain`.

3. **`grep -ic restic backup-freshness-alert.sh` → 0** (`ba60fda`). That is the
   WRAPPER. The checking lives in `check-backup-freshness.sh`, which had always
   checked seven hosts. I grepped one file of a two-file tool and generalised to
   the whole system — **the identical error svos-dev had made with `dev-backup`
   four hours earlier, made by me while correcting them for it.**

4. **`restic ls` always prints a header line** (`ba60fda`). Absent path returns
   1 line, real path returned 6; counting with `grep -c .` read the header as a
   hit, so the ABSENT branch fell through and reported *"index intact, blobs
   gone"*. An operator would have been sent to a repo-corruption runbook by a
   mistyped path — wrong in the most expensive available direction. Found ONLY
   by running the negative control. Fix: `grep -c '^/'`.

5. **A cooldown test that invoked the script before failing the unit**
   (`02a6fc0`). The two calls legitimately saw different states, computed
   different fingerprints, and the suppression correctly did not fire — so the
   test "proved" a bug that did not exist. The code was right; the test was
   wrong.

Adjacent, same family: **"nothing to count is not the same as not counting."**
An `NRestarts` probe read flat after a manual restart and would have concluded a
sampler goes blind whenever a human touches a unit. Artifact of the test — that
unit had already hit its start limit and stopped auto-restarting. The instrument
was reading correctly from a subject that had stopped producing.

## The rule this cost

**Not "verify". *Verify, then ask what the verification could not have seen.***

svos-dev sharpened it one notch and the sharpening is the useful half:

> The rung where you are CHECKING an instrument is the rung where you are least
> likely to check, because you are already in verification posture and it feels
> like the checking is happening.

## Why it is written down

Four of the five were mine, one was svos-dev's, and mine included making theirs.
Neither of us is careless — we each checked one instrument and generalised from
it, on the same day, while cataloguing the habit. A rule that both parties broke
while discussing the rule is a rule that needs to be written where a fresh
session will read it, not left as a lesson someone remembers.

Related: [[2026-09-21-booth-two-dead-controls]] — same family, found by the
operator rather than by me, because I read templates instead of rendering them.

## Addendum — the sixth, and the one that cost an operator approval

`[2026-09-22, later]` Found by svos-dev verifying a mechanism I had merely
inferred, and it corrects my own framing.

I told the operator that D-0010's `/clear` "cost nothing by luck of the delivery
mechanism". **It was not luck.** `svos/server/bifrost.py:1322` composes
`typed_text=f"[MIRANDA {card_id}] {text}"` — every relayed string is prefixed
with the card id, so a slash command never begins the line and the receiving
harness reads it as prose. **`pane_send` structurally cannot execute a harness
command in a target pane.** A property of the tool, on every send.

Which makes the card worse than a misroute. It promised: *"Approve to type
/clear into terminal_2 and clear booth-dev's context."* **That effect was never
achievable** — not with the right pane, not with identity resolved perfectly.
And the executed event recorded `{"sent": true}`: true about the bytes, false
about the effect.

Three defects in one approved action — wrong pane, an impossible promise, and a
receipt reporting success for an outcome that never occurred. Only the first was
visible to anyone.

**Why this one is the sharpest of the set.** Every other instrument error here
cost an agent some triage time. This one *consumed an operator approval*: Vuong
approved an action believing it would take effect, and the system reported that
it had. A receipt that describes the bytes rather than the outcome is the same
class as `restic ls`'s header line or systemd's `●` — an instrument reporting
cleanly about something it never checked — but pointed at the human whose
judgement the whole approval gate exists to capture.

The general form, worth carrying beyond SVOS: **`{"sent": true}` is a claim
about transmission, never about effect.** Any receipt that cannot distinguish
the two will eventually report success for something that did not happen.

Fix is svos-dev's (identity asserted where telemetry can claim the pane,
disclosed as title-only where it cannot; and `pane_send` refusing or the card
declaring text whose leading token is a harness command it cannot deliver).