# Five instrument errors in one day — and they are one shape

`[2026-09-22]` · infra-ops, in a running exchange with svos-dev

## The shape

> **A tool that enumerates "things that are fine" and acts on them has selected
> against its own subject.** (svos-dev's framing, taken close to verbatim.)

None of these failed randomly. Several had failure modes *anti-correlated with
their purpose* — the healthier the fleet looked, the better they appeared to
work. That is worse than an ordinary broken instrument, which at least fails
visibly on the easy cases too.

## The five

1. **`--state=running` in `install.sh`** (`ef70b2f`). Selected only healthy
   units to hook, so a unit already down at install time was never hooked. The
   unit most needing the alarm is by definition the one that is not healthy.
   Found because `talk.service` was sitting in `failed` and uncovered.

2. **`awk '{print $1}'` over `systemctl list-units`** (`ef70b2f`). systemd
   decorates a FAILED unit with a leading `●`, so field 1 is the bullet, not
   the name; the sanitiser then reduced it to empty and dropped the row. The
   parser silently lost *exactly the rows that matter*. Fix: `--plain`.

3. **`grep -ic restic backup-freshness-alert.sh` → 0** (`ba60fda`). That is the
   WRAPPER. The checking lives in `check-backup-freshness.sh`, which had always
   checked seven hosts. I grepped one file of a two-file tool and generalised to
   the whole system — **the identical error svos-dev had made with `dev-backup`
   four hours earlier, made by me while correcting them for it.**

4. **`restic ls` always prints a header line** (`ba60fda`). Absent path returns
   1 line, real path returned 6; counting with `grep -c .` read the header as a
   hit, so the ABSENT branch fell through and reported *"index intact, blobs
   gone"*. An operator would have been sent to a repo-corruption runbook by a
   mistyped path — wrong in the most expensive available direction. Found ONLY
   by running the negative control. Fix: `grep -c '^/'`.

5. **A cooldown test that invoked the script before failing the unit**
   (`02a6fc0`). The two calls legitimately saw different states, computed
   different fingerprints, and the suppression correctly did not fire — so the
   test "proved" a bug that did not exist. The code was right; the test was
   wrong.

Adjacent, same family: **"nothing to count is not the same as not counting."**
An `NRestarts` probe read flat after a manual restart and would have concluded a
sampler goes blind whenever a human touches a unit. Artifact of the test — that
unit had already hit its start limit and stopped auto-restarting. The instrument
was reading correctly from a subject that had stopped producing.

## The rule this cost

**Not "verify". *Verify, then ask what the verification could not have seen.***

svos-dev sharpened it one notch and the sharpening is the useful half:

> The rung where you are CHECKING an instrument is the rung where you are least
> likely to check, because you are already in verification posture and it feels
> like the checking is happening.

## Why it is written down

Four of the five were mine, one was svos-dev's, and mine included making theirs.
Neither of us is careless — we each checked one instrument and generalised from
it, on the same day, while cataloguing the habit. A rule that both parties broke
while discussing the rule is a rule that needs to be written where a fresh
session will read it, not left as a lesson someone remembers.

Related: [[2026-09-21-booth-two-dead-controls]] — same family, found by the
operator rather than by me, because I read templates instead of rendering them.