Two defects, both found by the tool failing to cover a unit that was already
broken. talk.service has been sitting in `failed` (exit 143) while its
containers keep serving 200 on :8092 -- precisely the "unit died, the thing
keeps serving, nobody knows" case this alarm was built for -- and the installer
had not hooked it.
1. SELECTION: --state=running skipped anything not already healthy. A unit that
is down at install time never got hooked, which inverts the tool's purpose.
Now the union of enabled unit-files and all loaded units.
2. PARSING, and this is the sharper one: systemd decorates a FAILED unit with a
leading "●", so `awk '{print $1}'` returned the bullet instead of the name,
and the sanitiser reduced it to an empty string and dropped the row. The
parser silently lost exactly the rows that matter. --plain suppresses the
decoration. Same shape as every other instrument error today -- it reported
cleanly while looking at the wrong thing.
Selection now 23 units. Deliberately INCLUDES the timer-driven oneshots
(dev-backup, ha-backup, fleet-tls-cert-check, headscale-ddns,
seat-inventory-drift, brokkr-landscape-scan, soong-ci-relay): a backup or a
cert check that fails silently is the same class, and all seven were clean at
install time so they are not a known noise source. EXCLUDES dbus, gpg-agent and
dirmngr as OS plumbing, the notifier's own template and instances, and
svos-failed-alarm -- hooking an alarm with an alarm is a loop with extra steps.
svos-dev challenged a claim this README made -- that a crash-loop yields one
message per episode -- with a measurement: one 2026-09-19 boot-gate refusal on
svos.service produced FIVE transitions into failed, and the operator got five
messages.
Measured here before accepting it, because a peer's number is still a number
someone else took: a unit with Restart=on-failure, burst 3, interval 30s
produced 7 journal failure lines and exactly ONE notifier invocation. So the
multiplier is not universal -- it needs retries spanning start-limit windows or
an external restarter. svos.service carries StartLimitIntervalSec=5min, which
is how it accumulated five.
Both conditions exist on this box, so the guard goes in as cheap insurance
rather than as a fix for something proven here. The README now states both
numbers and which restart policy each of the twelve units carries, since that
is what decides the exposure. Noted with it: hermes-gateway has start limiting
DISABLED, so it retries forever and may never reach failed at all -- worth
knowing before trusting this alarm to cover it.
Design, taken from svos-dev's shape:
- Keyed on a hash of the CAUSE (unit + result + exit status + the shape of
its last error lines), never the unit name alone. A genuinely different
failure inside the window is a new fact and must still page; suppressing by
unit would hide a second, worse failure behind the first.
- Suppression is LOGGED to suppressed.log, never silent. An alarm that
quietly declines to fire is indistinguishable from one that is broken.
⚠ The first test of this appeared to show the cooldown not working, and the
test was wrong rather than the code -- it invoked the script BEFORE failing the
unit, so the two calls legitimately saw different states and computed different
fingerprints. Re-run the way systemd actually invokes it: same cause 3x -> 1
sent, 2 suppressed and logged; a different cause inside the same window -> sent.
svos-dev found the failure neither Beszel nor Uptime Kuma can see, and
measured it: a config change on 09-19 made svos.service refuse to boot, the
RUNNING process predated the change and kept serving, and the service sat one
restart from dark for three days. Every uptime probe was green and correct --
the thing was up. The signal that catches this is failed-START, not down.
A count after that conversation: 13 running user units on nh3-dev, ZERO with
an OnFailure hook. Including althing-po-herald, whose silent failure cuts
infra-ops's own mail delivery -- a blind spot in the notification path every
other alarm on this fleet depends on.
One template, one drop-in, 12 units hooked (dbus excluded as systemd's own
plumbing). Not noise: OnFailure does not fire on a clean restart or a
deliberate stop, and with Restart=on-failure a crash-loop yields one message
per episode rather than one per attempt.
⚠ %i, NEVER %I -- and the acceptance test is the only reason this is right.
%I unescapes the instance name and systemd escaping maps "-" to "/", so the
first run delivered a message for "onfailure/selftest.service", a unit that
does not exist, with a spool path that tried to create directories.
althing-po-herald.service would have arrived as althing/po/herald.service.
It "worked" -- mail was delivered -- which is exactly the kind of success that
is not one. The referring unit passes %n raw, so literal %i is correct.
⚠ The notifier never reports itself, guarded twice on purpose: the template
carries no OnFailure, and the script bails on its own instance name. A
notification loop is the one bug that pages you forever.
Delivery is spool-first. postbox has no outbox, and an alarm for moments
nobody is watching must survive the post office being one of the things that
is down. If the herald itself fails, the message still REACHES the post office
(postbox talks to it directly; the herald only delivers inbound pokes) -- not
pushed, but stored for the next read.
Acceptance-tested twice against a unit that exits 42: once to catch the %I
bug, once to confirm the fix. Test unit removed.
vh