vh
1be39d1ea2
svos-dev found the failure neither Beszel nor Uptime Kuma can see, and measured it: a config change on 09-19 made svos.service refuse to boot, the RUNNING process predated the change and kept serving, and the service sat one restart from dark for three days. Every uptime probe was green and correct -- the thing was up. The signal that catches this is failed-START, not down. A count after that conversation: 13 running user units on nh3-dev, ZERO with an OnFailure hook. Including althing-po-herald, whose silent failure cuts infra-ops's own mail delivery -- a blind spot in the notification path every other alarm on this fleet depends on. One template, one drop-in, 12 units hooked (dbus excluded as systemd's own plumbing). Not noise: OnFailure does not fire on a clean restart or a deliberate stop, and with Restart=on-failure a crash-loop yields one message per episode rather than one per attempt. ⚠ %i, NEVER %I -- and the acceptance test is the only reason this is right. %I unescapes the instance name and systemd escaping maps "-" to "/", so the first run delivered a message for "onfailure/selftest.service", a unit that does not exist, with a spool path that tried to create directories. althing-po-herald.service would have arrived as althing/po/herald.service. It "worked" -- mail was delivered -- which is exactly the kind of success that is not one. The referring unit passes %n raw, so literal %i is correct. ⚠ The notifier never reports itself, guarded twice on purpose: the template carries no OnFailure, and the script bails on its own instance name. A notification loop is the one bug that pages you forever. Delivery is spool-first. postbox has no outbox, and an alarm for moments nobody is watching must survive the post office being one of the things that is down. If the herald itself fails, the message still REACHES the post office (postbox talks to it directly; the herald only delivers inbound pokes) -- not pushed, but stored for the next read. Acceptance-tested twice against a unit that exits 42: once to catch the %I bug, once to confirm the fix. Test unit removed.
3.2 KiB
althing-notify-failure — failed-START alarms for fleet user units
A systemd OnFailure template that reports any unit entering failed state
to the althing infra-ops inbox. Installed on nh3-dev; hooks 12 user units.
services/althing-notify-failure/install.sh --dry-run # see what it would hook
services/althing-notify-failure/install.sh # idempotent; re-run to pick up new units
The failure it exists to catch
An uptime check cannot see the dangerous one. Measured by svos-dev, 2026-09-22:
a config change on 09-19 made svos.service refuse to boot, the running
process predated the change and kept serving, and the service sat one restart
from dark for three days. Every uptime probe was green and correct the whole
time — the thing was up. Nothing would have fired until the next restart, and
then it would have been an outage rather than a warning.
The signal that catches it is failed-START, not down. A count taken after that conversation:
13 running user units on nh3-dev. Zero with an
OnFailurehook.
Including althing-po-herald — whose silent failure cuts infra-ops's own mail
delivery, a blind spot in the notification path every other alarm depends on.
Why this is not noise
OnFailure does not fire on a clean restart or a deliberate stop. svos-dev's
four restarts and three deploys in one day would have produced zero alerts.
With Restart=on-failure, a unit enters failed state only after exhausting its
start-limit burst, so a crash-loop yields one message per episode.
Two traps, both hit during the build
⚠ %i, never %I. %I unescapes the instance name and systemd escaping
maps - to /. The acceptance test fired with %I and delivered a message for
onfailure/selftest.service — a unit that does not exist — with a spool path
that tried to create directories. althing-po-herald.service would have arrived
as althing/po/herald.service. The referring unit passes %n raw, so the
literal %i is correct.
⚠ The notifier must never report itself. Guarded twice on purpose: the
template carries no OnFailure of its own, and the script bails on an instance
name matching itself. A notification loop is the one bug that pages you forever.
Delivery, honestly
The script writes a durable local record first
(~/.local/state/althing-notify-failure/), then sends. postbox has no outbox —
a send that cannot reach the post office is dropped — and this alarm exists
precisely for moments nobody is watching, so the signal must survive the
post office being one of the things that is down.
If althing-po-herald is the unit that failed, the message still reaches
the post office (postbox talks to it directly; the herald only delivers inbound
pokes). It will not be pushed into a live session, but it is stored and the next
postbox read finds it — a memo to the successor, which is the point.
Verified
Acceptance-tested by firing a real unit that exits 42, twice: once to catch the
%I bug, once to confirm the fix. Delivered subject
[systemd] onfailure-selftest.service FAILED on nh3-dev, with state, result,
exit status, timestamp and the last 25 journal lines; spool file written. The
test unit was removed afterwards.