vh
02a6fc019d
svos-dev challenged a claim this README made -- that a crash-loop yields one
message per episode -- with a measurement: one 2026-09-19 boot-gate refusal on
svos.service produced FIVE transitions into failed, and the operator got five
messages.
Measured here before accepting it, because a peer's number is still a number
someone else took: a unit with Restart=on-failure, burst 3, interval 30s
produced 7 journal failure lines and exactly ONE notifier invocation. So the
multiplier is not universal -- it needs retries spanning start-limit windows or
an external restarter. svos.service carries StartLimitIntervalSec=5min, which
is how it accumulated five.
Both conditions exist on this box, so the guard goes in as cheap insurance
rather than as a fix for something proven here. The README now states both
numbers and which restart policy each of the twelve units carries, since that
is what decides the exposure. Noted with it: hermes-gateway has start limiting
DISABLED, so it retries forever and may never reach failed at all -- worth
knowing before trusting this alarm to cover it.
Design, taken from svos-dev's shape:
- Keyed on a hash of the CAUSE (unit + result + exit status + the shape of
its last error lines), never the unit name alone. A genuinely different
failure inside the window is a new fact and must still page; suppressing by
unit would hide a second, worse failure behind the first.
- Suppression is LOGGED to suppressed.log, never silent. An alarm that
quietly declines to fire is indistinguishable from one that is broken.
⚠ The first test of this appeared to show the cooldown not working, and the
test was wrong rather than the code -- it invoked the script BEFORE failing the
unit, so the two calls legitimately saw different states and computed different
fingerprints. Re-run the way systemd actually invokes it: same cause 3x -> 1
sent, 2 suppressed and logged; a different cause inside the same window -> sent.
althing-notify-failure — failed-START alarms for fleet user units
A systemd OnFailure template that reports any unit entering failed state
to the althing infra-ops inbox. Installed on nh3-dev; hooks 12 user units.
services/althing-notify-failure/install.sh --dry-run # see what it would hook
services/althing-notify-failure/install.sh # idempotent; re-run to pick up new units
The failure it exists to catch
An uptime check cannot see the dangerous one. Measured by svos-dev, 2026-09-22:
a config change on 09-19 made svos.service refuse to boot, the running
process predated the change and kept serving, and the service sat one restart
from dark for three days. Every uptime probe was green and correct the whole
time — the thing was up. Nothing would have fired until the next restart, and
then it would have been an outage rather than a warning.
The signal that catches it is failed-START, not down. A count taken after that conversation:
13 running user units on nh3-dev. Zero with an
OnFailurehook.
Including althing-po-herald — whose silent failure cuts infra-ops's own mail
delivery, a blind spot in the notification path every other alarm depends on.
Why this is not noise
OnFailure does not fire on a clean restart or a deliberate stop. svos-dev's
four restarts and three deploys in one day would have produced zero alerts.
Duplicate suppression, and a claim I had to correct
This README originally asserted that a crash-loop "yields one message per
episode". svos-dev challenged it with a measurement: one 2026-09-19 boot-gate
refusal on svos.service produced five transitions into failed, and the
operator got five messages.
Measured here before accepting or rejecting it: a unit with
Restart=on-failure, burst 3, interval 30s produced 7 journal failure lines
and exactly 1 notifier invocation. So the multiplier is not universal — it
needs retries spanning start-limit windows, or an external restarter. svos.service
has StartLimitIntervalSec=5min, which is how it accumulated five.
Both conditions exist on this box, so there is now a cooldown (900s default,
NOTIFY_COOLDOWN_SECONDS) — cheap insurance rather than a fix for something
proven here. Two properties it must have, both tested:
- Keyed on the CAUSE, not the unit. A genuinely different failure inside the window is a new fact and still pages. Suppressing by unit name alone would hide a second, worse failure behind the first.
- Suppression is logged (
suppressed.log), never silent. An alarm that quietly declines to fire is indistinguishable from one that is broken, and this whole mechanism exists because something that looked fine was not.
Verified: same cause fired 3× → 1 sent, 2 suppressed and logged; a different cause inside the same window → sent.
Restart policy decides the exposure
| policy | units |
|---|---|
Restart=always |
althing-po-herald, hermes-gateway, lrpg-demo, ttyd-caddy, ttyd-ro, ttyd-rw |
Restart=on-failure |
althing-seat-page, booth, peedlar, svos, wherethef, zellij-web |
hermes-gateway has StartLimitIntervalUSec=0 — start limiting disabled, so it
retries forever and may never reach failed at all. Worth knowing before
trusting this alarm to cover it.
Two traps, both hit during the build
⚠ %i, never %I. %I unescapes the instance name and systemd escaping
maps - to /. The acceptance test fired with %I and delivered a message for
onfailure/selftest.service — a unit that does not exist — with a spool path
that tried to create directories. althing-po-herald.service would have arrived
as althing/po/herald.service. The referring unit passes %n raw, so the
literal %i is correct.
⚠ The notifier must never report itself. Guarded twice on purpose: the
template carries no OnFailure of its own, and the script bails on an instance
name matching itself. A notification loop is the one bug that pages you forever.
Delivery, honestly
The script writes a durable local record first
(~/.local/state/althing-notify-failure/), then sends. postbox has no outbox —
a send that cannot reach the post office is dropped — and this alarm exists
precisely for moments nobody is watching, so the signal must survive the
post office being one of the things that is down.
If althing-po-herald is the unit that failed, the message still reaches
the post office (postbox talks to it directly; the herald only delivers inbound
pokes). It will not be pushed into a live session, but it is stored and the next
postbox read finds it — a memo to the successor, which is the point.
Verified
Acceptance-tested by firing a real unit that exits 42, twice: once to catch the
%I bug, once to confirm the fix. Delivered subject
[systemd] onfailure-selftest.service FAILED on nh3-dev, with state, result,
exit status, timestamp and the last 25 journal lines; spool file written. The
test unit was removed afterwards.