vh
163bb97dea
OnFailure fires when a unit enters failed. A unit that keeps re-earning its retries never enters failed, so the alarm never fires for it. svos-dev warned this was the consequence of the interval divergence; measuring it showed the gap is most of the box. MEASURED: a unit shaped like booth/althing-po-herald (Restart=always, RestartSec=5s, burst 5, interval 10s) ran 9 restarts over 45 seconds cycling active/activating and NEVER reached failed. No alarm would have fired. Covered: the 7 timer-driven oneshots (Restart=no, so any failure lands in failed immediately) and svos.service (burst 3 per 5min -- it genuinely gives up). NOT covered: ten daemons that flap instead, and hermes-gateway, whose start limiting is disabled outright. svos.service's divergent 5min window is deliberate and load-bearing (operator ruling 2026-09-11, 'fatal both ways'). NOT to be harmonised: that would restore the flapping the ruling forbids AND silence the alarm on the one daemon it currently works for. Whether the OTHERS move to 5min is a behaviour change for ten services and an operator call. Also recorded, svos-dev's framing of the two installer bugs: a tool that enumerates 'things that are fine' and acts on them has selected against its own subject. Neither bug failed randomly -- both were anti-correlated with purpose, working better the healthier the fleet looked.
althing-notify-failure — failed-START alarms for fleet user units
A systemd OnFailure template that reports any unit entering failed state
to the althing infra-ops inbox. Installed on nh3-dev; hooks 12 user units.
services/althing-notify-failure/install.sh --dry-run # see what it would hook
services/althing-notify-failure/install.sh # idempotent; re-run to pick up new units
The failure it exists to catch
An uptime check cannot see the dangerous one. Measured by svos-dev, 2026-09-22:
a config change on 09-19 made svos.service refuse to boot, the running
process predated the change and kept serving, and the service sat one restart
from dark for three days. Every uptime probe was green and correct the whole
time — the thing was up. Nothing would have fired until the next restart, and
then it would have been an outage rather than a warning.
The signal that catches it is failed-START, not down. A count taken after that conversation:
13 running user units on nh3-dev. Zero with an
OnFailurehook.
Including althing-po-herald — whose silent failure cuts infra-ops's own mail
delivery, a blind spot in the notification path every other alarm depends on.
Why this is not noise
OnFailure does not fire on a clean restart or a deliberate stop. svos-dev's
four restarts and three deploys in one day would have produced zero alerts.
Duplicate suppression, and a claim I had to correct
This README originally asserted that a crash-loop "yields one message per
episode". svos-dev challenged it with a measurement: one 2026-09-19 boot-gate
refusal on svos.service produced five transitions into failed, and the
operator got five messages.
Measured here before accepting or rejecting it: a unit with
Restart=on-failure, burst 3, interval 30s produced 7 journal failure lines
and exactly 1 notifier invocation. So the multiplier is not universal — it
needs retries spanning start-limit windows, or an external restarter. svos.service
has StartLimitIntervalSec=5min, which is how it accumulated five.
Both conditions exist on this box, so there is now a cooldown (900s default,
NOTIFY_COOLDOWN_SECONDS) — cheap insurance rather than a fix for something
proven here. Two properties it must have, both tested:
- Keyed on the CAUSE, not the unit. A genuinely different failure inside the window is a new fact and still pages. Suppressing by unit name alone would hide a second, worse failure behind the first.
- Suppression is logged (
suppressed.log), never silent. An alarm that quietly declines to fire is indistinguishable from one that is broken, and this whole mechanism exists because something that looked fine was not.
Verified: same cause fired 3× → 1 sent, 2 suppressed and logged; a different cause inside the same window → sent.
⚠ COVERAGE — read this before trusting the alarm
OnFailure fires when a unit enters failed. A unit that keeps re-earning
its retries never enters failed, so the alarm never fires for it. That is
not a subtlety; it silently excludes most daemons on this box.
A unit reaches failed only by exhausting StartLimitBurst starts inside
StartLimitIntervalSec. With RestartSec=5s against burst 5 per 10s, roughly
two restarts fit in a window — the burst is unexhaustible.
Measured 2026-09-22, a unit shaped like booth / althing-po-herald
(Restart=always, RestartSec=5s, burst 5, interval 10s): 9 restarts over
45 seconds, state cycling active/activating, never once failed. No
alarm would have fired.
| coverage | units | why |
|---|---|---|
| Covered | the 7 timer-driven oneshots (dev-backup, ha-backup, fleet-tls-cert-check, headscale-ddns, seat-inventory-drift, brokkr-landscape-scan, soong-ci-relay) | Restart=no — any failure lands in failed immediately |
| Covered | svos.service |
burst 3 per 5min with RestartSec=5s — 3 restarts fit in 15s, so it genuinely gives up |
| NOT covered | booth, althing-po-herald, althing-seat-page, peedlar, wherethef, ttyd-caddy, ttyd-ro, ttyd-rw, lrpg-demo, zellij-web | RestartSec ≥ 2s against burst 5 per 10s — they flap forever instead |
| NOT covered, definitively | hermes-gateway |
StartLimitIntervalUSec=0 — start limiting disabled, it never gives up at all |
svos.service's divergent 5-minute window is deliberate and load-bearing
(operator ruling 2026-09-11, "fatal both ways"). Do not harmonise it to 10s:
that would restore the flapping the ruling forbids and silence the alarm on
the one unit that currently works. The open question is whether the others
should move to 5min — a behaviour change for ten services, so an operator
call, not a tidy-up.
The shape both of this tool's bugs shared
svos-dev's framing, worth keeping: a tool that enumerates "things that are fine" and acts on them has selected against its own subject.
install.shselected--state=running, so a unit already down at install time was never hooked — the unit most needing the alarm is by definition the one that is not healthy.- The parser took
awk '{print $1}', which returns systemd's●decoration for a failed unit, so those rows were reduced to empty and dropped.
Neither failed randomly. Both failure modes were perfectly anti-correlated with the purpose: the healthier the fleet, the better they appeared to work. That is worse than a broken instrument, which at least fails visibly on the easy cases too.