vh
1be39d1ea2
svos-dev found the failure neither Beszel nor Uptime Kuma can see, and measured it: a config change on 09-19 made svos.service refuse to boot, the RUNNING process predated the change and kept serving, and the service sat one restart from dark for three days. Every uptime probe was green and correct -- the thing was up. The signal that catches this is failed-START, not down. A count after that conversation: 13 running user units on nh3-dev, ZERO with an OnFailure hook. Including althing-po-herald, whose silent failure cuts infra-ops's own mail delivery -- a blind spot in the notification path every other alarm on this fleet depends on. One template, one drop-in, 12 units hooked (dbus excluded as systemd's own plumbing). Not noise: OnFailure does not fire on a clean restart or a deliberate stop, and with Restart=on-failure a crash-loop yields one message per episode rather than one per attempt. ⚠ %i, NEVER %I -- and the acceptance test is the only reason this is right. %I unescapes the instance name and systemd escaping maps "-" to "/", so the first run delivered a message for "onfailure/selftest.service", a unit that does not exist, with a spool path that tried to create directories. althing-po-herald.service would have arrived as althing/po/herald.service. It "worked" -- mail was delivered -- which is exactly the kind of success that is not one. The referring unit passes %n raw, so literal %i is correct. ⚠ The notifier never reports itself, guarded twice on purpose: the template carries no OnFailure, and the script bails on its own instance name. A notification loop is the one bug that pages you forever. Delivery is spool-first. postbox has no outbox, and an alarm for moments nobody is watching must survive the post office being one of the things that is down. If the herald itself fails, the message still REACHES the post office (postbox talks to it directly; the herald only delivers inbound pokes) -- not pushed, but stored for the next read. Acceptance-tested twice against a unit that exits 42: once to catch the %I bug, once to confirm the fix. Test unit removed.
28 lines
1.4 KiB
Desktop File
28 lines
1.4 KiB
Desktop File
# Template instantiated by another unit's OnFailure=. The instance name is the
|
|
# FAILING unit (passed as %n by the referrer), so one template covers every
|
|
# service on the box.
|
|
#
|
|
# ⚠ This unit deliberately has NO OnFailure of its own. A notifier that
|
|
# reported its own failures would be reported by another notifier, and a
|
|
# notification loop is the one bug that pages you forever. The script carries
|
|
# the same guard independently.
|
|
[Unit]
|
|
Description=Report %i entering failed state to the althing infra-ops inbox
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
Environment=ALTHING_HANDLE=infra-ops
|
|
Environment=ALTHING_POST_OFFICE=http://10.100.50.40:8390
|
|
Environment=POSTBOX=%h/.local/bin/postbox
|
|
Environment=ALERT_RECIPIENT=infra-ops
|
|
# ⚠ %i, NEVER %I. %I UNESCAPES the instance name, and systemd escaping maps
|
|
# "-" to "/" -- so a hooked unit called althing-po-herald.service arrives as
|
|
# "althing/po/herald.service": wrong name in the subject, and a spool path that
|
|
# tries to create directories. Caught by the acceptance test on 2026-09-22,
|
|
# which fired with %I and delivered a message naming a unit that does not exist.
|
|
# The referring unit passes %n RAW (never escaped), so the literal %i is right.
|
|
ExecStart=%h/.local/bin/althing-notify-failure %i
|
|
# The notifier must not hang: a stuck postbox call would leave a oneshot
|
|
# activating forever and mask the very failure it was invoked for.
|
|
TimeoutStartSec=60
|