# althing-notify-failure — failed-START alarms for fleet user units

A systemd `OnFailure` template that reports any unit entering **failed** state
to the althing **infra-ops** inbox. Installed on nh3-dev; hooks 12 user units.

```sh
services/althing-notify-failure/install.sh --dry-run   # see what it would hook
services/althing-notify-failure/install.sh             # idempotent; re-run to pick up new units
```

## The failure it exists to catch

An uptime check cannot see the dangerous one. Measured by svos-dev, 2026-09-22:
a config change on 09-19 made `svos.service` refuse to boot, **the running
process predated the change and kept serving**, and the service sat one restart
from dark for three days. Every uptime probe was green and correct the whole
time — the thing *was* up. Nothing would have fired until the next restart, and
then it would have been an outage rather than a warning.

The signal that catches it is **failed-START, not down**. A count taken after
that conversation:

> **13 running user units on nh3-dev. Zero with an `OnFailure` hook.**

Including `althing-po-herald` — whose silent failure cuts infra-ops's own mail
delivery, a blind spot in the notification path every other alarm depends on.

## Why this is not noise

`OnFailure` does not fire on a clean restart or a deliberate stop. svos-dev's
four restarts and three deploys in one day would have produced **zero** alerts.
With `Restart=on-failure`, a unit enters failed state only after exhausting its
start-limit burst, so a crash-loop yields **one** message per episode.

## Two traps, both hit during the build

⚠ **`%i`, never `%I`.** `%I` *unescapes* the instance name and systemd escaping
maps `-` to `/`. The acceptance test fired with `%I` and delivered a message for
`onfailure/selftest.service` — a unit that does not exist — with a spool path
that tried to create directories. `althing-po-herald.service` would have arrived
as `althing/po/herald.service`. The referring unit passes `%n` **raw**, so the
literal `%i` is correct.

⚠ **The notifier must never report itself.** Guarded twice on purpose: the
template carries no `OnFailure` of its own, and the script bails on an instance
name matching itself. A notification loop is the one bug that pages you forever.

## Delivery, honestly

The script writes a **durable local record first**
(`~/.local/state/althing-notify-failure/`), then sends. `postbox` has no outbox —
a send that cannot reach the post office is dropped — and this alarm exists
precisely for moments nobody is watching, so the signal must survive the
post office being one of the things that is down.

If `althing-po-herald` is the unit that failed, the message still **reaches**
the post office (postbox talks to it directly; the herald only delivers inbound
pokes). It will not be pushed into a live session, but it is stored and the next
`postbox read` finds it — a memo to the successor, which is the point.

## Verified

Acceptance-tested by firing a real unit that exits 42, twice: once to catch the
`%I` bug, once to confirm the fix. Delivered subject
`[systemd] onfailure-selftest.service FAILED on nh3-dev`, with state, result,
exit status, timestamp and the last 25 journal lines; spool file written. The
test unit was removed afterwards.