# althing-notify-failure — failed-START alarms for fleet user units

A systemd `OnFailure` template that reports any unit entering **failed** state
to the althing **infra-ops** inbox. Installed on nh3-dev; hooks 12 user units.

```sh
services/althing-notify-failure/install.sh --dry-run   # see what it would hook
services/althing-notify-failure/install.sh             # idempotent; re-run to pick up new units
```

## The failure it exists to catch

An uptime check cannot see the dangerous one. Measured by svos-dev, 2026-09-22:
a config change on 09-19 made `svos.service` refuse to boot, **the running
process predated the change and kept serving**, and the service sat one restart
from dark for three days. Every uptime probe was green and correct the whole
time — the thing *was* up. Nothing would have fired until the next restart, and
then it would have been an outage rather than a warning.

The signal that catches it is **failed-START, not down**. A count taken after
that conversation:

> **13 running user units on nh3-dev. Zero with an `OnFailure` hook.**

Including `althing-po-herald` — whose silent failure cuts infra-ops's own mail
delivery, a blind spot in the notification path every other alarm depends on.

## Why this is not noise

`OnFailure` does not fire on a clean restart or a deliberate stop. svos-dev's
four restarts and three deploys in one day would have produced **zero** alerts.

### Duplicate suppression, and a claim I had to correct

This README originally asserted that a crash-loop "yields one message per
episode". svos-dev challenged it with a measurement: one 2026-09-19 boot-gate
refusal on `svos.service` produced **five** transitions into `failed`, and the
operator got five messages.

Measured here before accepting or rejecting it: a unit with
`Restart=on-failure`, burst 3, interval 30s produced **7 journal failure lines
and exactly 1 notifier invocation**. So the multiplier is *not* universal — it
needs retries spanning start-limit windows, or an external restarter. svos.service
has `StartLimitIntervalSec=5min`, which is how it accumulated five.

Both conditions exist on this box, so there is now a **cooldown** (900s default,
`NOTIFY_COOLDOWN_SECONDS`) — cheap insurance rather than a fix for something
proven here. Two properties it must have, both tested:

- **Keyed on the CAUSE, not the unit.** A genuinely different failure inside the
  window is a new fact and still pages. Suppressing by unit name alone would
  hide a second, worse failure behind the first.
- **Suppression is logged** (`suppressed.log`), never silent. An alarm that
  quietly declines to fire is indistinguishable from one that is broken, and
  this whole mechanism exists because something that looked fine was not.

Verified: same cause fired 3× → 1 sent, 2 suppressed and logged; a different
cause inside the same window → sent.

### Restart policy decides the exposure

| policy | units |
|---|---|
| `Restart=always` | althing-po-herald, hermes-gateway, lrpg-demo, ttyd-caddy, ttyd-ro, ttyd-rw |
| `Restart=on-failure` | althing-seat-page, booth, peedlar, svos, wherethef, zellij-web |

`hermes-gateway` has `StartLimitIntervalUSec=0` — start limiting disabled, so it
retries forever and may never reach `failed` at all. Worth knowing before
trusting this alarm to cover it.

## Two traps, both hit during the build

⚠ **`%i`, never `%I`.** `%I` *unescapes* the instance name and systemd escaping
maps `-` to `/`. The acceptance test fired with `%I` and delivered a message for
`onfailure/selftest.service` — a unit that does not exist — with a spool path
that tried to create directories. `althing-po-herald.service` would have arrived
as `althing/po/herald.service`. The referring unit passes `%n` **raw**, so the
literal `%i` is correct.

⚠ **The notifier must never report itself.** Guarded twice on purpose: the
template carries no `OnFailure` of its own, and the script bails on an instance
name matching itself. A notification loop is the one bug that pages you forever.

## Delivery, honestly

The script writes a **durable local record first**
(`~/.local/state/althing-notify-failure/`), then sends. `postbox` has no outbox —
a send that cannot reach the post office is dropped — and this alarm exists
precisely for moments nobody is watching, so the signal must survive the
post office being one of the things that is down.

If `althing-po-herald` is the unit that failed, the message still **reaches**
the post office (postbox talks to it directly; the herald only delivers inbound
pokes). It will not be pushed into a live session, but it is stored and the next
`postbox read` finds it — a memo to the successor, which is the point.

## Verified

Acceptance-tested by firing a real unit that exits 42, twice: once to catch the
`%I` bug, once to confirm the fix. Delivered subject
`[systemd] onfailure-selftest.service FAILED on nh3-dev`, with state, result,
exit status, timestamp and the last 25 journal lines; spool file written. The
test unit was removed afterwards.