Front door: request bodies capped at 1 MiB, headers within 5 s, body within 15 s, no response deadlines (SSE stays open). Validated live: 413 over the cap, stalled body cut at 15 s, signed ping and an SSE stream unaffected. svos-front-inventory also serves the previous release's hashed assets credential-free from that release's static dir, hash-checked. Backup check: reads the logs of vzdump tasks PVE records as OK and reports ERROR lines; an fs-freeze/thaw error is an application-consistency failure, new verdict DEGRADED (exit 4), wired into the alert wrapper. nh3-pve-2 added to the node and namespace lists (its job results were never read). The svos drill task is acknowledged by UPID; BACKUP_IGNORE_ACKS=1 counts it (positive control: exit 4).
The check collapsed two different findings into one verdict. On 2026-09-20 it
printed "RESULT: STALE" while reporting 37 FRESH layers and zero stale ones --
every backup body provably current, the three ❌ rows all yesterday's pre-fix
runs aging out of the 36h window. infra-hermes caught it in triage: a reader,
or a forwarder, could page someone over a state where nothing is stale.
STALE is a claim about backup AGE. A job that ran and errored is a different
claim with different urgency. They now have different words and different exit
codes:
0 all backups fresh
1 STALE -- a body past the threshold, or an endpoint down
3 ERRORED-JOBS -- every body fresh, a vzdump job errored recently
The alert wrapper mirrors the code and matches its own wording to the finding:
🟡 "Backup jobs errored — all bodies fresh" instead of 🔴 "Backup freshness
ALERT", and it now exits with the check's code rather than flattening
everything to 1, so `systemctl status` distinguishes the states too.
This is the same defect class the rest of this script was built to fix, one
level up: not an instrument that fails to look, but one that looks correctly
and then reports the wrong word for what it saw. An alarm that cries outage
over a healthy fleet earns being ignored exactly as fast as one that stays
silent over a broken one.
Verified all three states by forcing each: BACKUP_JOB_WINDOW_HOURS=1 -> exit 0,
default -> exit 3, BACKUP_MAX_AGE_HOURS=1 -> exit 1.
The daily backup-freshness check has been unable to raise an alert since the
2026-08-28 althing v3 cutover. It called althing-cli, which v3 DELETED rather
than deprecated. The check itself never stopped working: it detected three
stale backups every morning and told nobody, and the only trace was a WARN
line inside a unit that was already reporting `failed` for the stale backups
themselves. Three weeks, silent.
Four changes, because swapping the binary alone would have left it dead:
* althing-cli -> postbox.
* Add ALTHING_POST_OFFICE to the systemd user unit AND to the installer that
writes it. postbox has no default address by design and a user unit
inherits nothing from the interactive shell, so the binary swap on its own
would have failed with a different message. Fixing only the live unit
would have been undone by the next installer run; the two are now verified
to agree.
* Recipient infra-ops -> infra-hermes. This runs AS infra-ops, so the old
address mailed the alarm to itself — the mirror trap named in CLAUDE.md.
Day-to-day checks are infra-hermes's half of the split; he escalates.
* Split the exit codes. 1 now means "backups stale, someone was told";
2 means "the alert path itself failed". A broken alarm is a worse fault
than the thing it watches and must not be indistinguishable from it.
Adds --test-alert: a positive control that sends a real message through the
real path on demand. The wire was cut for three weeks precisely because
nothing ever exercised it in the healthy state, and an alarm whose success
path is never run is not known to work.
Verified: positive control delivered; missing-address and unreachable-post-
office both correctly exit 2; a real run through systemd delivered the alert
and exited 1.
vh