diff --git a/servers/nh3-dev/README.md b/servers/nh3-dev/README.md index b4243be..3c559b3 100644 --- a/servers/nh3-dev/README.md +++ b/servers/nh3-dev/README.md @@ -96,6 +96,21 @@ local Bash already executes here — no SSH-to-self needed for non-privileged wo is what SVOS's `_hermes_roster` derives its required-config line from — narrowing does not blind it. Becomes `[svos_miranda]` once SVOS's plugin lands in `$HERMES_HOME/plugins/`. + ⚠ **A deliberate restart can exit 1, and that is a Hermes race, not a crash** + (2026-09-23). `ExecStop=gateway.systemd_stop_mark` writes a planned-stop + marker; the gateway's marker-watcher thread sometimes sees it before systemd's + SIGTERM lands, runs the shutdown handler ("Received UNKNOWN as a planned + gateway stop"), and consumes the marker. The SIGTERM then runs the handler a + second time, finds no marker, and exits 1 ("signal-initiated shutdown without + restart request"). `~/.hermes/logs/gateway.log` shows both lines 26 ms apart. + Before the 09-21 Hermes update every stop exited 1; since then it is + intermittent. The exit 1 used to page infra-ops through the `OnFailure` hook, + so drop-in `hermes-gateway.service.d/20-planned-stop-exit1-is-clean.conf` + sets `SuccessExitStatus=1`. That costs nothing: `Restart=always` restarts + whatever the exit status, and `StartLimitIntervalSec=0` means the unit never + reaches `failed` on a start failure, so the hook can't catch anything real on + this unit anyway. Delete the drop-in once upstream makes the handler + idempotent. - **Hermes model backend → `gen-large` on the fleet LiteLLM gateway** (free local compute), set 2026-09-14 per operator ruling. Until then `model.default` said `anthropic/claude-opus-4.6` with `model.base_url` at openrouter, but diff --git a/services/althing-notify-failure/README.md b/services/althing-notify-failure/README.md index 0874586..ffdbcc1 100644 --- a/services/althing-notify-failure/README.md +++ b/services/althing-notify-failure/README.md @@ -30,6 +30,18 @@ delivery, a blind spot in the notification path every other alarm depends on. `OnFailure` does not fire on a clean restart or a deliberate stop. svos-dev's four restarts and three deploys in one day would have produced **zero** alerts. +⚠ **"Clean" means the main process exits with a success status when it gets +SIGTERM.** A daemon that exits non-zero on a *deliberate* `systemctl restart` +or `stop` lands the unit in `failed` for an instant, and `OnFailure` fires even +though the restart then carries on and succeeds. The page it sends reads +`state active (running)` — that combination is the tell. Measured 2026-09-23 on +a throwaway unit that exits 1 on SIGTERM: 3/3 restarts paged. Adding +`SuccessExitStatus=` to the unit brought it to 0/3, and an exit outside a +stop job still auto-restarted under `Restart=always`. A survey of every stop on +nh3-dev since 09-15 (booth 49, peedlar 30, svos 10, …) found **one** unit that +does this — `hermes-gateway`, below — so the fix is per-unit, not in the +notifier. + ### Duplicate suppression, and a claim I had to correct This README originally asserted that a crash-loop "yields one message per @@ -77,7 +89,7 @@ alarm would have fired. | **Covered** | the 7 timer-driven oneshots (dev-backup, ha-backup, fleet-tls-cert-check, headscale-ddns, seat-inventory-drift, brokkr-landscape-scan, soong-ci-relay) | `Restart=no` — any failure lands in `failed` immediately | | **Covered** | `svos.service` | burst 3 per **5min** with `RestartSec=5s` — 3 restarts fit in 15s, so it genuinely gives up | | **NOT covered** | booth, althing-po-herald, althing-seat-page, peedlar, wherethef, ttyd-caddy, ttyd-ro, ttyd-rw, lrpg-demo, zellij-web | `RestartSec` ≥ 2s against burst 5 per 10s — they flap forever instead | -| **NOT covered, definitively** | `hermes-gateway` | `StartLimitIntervalUSec=0` — start limiting disabled, it never gives up at all | +| **NOT covered, definitively** | `hermes-gateway` | `StartLimitIntervalUSec=0` — start limiting disabled, it never gives up at all. Its hook could only ever fire false positives (a stop-time exit 1, from a Hermes race; see `servers/nh3-dev/README.md`), so drop-in `20-planned-stop-exit1-is-clean.conf` sets `SuccessExitStatus=1`, 2026-09-23 | `svos.service`'s divergent 5-minute window is **deliberate and load-bearing** (operator ruling 2026-09-11, "fatal both ways"). Do **not** harmonise it to 10s: