From 00a03921ff20cd05f1f9e43245d607e34c331c82 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 22 Sep 2026 08:59:21 -0700 Subject: [PATCH] docs(alerts): reset-failed sits on the remediation path, which a flap sampler must design for svos-dev found an interaction between the two detectors by doing the recovery rather than reasoning about it: clearing a failed unit is REQUIRED before systemd will start it again, and reset-failed zeroes NRestarts in the same motion. Verified here -- a start-limited unit refused to start until reset-failed, which took NRestarts 3 -> 0. So the standard recovery from a failed-START alarm erases the history a flap sampler would depend on, routinely rather than as an edge case. A unit that fails, is cleared, then flaps starts its history from zero, delaying the first flap alarm by the whole threshold -- exactly the window where a just-fixed unit is most likely still sick. Mitigation recorded: store a last-seen timestamp so a counter going BACKWARDS is itself detectable as a reset rather than read as quiet. Also recorded as its own shape: 'nothing to count is not the same as not counting'. The first manual-restart check read NRestarts flat and would have concluded a sampler goes blind whenever a human touches a unit. Artifact of the test -- that unit had already hit its start limit and stopped auto-restarting, so the instrument was reading correctly from a subject that had stopped producing. Verify, then ask what the verification could not have seen. --- services/althing-notify-failure/README.md | 38 +++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/services/althing-notify-failure/README.md b/services/althing-notify-failure/README.md index 8f374e0..0874586 100644 --- a/services/althing-notify-failure/README.md +++ b/services/althing-notify-failure/README.md @@ -101,3 +101,41 @@ Neither failed randomly. Both failure modes were *perfectly anti-correlated with the purpose*: the healthier the fleet, the better they appeared to work. That is worse than a broken instrument, which at least fails visibly on the easy cases too. + +## If a flap sampler is ever built, design for these first + +The uncovered class above is detectable but not *eventable*: a flapping unit +increments `NRestarts` on every automatic restart while never entering `failed`. +That needs a sampler (timer reads the counter, alarms on delta), not a hook. +Design notes from svos-dev, with what was measured here rather than assumed: + +| property | measured 2026-09-22 | +|---|---| +| survives `daemon-reload` | **yes** (4 → 4) | +| survives a manual `restart` | **yes**, does not reset | +| keeps counting after one | **yes** (3 → 4 → 7 → 10) | +| `reset-failed` | **zeroes it** (4 → 0) | +| stop → start | resets | + +⚠ **`reset-failed` is on the REMEDIATION PATH of the other alarm, so the reset +is routine rather than an edge case.** A unit that exhausts its start limit +lands in `failed` and systemd refuses further starts until it is cleared — +verified: a start-limited unit could only be restarted after `reset-failed`, +which zeroed `NRestarts` from 3 to 0 in the same motion. So the standard +recovery from a failed-START alarm erases the history a flap sampler depends on, +for that unit, every time. + +Consequence to design for rather than discover: a unit that fails, is cleared, +then begins flapping starts its flap history from zero, so the first alarm after +a reset is delayed by the whole threshold — precisely the window in which a +just-"fixed" unit is most likely to still be sick. **Mitigation (svos-dev): store +a last-seen timestamp beside the count, so a counter going BACKWARDS is itself a +detectable event** and can be recorded as a reset rather than read as quiet. + +⚠ **"Nothing to count is not the same as not counting."** The first run of the +manual-restart check showed `NRestarts` flat and would have concluded a sampler +goes blind whenever a human touches a unit. It was an artifact of the test: that +unit had burst 5 per 10s, had already hit its limit, and had stopped +auto-restarting — so there was nothing to increment. The instrument was reading +correctly from a subject that had stopped producing. Its own shape, alongside +the `●` bug: **verify, then ask what the verification could not have seen.**