docs(alerts): reset-failed sits on the remediation path, which a flap sampler must design for

svos-dev found an interaction between the two detectors by doing the recovery
rather than reasoning about it: clearing a failed unit is REQUIRED before
systemd will start it again, and reset-failed zeroes NRestarts in the same
motion. Verified here -- a start-limited unit refused to start until
reset-failed, which took NRestarts 3 -> 0.

So the standard recovery from a failed-START alarm erases the history a flap
sampler would depend on, routinely rather than as an edge case. A unit that
fails, is cleared, then flaps starts its history from zero, delaying the first
flap alarm by the whole threshold -- exactly the window where a just-fixed unit
is most likely still sick. Mitigation recorded: store a last-seen timestamp so
a counter going BACKWARDS is itself detectable as a reset rather than read as
quiet.

Also recorded as its own shape: 'nothing to count is not the same as not
counting'. The first manual-restart check read NRestarts flat and would have
concluded a sampler goes blind whenever a human touches a unit. Artifact of the
test -- that unit had already hit its start limit and stopped auto-restarting,
so the instrument was reading correctly from a subject that had stopped
producing. Verify, then ask what the verification could not have seen.
This commit is contained in:
vh
2026-09-22 08:59:21 -07:00
parent 163bb97dea
commit 00a03921ff
+38
View File
@@ -101,3 +101,41 @@ Neither failed randomly. Both failure modes were *perfectly anti-correlated
with the purpose*: the healthier the fleet, the better they appeared to work.
That is worse than a broken instrument, which at least fails visibly on the
easy cases too.
## If a flap sampler is ever built, design for these first
The uncovered class above is detectable but not *eventable*: a flapping unit
increments `NRestarts` on every automatic restart while never entering `failed`.
That needs a sampler (timer reads the counter, alarms on delta), not a hook.
Design notes from svos-dev, with what was measured here rather than assumed:
| property | measured 2026-09-22 |
|---|---|
| survives `daemon-reload` | **yes** (4 → 4) |
| survives a manual `restart` | **yes**, does not reset |
| keeps counting after one | **yes** (3 → 4 → 7 → 10) |
| `reset-failed` | **zeroes it** (4 → 0) |
| stop → start | resets |
⚠ **`reset-failed` is on the REMEDIATION PATH of the other alarm, so the reset
is routine rather than an edge case.** A unit that exhausts its start limit
lands in `failed` and systemd refuses further starts until it is cleared —
verified: a start-limited unit could only be restarted after `reset-failed`,
which zeroed `NRestarts` from 3 to 0 in the same motion. So the standard
recovery from a failed-START alarm erases the history a flap sampler depends on,
for that unit, every time.
Consequence to design for rather than discover: a unit that fails, is cleared,
then begins flapping starts its flap history from zero, so the first alarm after
a reset is delayed by the whole threshold — precisely the window in which a
just-"fixed" unit is most likely to still be sick. **Mitigation (svos-dev): store
a last-seen timestamp beside the count, so a counter going BACKWARDS is itself a
detectable event** and can be recorded as a reset rather than read as quiet.
⚠ **"Nothing to count is not the same as not counting."** The first run of the
manual-restart check showed `NRestarts` flat and would have concluded a sampler
goes blind whenever a human touches a unit. It was an artifact of the test: that
unit had burst 5 per 10s, had already hit its limit, and had stopped
auto-restarting — so there was nothing to increment. The instrument was reading
correctly from a subject that had stopped producing. Its own shape, alongside
the `●` bug: **verify, then ask what the verification could not have seen.**