docs(alerts): reset-failed sits on the remediation path, which a flap sampler must design for
svos-dev found an interaction between the two detectors by doing the recovery rather than reasoning about it: clearing a failed unit is REQUIRED before systemd will start it again, and reset-failed zeroes NRestarts in the same motion. Verified here -- a start-limited unit refused to start until reset-failed, which took NRestarts 3 -> 0. So the standard recovery from a failed-START alarm erases the history a flap sampler would depend on, routinely rather than as an edge case. A unit that fails, is cleared, then flaps starts its history from zero, delaying the first flap alarm by the whole threshold -- exactly the window where a just-fixed unit is most likely still sick. Mitigation recorded: store a last-seen timestamp so a counter going BACKWARDS is itself detectable as a reset rather than read as quiet. Also recorded as its own shape: 'nothing to count is not the same as not counting'. The first manual-restart check read NRestarts flat and would have concluded a sampler goes blind whenever a human touches a unit. Artifact of the test -- that unit had already hit its start limit and stopped auto-restarting, so the instrument was reading correctly from a subject that had stopped producing. Verify, then ask what the verification could not have seen.
This commit is contained in:
@@ -101,3 +101,41 @@ Neither failed randomly. Both failure modes were *perfectly anti-correlated
|
||||
with the purpose*: the healthier the fleet, the better they appeared to work.
|
||||
That is worse than a broken instrument, which at least fails visibly on the
|
||||
easy cases too.
|
||||
|
||||
## If a flap sampler is ever built, design for these first
|
||||
|
||||
The uncovered class above is detectable but not *eventable*: a flapping unit
|
||||
increments `NRestarts` on every automatic restart while never entering `failed`.
|
||||
That needs a sampler (timer reads the counter, alarms on delta), not a hook.
|
||||
Design notes from svos-dev, with what was measured here rather than assumed:
|
||||
|
||||
| property | measured 2026-09-22 |
|
||||
|---|---|
|
||||
| survives `daemon-reload` | **yes** (4 → 4) |
|
||||
| survives a manual `restart` | **yes**, does not reset |
|
||||
| keeps counting after one | **yes** (3 → 4 → 7 → 10) |
|
||||
| `reset-failed` | **zeroes it** (4 → 0) |
|
||||
| stop → start | resets |
|
||||
|
||||
⚠ **`reset-failed` is on the REMEDIATION PATH of the other alarm, so the reset
|
||||
is routine rather than an edge case.** A unit that exhausts its start limit
|
||||
lands in `failed` and systemd refuses further starts until it is cleared —
|
||||
verified: a start-limited unit could only be restarted after `reset-failed`,
|
||||
which zeroed `NRestarts` from 3 to 0 in the same motion. So the standard
|
||||
recovery from a failed-START alarm erases the history a flap sampler depends on,
|
||||
for that unit, every time.
|
||||
|
||||
Consequence to design for rather than discover: a unit that fails, is cleared,
|
||||
then begins flapping starts its flap history from zero, so the first alarm after
|
||||
a reset is delayed by the whole threshold — precisely the window in which a
|
||||
just-"fixed" unit is most likely to still be sick. **Mitigation (svos-dev): store
|
||||
a last-seen timestamp beside the count, so a counter going BACKWARDS is itself a
|
||||
detectable event** and can be recorded as a reset rather than read as quiet.
|
||||
|
||||
⚠ **"Nothing to count is not the same as not counting."** The first run of the
|
||||
manual-restart check showed `NRestarts` flat and would have concluded a sampler
|
||||
goes blind whenever a human touches a unit. It was an artifact of the test: that
|
||||
unit had burst 5 per 10s, had already hit its limit, and had stopped
|
||||
auto-restarting — so there was nothing to increment. The instrument was reading
|
||||
correctly from a subject that had stopped producing. Its own shape, alongside
|
||||
the `●` bug: **verify, then ask what the verification could not have seen.**
|
||||
|
||||
Reference in New Issue
Block a user