memory: snapshot — run 3 gated DO-NOT-SERVE, run 3c held on a tripped breaker
Run 3 trained, gated and dispositioned do-not-serve on a measured 44pp self-harm guardrail regression that its own preregistered rule passed -- a pooled preserve-list test cannot see a single-axis collapse. Run 3c (lr 20x cut, single variable) launched, killed by an Anaheim power-breaker trip at step 80, relaunched, then stopped by the operator at step 22 pending a weekend power triage. Also captured: the corpus mix was specified in a unit the optimiser never sees (45.8% dialogue by context, 24.2% by loss); the dose-response says benefit and damage are one direction in weight space, so the merge-back measures the problem rather than fixing it; four guests including the storage SPOF had onboot unset and never came back from the outage, now fixed with dependency ordering; and a transport failure that enters a measurement as a value looks like whatever you hoped to find -- which found a live defect in another agent's instrument an hour after it was reported. Auto-archived 8 entries to archival-memory.md (Recent decisions: 8, Tried and abandoned: 0); 4 held back on open deferred-work pointers.
This commit is contained in:
@@ -0,0 +1,56 @@
|
||||
# `[2026-08-27]` A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find
|
||||
|
||||
The most transferable thing the Anaheim outage produced, and it came from a peer losing half a
|
||||
panel rather than from anyone investigating.
|
||||
|
||||
## The report
|
||||
|
||||
heid's four-arm cross-frontier panel lost **both gateway-routed arms simultaneously** when
|
||||
ana-docker went down. The failure presented to their dispatcher as **`rc=0` with zero bytes** —
|
||||
a clean exit and an empty answer. Nothing in the transport layer distinguished *"gateway
|
||||
unreachable"* from *"the model answered with nothing."*
|
||||
|
||||
## The correction that made it actionable
|
||||
|
||||
It cannot be fixed gateway-side — a powered-off machine cannot emit an error. **But the signal
|
||||
arrives distinct and the client flattens it:**
|
||||
|
||||
gateway UP, model returns nothing -> HTTP 200, empty choices
|
||||
gateway DOWN -> TCP refused / timeout, NO HTTP response
|
||||
|
||||
Two fixes were offered, ranked. **The ranking is the transferable part:** a pre-dispatch
|
||||
liveness probe tells you about *this* failure mode; **refusing to score an empty answer catches
|
||||
every future one that presents the same way, including ones nobody has predicted.**
|
||||
Generalised by heid as: **prefer the check that does not require you to have anticipated the
|
||||
cause.**
|
||||
|
||||
## ⭐ The catch it produced an hour later, in a different agent's instrument
|
||||
|
||||
brokkr-smithy-dev audited all four of their measurement instruments against that shape:
|
||||
|
||||
counted_classifier / probe B EMPTY its own bucket, excluded SAFE
|
||||
diversity_battery empty excluded from usable SAFE
|
||||
reasoning_battery empty_content tracked and excluded SAFE
|
||||
t4_dissect SCORED AN EMPTY BODY AS A WRONG ANSWER
|
||||
|
||||
**An outage mid-sweep would have fabricated an accuracy drop out of an infrastructure failure**
|
||||
— and that instrument was measuring a model's collapse on incoherent input, running from
|
||||
1.0000 down toward 0.56. **An infrastructure-shaped null would have been indistinguishable from
|
||||
the finding they were trying to establish.**
|
||||
|
||||
Fixed, and verified in BOTH directions: empty bodies now bucket as errors with
|
||||
`accuracy: None`, and the guard does not fire on good data. **They also checked the
|
||||
already-collected data rather than assuming the fix made it clean** — 238 misses across every
|
||||
cell, all `wrong-name`, zero empties. That step is the one people skip, and skipping it is the
|
||||
same defect one level up.
|
||||
|
||||
## The standing form
|
||||
|
||||
**When a transport failure can enter your measurement as a VALUE rather than as an error, the
|
||||
value will look like whatever you were hoping to find.** Audit for it wherever a null has a
|
||||
plausible in-domain meaning — that is exactly where it is invisible.
|
||||
|
||||
Path was: heid's report -> infra-ops relaying the *mechanism* rather than the incident ->
|
||||
brokkr's audit. Three agents, none working on the same thing, inside an hour. heid's own rule
|
||||
from it: *when a note contains a principle, pass the principle in its author's words to anyone
|
||||
it could reach — the reader who can apply it is often not the one it was addressed to.*
|
||||
Reference in New Issue
Block a user