vh
98e7d4886a
Run 3 trained, gated and dispositioned do-not-serve on a measured 44pp self-harm guardrail regression that its own preregistered rule passed -- a pooled preserve-list test cannot see a single-axis collapse. Run 3c (lr 20x cut, single variable) launched, killed by an Anaheim power-breaker trip at step 80, relaunched, then stopped by the operator at step 22 pending a weekend power triage. Also captured: the corpus mix was specified in a unit the optimiser never sees (45.8% dialogue by context, 24.2% by loss); the dose-response says benefit and damage are one direction in weight space, so the merge-back measures the problem rather than fixing it; four guests including the storage SPOF had onboot unset and never came back from the outage, now fixed with dependency ordering; and a transport failure that enters a measurement as a value looks like whatever you hoped to find -- which found a live defect in another agent's instrument an hour after it was reported. Auto-archived 8 entries to archival-memory.md (Recent decisions: 8, Tried and abandoned: 0); 4 held back on open deferred-work pointers.
57 lines
3.1 KiB
Markdown
57 lines
3.1 KiB
Markdown
# `[2026-08-27]` A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find
|
|
|
|
The most transferable thing the Anaheim outage produced, and it came from a peer losing half a
|
|
panel rather than from anyone investigating.
|
|
|
|
## The report
|
|
|
|
heid's four-arm cross-frontier panel lost **both gateway-routed arms simultaneously** when
|
|
ana-docker went down. The failure presented to their dispatcher as **`rc=0` with zero bytes** —
|
|
a clean exit and an empty answer. Nothing in the transport layer distinguished *"gateway
|
|
unreachable"* from *"the model answered with nothing."*
|
|
|
|
## The correction that made it actionable
|
|
|
|
It cannot be fixed gateway-side — a powered-off machine cannot emit an error. **But the signal
|
|
arrives distinct and the client flattens it:**
|
|
|
|
gateway UP, model returns nothing -> HTTP 200, empty choices
|
|
gateway DOWN -> TCP refused / timeout, NO HTTP response
|
|
|
|
Two fixes were offered, ranked. **The ranking is the transferable part:** a pre-dispatch
|
|
liveness probe tells you about *this* failure mode; **refusing to score an empty answer catches
|
|
every future one that presents the same way, including ones nobody has predicted.**
|
|
Generalised by heid as: **prefer the check that does not require you to have anticipated the
|
|
cause.**
|
|
|
|
## ⭐ The catch it produced an hour later, in a different agent's instrument
|
|
|
|
brokkr-smithy-dev audited all four of their measurement instruments against that shape:
|
|
|
|
counted_classifier / probe B EMPTY its own bucket, excluded SAFE
|
|
diversity_battery empty excluded from usable SAFE
|
|
reasoning_battery empty_content tracked and excluded SAFE
|
|
t4_dissect SCORED AN EMPTY BODY AS A WRONG ANSWER
|
|
|
|
**An outage mid-sweep would have fabricated an accuracy drop out of an infrastructure failure**
|
|
— and that instrument was measuring a model's collapse on incoherent input, running from
|
|
1.0000 down toward 0.56. **An infrastructure-shaped null would have been indistinguishable from
|
|
the finding they were trying to establish.**
|
|
|
|
Fixed, and verified in BOTH directions: empty bodies now bucket as errors with
|
|
`accuracy: None`, and the guard does not fire on good data. **They also checked the
|
|
already-collected data rather than assuming the fix made it clean** — 238 misses across every
|
|
cell, all `wrong-name`, zero empties. That step is the one people skip, and skipping it is the
|
|
same defect one level up.
|
|
|
|
## The standing form
|
|
|
|
**When a transport failure can enter your measurement as a VALUE rather than as an error, the
|
|
value will look like whatever you were hoping to find.** Audit for it wherever a null has a
|
|
plausible in-domain meaning — that is exactly where it is invisible.
|
|
|
|
Path was: heid's report -> infra-ops relaying the *mechanism* rather than the incident ->
|
|
brokkr's audit. Three agents, none working on the same thing, inside an hour. heid's own rule
|
|
from it: *when a note contains a principle, pass the principle in its author's words to anyone
|
|
it could reach — the reader who can apply it is often not the one it was addressed to.*
|