Operator supplied the topology: "the entire rack is on the same circuit,
public ip is served by firewall on the same circuit. load tripped
breaker, entire rack goes dark."
That inverts the argument I committed one commit ago in 3cc55b4. I held
the load attribution partly because a site-wide blackout looked like too
large a blast radius for one GPU box. It is not — it is exactly the
blast radius a shared circuit predicts. The observation I treated as
unexplained was the signature of the mechanism.
Kept the wrong version visible with a superseded marker rather than
editing it away, per the superseded-claims convention: the reasoning
circulated in two althing messages and a peer adopted it.
Holding the attribution as unmeasured was still correct; the argument
used to hold it was reasoning about wiring neither side had seen and
neither side asked for.
Records the finding that outlives the incident: the same circuit powers
ana-ml2, ana-gw, ana-wg, ana-nas and the BMCs, so the load takes every
recovery path with it. Composed with ana-gw's WAN admin being closed
(tunnel-only) and the tunnel terminating on ana-wg, there is no remote
management path to Anaheim during a rack power event, in-band or out.
Open question for the weekend triage: is any of that rack on UPS or a
second circuit?
The record read "power capacity is the open item" next to ana-ml2's
~600 W, which reads as a cause. It is not one. The trip and its timing
are measured; the attribution to the training load is the operator's
working read and the reason for the weekend triage.
The observation that makes the single-load story incomplete on its own
terms: a site-wide blackout is a larger blast radius than one GPU box
accounts for. If ana-ml2's draw were the whole story, ana-nas, ana-wg
and the public address would not have gone dark with it.
Shedding seats may still be the right first move and it is cheap. That
is not the same as having identified what loaded the circuit, and the
distinction matters going into a triage that will act on it.
brokkr-smithy-dev asked how many times 3c was launched rather than
reconstructing it, and their reading was three. It was two.
#1 17:53:33 PDT killed by the power loss at step 80/604
#2 20:58:41 PDT stopped deliberately at 21:07:40, healthy
The phantom third came from a report I wrote at 23:03 narrating the
21:07 kill in the present tense with no timestamp. Every fact in it was
accurate; it was unreadable in sequence against a correctly-observed
22:46 snapshot of an idle GPU.
Evidence is ZFS birth times (a `>` redirect truncates the log but keeps
its birth, so mtime alone cannot separate "rewritten" from "created"),
plus the absence of any mtime under /tank/erp-tune after 21:07:34 — a
relaunch would have rewritten three files there.
Also pins the outage window to 18:14:45-18:17:00 PDT and corrects the
downtime from "~90 minutes" to 1h58m: the last journald entry before a
hard power loss is the last time anything wanted to log, not the moment
of the loss, and here it was 20 minutes early.
Corrects the in-flight header (step 22 -> last-logged step 24, stop
deliberate) and its stale "as of" stamp.
Run 3 trained, gated and dispositioned do-not-serve on a measured 44pp self-harm
guardrail regression that its own preregistered rule passed -- a pooled preserve-list
test cannot see a single-axis collapse. Run 3c (lr 20x cut, single variable) launched,
killed by an Anaheim power-breaker trip at step 80, relaunched, then stopped by the
operator at step 22 pending a weekend power triage.
Also captured: the corpus mix was specified in a unit the optimiser never sees
(45.8% dialogue by context, 24.2% by loss); the dose-response says benefit and damage
are one direction in weight space, so the merge-back measures the problem rather than
fixing it; four guests including the storage SPOF had onboot unset and never came back
from the outage, now fixed with dependency ordering; and a transport failure that
enters a measurement as a value looks like whatever you hoped to find -- which found a
live defect in another agent's instrument an hour after it was reported.
Auto-archived 8 entries to archival-memory.md (Recent decisions: 8, Tried and
abandoned: 0); 4 held back on open deferred-work pointers.
vh