From ea818380ff5a9ec4fe973315379b089d2be8d76d Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 27 Aug 2026 11:04:42 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20the=20rack=20is=20one=20circuit=20?= =?UTF-8?q?=E2=80=94=20my=20blast-radius=20objection=20was=20wrong?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator supplied the topology: "the entire rack is on the same circuit, public ip is served by firewall on the same circuit. load tripped breaker, entire rack goes dark." That inverts the argument I committed one commit ago in 3cc55b4. I held the load attribution partly because a site-wide blackout looked like too large a blast radius for one GPU box. It is not — it is exactly the blast radius a shared circuit predicts. The observation I treated as unexplained was the signature of the mechanism. Kept the wrong version visible with a superseded marker rather than editing it away, per the superseded-claims convention: the reasoning circulated in two althing messages and a peer adopted it. Holding the attribution as unmeasured was still correct; the argument used to hold it was reasoning about wiring neither side had seen and neither side asked for. Records the finding that outlives the incident: the same circuit powers ana-ml2, ana-gw, ana-wg, ana-nas and the BMCs, so the load takes every recovery path with it. Composed with ana-gw's WAN admin being closed (tunnel-only) and the tunnel terminating on ana-wg, there is no remote management path to Anaheim during a rack power event, in-band or out. Open question for the weekend triage: is any of that rack on UPS or a second circuit? --- ...26-08-27-anaheim-breaker-and-onboot-gap.md | 53 +++++++++++++++---- persistent-memory.md | 2 +- 2 files changed, 45 insertions(+), 10 deletions(-) diff --git a/persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md b/persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md index 90cd9da..44340d6 100644 --- a/persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md +++ b/persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md @@ -66,20 +66,55 @@ Operator: *"we'll triage this weekend, probably shut down some seats."* ana-ml2 pulling ~600 W across both GPUs at their 300 W caps during training. `gen` stays up by instruction; everything else on that box is idle. -⚠ **Keep the measured and the hypothesised apart here — the load attribution is NOT established:** - MEASURED a breaker tripped (operator-confirmed) MEASURED power lost between 18:14:45 and 18:17:00 PDT MEASURED the WHOLE Anaheim site went dark from the internet + TOPOLOGY ONE CIRCUIT FEEDS THE ENTIRE RACK, including the firewall serving + the public IP (operator, 2026-08-27) - HYPOTHESIS the training load tripped it — the operator's working read and the reason - for the weekend triage. There is no per-circuit meter on that panel, and - a site-wide blackout is a larger blast radius than one GPU box's ~600 W - accounts for on its own: if ana-ml2's draw were the whole story, why did - ana-nas, ana-wg and the public address go dark with it? + OPERATOR "load tripped breaker, entire rack goes dark." Consistent with every + DETERMINATION observation. Not a per-circuit measurement, and no longer needs to be — + the operator has the site knowledge and has ruled. -**Held as the right thing to triage first, not as the cause.** Shedding GPU seats may well be -correct and is cheap; it is not the same as having identified what loaded the circuit. +⚠ **SUPERSEDED — the blast-radius objection I raised here was WRONG, and it was committed.** +An earlier revision of this section (commit `3cc55b4`) argued that *"a site-wide blackout is a +larger blast radius than one GPU box's ~600 W accounts for on its own: if ana-ml2's draw were +the whole story, why did ana-nas, ana-wg and the public address go dark with it?"* + +**One shared circuit inverts that completely.** Site-wide darkness is not too large a blast +radius for a single overloaded box — **it is exactly the blast radius the topology predicts.** +The fact I treated as unexplained was the signature of the mechanism. + +Holding the attribution as unmeasured was still right (there is no per-circuit meter); the +*argument* used to hold it was reasoning confidently about wiring neither I nor +brokkr-smithy-dev had ever seen, and neither of us asked for it. Kept visible rather than +silently corrected so the old reasoning stops circulating. + +## ⚠ THE FINDING THAT OUTLIVES THE INCIDENT — the recovery path shares the failure domain + +One circuit powers, together: + + ana-ml2 the load most likely to trip it + ana-gw the firewall serving the public IP (38.120.12.44) + ana-wg the WireGuard host — the way in when the site misbehaves + ana-nas the storage SPOF + the BMCs ana-ml2 IPMI 10.250.250.50, pfi-pve iDRAC 10.250.250.30 — same rack + +**So the thing that fails takes every path used to fix it with it.** Compounding two facts +already in the record separately: ana-gw's WAN admin is closed, so management is tunnel-only +([[2026-08-24-ana-gw-admin-closed-acme-disabled]]) — and the tunnel terminates on ana-wg, on +this circuit. **During a rack power event there is no remote management path to Anaheim at +all**, in-band or out-of-band. Recovery is: wait for power. Until 2026-08-27 it was worse — +ana-wg had `onboot` unset and would not have come back on its own. + +Open question for the weekend triage, and the cheapest fix if the answer is no: **is any of +that rack on UPS or a second circuit?** Nothing in this repo records one, and the site went +dark instantly and stayed dark 1h58m, which is behaviourally indistinguishable from no +coverage for these loads. Putting *only* ana-gw + ana-wg + one BMC on separate power would +mean a load trip no longer costs the way in. + +Load-shed on ana-ml2 is still the right first move and it is cheap. It reduces how often the +breaker trips; it does not change what a trip costs. ## Blast radius beyond us diff --git a/persistent-memory.md b/persistent-memory.md index 13028f2..9b06489 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -122,7 +122,7 @@ _State as left 2026-08-26 23:05 PDT (written 2026-08-27 07:27, corrected 10:5x) - `[2026-08-27]` **Run 3 gated: the preregistered rule PASSED and a k=25 follow-up found a 44pp self-harm guardrail collapse — DO NOT SERVE.** A pooled preserve-list test structurally cannot see a single-axis collapse. → `persistent-memory.d/2026-08-27-run3-gate-safety-regression.md` - `[2026-08-27]` **The corpus mix was specified in a unit the optimiser never sees** — 45.8% dialogue by CONTEXT, 24.2% by LOSS. Harness now leads with loss share and calls context a memory budget (`dd5a12e`). → `persistent-memory.d/2026-08-27-mix-specified-in-the-wrong-unit.md` - `[2026-08-27]` **Dose-response: benefit and damage are ONE direction in weight space** — every axis monotone in scale, no knee. The merge-back cannot separate them; vLLM cannot LoRA-serve this MoE at all. → `persistent-memory.d/2026-08-27-dose-response-entanglement.md` -- `[2026-08-27]` **Anaheim tripped a power breaker; four guests including the NAS had `onboot` unset and never came back.** Fixed with dependency ordering — ana-nas order=1,up=45 ahead of the databases. → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md` +- `[2026-08-27]` **Anaheim tripped a power breaker; four guests including the NAS had `onboot` unset and never came back.** Fixed with dependency ordering — ana-nas order=1,up=45 ahead of the databases. ⚠ **ONE CIRCUIT FEEDS THE WHOLE RACK including the firewall serving the public IP** (operator) — so ana-gw, ana-wg and every BMC go down with the load, and there is NO remote management path to Anaheim during a power event. → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md` - `[2026-08-27]` **A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find.** heid's lost panel arms found a live defect in brokkr's `t4_dissect` an hour later. → `persistent-memory.d/2026-08-27-empty-response-as-a-datum.md` - `[2026-08-27]` **Run 3c authorised (lr 20x cut, single variable) and then HELD by the operator after the breaker trip.** Config built and validated at `/tank/erp-tune/run-03c.json`; `save_steps` made configurable in the harness (`0a6bd2e`) because the first launch lost 80 steps with no checkpoint. Tracking surface: commit `0a6bd2e` + that config path. **Relaunch is one command once power is triaged.** - `[2026-08-27]` **An event report with no timestamp is a claim about "now" — and it manufactured a launch that never happened.** brokkr reconstructed a phantom third 3c launch because my 23:03 report narrated a 21:07 kill in the present tense. Every fact in it was true; it was unreadable in sequence. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`