vh
ea818380ff
Operator supplied the topology: "the entire rack is on the same circuit,
public ip is served by firewall on the same circuit. load tripped
breaker, entire rack goes dark."
That inverts the argument I committed one commit ago in 3cc55b4. I held
the load attribution partly because a site-wide blackout looked like too
large a blast radius for one GPU box. It is not — it is exactly the
blast radius a shared circuit predicts. The observation I treated as
unexplained was the signature of the mechanism.
Kept the wrong version visible with a superseded marker rather than
editing it away, per the superseded-claims convention: the reasoning
circulated in two althing messages and a peer adopted it.
Holding the attribution as unmeasured was still correct; the argument
used to hold it was reasoning about wiring neither side had seen and
neither side asked for.
Records the finding that outlives the incident: the same circuit powers
ana-ml2, ana-gw, ana-wg, ana-nas and the BMCs, so the load takes every
recovery path with it. Composed with ana-gw's WAN admin being closed
(tunnel-only) and the tunnel terminating on ana-wg, there is no remote
management path to Anaheim during a rack power event, in-band or out.
Open question for the weekend triage: is any of that rack on UPS or a
second circuit?
6.7 KiB
[2026-08-27] Anaheim tripped a power breaker — and four guests including the NAS never came back
Site-wide outage. ana-ml2 was down 1h58m — power lost between 18:14:45 and 18:17:00 PDT,
boot at 20:13:14 (the "~90 minutes" first recorded here was an estimate; the window is now
pinned by run 3c's training log and the hourly cron that never ran — see
2026-08-27-run3c-launch-count-reconstruction). Operator-confirmed cause: a tripped power breaker, not a
fault and not the tunnel. The discriminator that established scope: ana-srv1
(38.120.12.44:443, Anaheim's PUBLIC address) was dark from the internet, so it was not the
NH3↔ANA IPsec tunnel stranding NH3 — the site was not answering on any path. ana-ml2 returned
with up 1 min, confirming a hard power event.
⚠ THE DURABLE FINDING — onboot was unset on four guests
pfi-pve came back and auto-started everything except:
CT109 ana-nas the storage SPOF
CT113 ana-wg the WireGuard remote-access path
CT112 ana-filebot
VM106 corviduo-dev
All four had onboot unset. Recovery was manual and would have been manual every time —
including for the NAS that postgres/PBS/cross-site-restic depend on, and the WireGuard host
that is the way in when the site misbehaves.
FIXED, with dependency ordering (operator-authorised):
CT109 ana-nas onboot=1 order=1,up=45 <- first; 45s for NFS to SERVE
CT113 ana-wg onboot=1 order=2 <- remote access before anything can fail
VM104/105 Mongo/Postgres order=3,up=60 (pre-existing)
VM102 ANA-Docker order=4 (pre-existing)
VM101 ANA-DC order=5,up=120 (pre-existing)
CT112 ana-filebot onboot=1 order=10
VM106 corviduo-dev onboot=1 order=10
Every guest on pfi-pve now auto-starts. ana-nas precedes the databases deliberately; the
up=45 is for NFS to be serving, not merely for the container to be running — the exact
distinction that killed rest-server on ana-docker, which came up before the NAS existed,
found nothing to serve, and exited 255.
⚠ /tank came back DEGRADED — a disk is genuinely gone
tank DEGRADED, raidz2-0, 7 devices ONLINE
9477159196657038377 FAULTED was /dev/nvme4n1p1
errors: No known data errors
Only 7 physical NVMe present where the pool expects 8 — checked, so not renumbering. One drive did not re-enumerate. raidz2 carries two disks of parity; one is spent. Operator taking it; chassis is a Supermicro AS-4125GS-TNRT2 with PCIe hot-plug slots, so a swap should not need a power-down.
⚠ /mnt/smithy is manual by design — it will be missing after EVERY reboot
Not in fstab, and deliberately so: a cross-site NFS entry can hang boot on a GPU host, and
it is soft rather than hard because ana-ml2 is cross-site from that NAS and a hard mount
turns a link blip into unkillable D-state. Remount with the recorded spec, do NOT "fix" it into
fstab:
sudo mount -t nfs4 -o ro,soft,timeo=30,retrans=3,proto=tcp,vers=4.1 \
10.100.50.50:/volume1/smithy /mnt/smithy
Full rationale: 2026-08-23-smithy-mount-ana-ml2.
Power capacity is now the open item
Operator: "we'll triage this weekend, probably shut down some seats." ana-ml2 alone was
pulling ~600 W across both GPUs at their 300 W caps during training. gen stays up by
instruction; everything else on that box is idle.
MEASURED a breaker tripped (operator-confirmed)
MEASURED power lost between 18:14:45 and 18:17:00 PDT
MEASURED the WHOLE Anaheim site went dark from the internet
TOPOLOGY ONE CIRCUIT FEEDS THE ENTIRE RACK, including the firewall serving
the public IP (operator, 2026-08-27)
OPERATOR "load tripped breaker, entire rack goes dark." Consistent with every
DETERMINATION observation. Not a per-circuit measurement, and no longer needs to be —
the operator has the site knowledge and has ruled.
⚠ SUPERSEDED — the blast-radius objection I raised here was WRONG, and it was committed.
An earlier revision of this section (commit 3cc55b4) argued that "a site-wide blackout is a
larger blast radius than one GPU box's ~600 W accounts for on its own: if ana-ml2's draw were
the whole story, why did ana-nas, ana-wg and the public address go dark with it?"
One shared circuit inverts that completely. Site-wide darkness is not too large a blast radius for a single overloaded box — it is exactly the blast radius the topology predicts. The fact I treated as unexplained was the signature of the mechanism.
Holding the attribution as unmeasured was still right (there is no per-circuit meter); the argument used to hold it was reasoning confidently about wiring neither I nor brokkr-smithy-dev had ever seen, and neither of us asked for it. Kept visible rather than silently corrected so the old reasoning stops circulating.
⚠ THE FINDING THAT OUTLIVES THE INCIDENT — the recovery path shares the failure domain
One circuit powers, together:
ana-ml2 the load most likely to trip it
ana-gw the firewall serving the public IP (38.120.12.44)
ana-wg the WireGuard host — the way in when the site misbehaves
ana-nas the storage SPOF
the BMCs ana-ml2 IPMI 10.250.250.50, pfi-pve iDRAC 10.250.250.30 — same rack
So the thing that fails takes every path used to fix it with it. Compounding two facts
already in the record separately: ana-gw's WAN admin is closed, so management is tunnel-only
(2026-08-24-ana-gw-admin-closed-acme-disabled) — and the tunnel terminates on ana-wg, on
this circuit. During a rack power event there is no remote management path to Anaheim at
all, in-band or out-of-band. Recovery is: wait for power. Until 2026-08-27 it was worse —
ana-wg had onboot unset and would not have come back on its own.
Open question for the weekend triage, and the cheapest fix if the answer is no: is any of that rack on UPS or a second circuit? Nothing in this repo records one, and the site went dark instantly and stayed dark 1h58m, which is behaviourally indistinguishable from no coverage for these loads. Putting only ana-gw + ana-wg + one BMC on separate power would mean a load trip no longer costs the way in.
Load-shed on ana-ml2 is still the right first move and it is cheap. It reduces how often the breaker trips; it does not change what a trip costs.
Blast radius beyond us
heid lost both gateway-routed arms of a four-arm panel mid-dispatch and discovered the outage by losing half a panel. That report produced the single most valuable artifact of the incident — see 2026-08-27-empty-response-as-a-datum.