# `[2026-08-27]` Anaheim tripped a power breaker — and four guests including the NAS never came back Site-wide outage. **ana-ml2 was down 1h58m** — power lost between **18:14:45 and 18:17:00 PDT**, boot at **20:13:14** (the "~90 minutes" first recorded here was an estimate; the window is now pinned by run 3c's training log and the hourly cron that never ran — see [[2026-08-27-run3c-launch-count-reconstruction]]). **Operator-confirmed cause: a tripped power breaker**, not a fault and not the tunnel. The discriminator that established scope: `ana-srv1` (38.120.12.44:443, Anaheim's PUBLIC address) was dark **from the internet**, so it was not the NH3↔ANA IPsec tunnel stranding NH3 — the site was not answering on any path. `ana-ml2` returned with `up 1 min`, confirming a hard power event. ## ⚠ THE DURABLE FINDING — `onboot` was unset on four guests pfi-pve came back and auto-started everything **except**: CT109 ana-nas the storage SPOF CT113 ana-wg the WireGuard remote-access path CT112 ana-filebot VM106 corviduo-dev All four had `onboot` unset. **Recovery was manual and would have been manual every time** — including for the NAS that postgres/PBS/cross-site-restic depend on, and the WireGuard host that is the way in when the site misbehaves. **FIXED, with dependency ordering** (operator-authorised): CT109 ana-nas onboot=1 order=1,up=45 <- first; 45s for NFS to SERVE CT113 ana-wg onboot=1 order=2 <- remote access before anything can fail VM104/105 Mongo/Postgres order=3,up=60 (pre-existing) VM102 ANA-Docker order=4 (pre-existing) VM101 ANA-DC order=5,up=120 (pre-existing) CT112 ana-filebot onboot=1 order=10 VM106 corviduo-dev onboot=1 order=10 Every guest on pfi-pve now auto-starts. ana-nas precedes the databases deliberately; the `up=45` is for NFS to be *serving*, not merely for the container to be *running* — the exact distinction that killed `rest-server` on ana-docker, which came up before the NAS existed, found nothing to serve, and exited 255. ## ⚠ `/tank` came back DEGRADED — a disk is genuinely gone tank DEGRADED, raidz2-0, 7 devices ONLINE 9477159196657038377 FAULTED was /dev/nvme4n1p1 errors: No known data errors **Only 7 physical NVMe present where the pool expects 8** — checked, so not renumbering. One drive did not re-enumerate. raidz2 carries two disks of parity; one is spent. Operator taking it; chassis is a Supermicro AS-4125GS-TNRT2 with PCIe hot-plug slots, so a swap should not need a power-down. ## ⚠ `/mnt/smithy` is manual by design — it will be missing after EVERY reboot Not in fstab, and **deliberately so**: a cross-site NFS entry can hang boot on a GPU host, and it is `soft` rather than `hard` because ana-ml2 is cross-site from that NAS and a hard mount turns a link blip into unkillable D-state. Remount with the recorded spec, do NOT "fix" it into fstab: sudo mount -t nfs4 -o ro,soft,timeo=30,retrans=3,proto=tcp,vers=4.1 \ 10.100.50.50:/volume1/smithy /mnt/smithy Full rationale: [[2026-08-23-smithy-mount-ana-ml2]]. ## Power capacity is now the open item Operator: *"we'll triage this weekend, probably shut down some seats."* ana-ml2 alone was pulling ~600 W across both GPUs at their 300 W caps during training. `gen` stays up by instruction; everything else on that box is idle. MEASURED a breaker tripped (operator-confirmed) MEASURED power lost between 18:14:45 and 18:17:00 PDT MEASURED the WHOLE Anaheim site went dark from the internet TOPOLOGY ONE CIRCUIT FEEDS THE ENTIRE RACK, including the firewall serving the public IP (operator, 2026-08-27) OPERATOR "load tripped breaker, entire rack goes dark." Consistent with every DETERMINATION observation. Not a per-circuit measurement, and no longer needs to be — the operator has the site knowledge and has ruled. ⚠ **SUPERSEDED — the blast-radius objection I raised here was WRONG, and it was committed.** An earlier revision of this section (commit `3cc55b4`) argued that *"a site-wide blackout is a larger blast radius than one GPU box's ~600 W accounts for on its own: if ana-ml2's draw were the whole story, why did ana-nas, ana-wg and the public address go dark with it?"* **One shared circuit inverts that completely.** Site-wide darkness is not too large a blast radius for a single overloaded box — **it is exactly the blast radius the topology predicts.** The fact I treated as unexplained was the signature of the mechanism. Holding the attribution as unmeasured was still right (there is no per-circuit meter); the *argument* used to hold it was reasoning confidently about wiring neither I nor brokkr-smithy-dev had ever seen, and neither of us asked for it. Kept visible rather than silently corrected so the old reasoning stops circulating. ## ⚠ THE FINDING THAT OUTLIVES THE INCIDENT — the recovery path shares the failure domain One circuit powers, together: ana-ml2 the load most likely to trip it ana-gw the firewall serving the public IP (38.120.12.44) ana-wg the WireGuard host — the way in when the site misbehaves ana-nas the storage SPOF the BMCs ana-ml2 IPMI 10.250.250.50, pfi-pve iDRAC 10.250.250.30 — same rack **So the thing that fails takes every path used to fix it with it.** Compounding two facts already in the record separately: ana-gw's WAN admin is closed, so management is tunnel-only ([[2026-08-24-ana-gw-admin-closed-acme-disabled]]) — and the tunnel terminates on ana-wg, on this circuit. **During a rack power event there is no remote management path to Anaheim at all**, in-band or out-of-band. Recovery is: wait for power. Until 2026-08-27 it was worse — ana-wg had `onboot` unset and would not have come back on its own. Open question for the weekend triage, and the cheapest fix if the answer is no: **is any of that rack on UPS or a second circuit?** Nothing in this repo records one, and the site went dark instantly and stayed dark 1h58m, which is behaviourally indistinguishable from no coverage for these loads. Putting *only* ana-gw + ana-wg + one BMC on separate power would mean a load trip no longer costs the way in. Load-shed on ana-ml2 is still the right first move and it is cheap. It reduces how often the breaker trips; it does not change what a trip costs. ## Blast radius beyond us heid lost **both gateway-routed arms of a four-arm panel** mid-dispatch and discovered the outage by losing half a panel. That report produced the single most valuable artifact of the incident — see [[2026-08-27-empty-response-as-a-datum]].