# `[2026-08-27]` Anaheim tripped a power breaker — and four guests including the NAS never came back

Site-wide outage. **ana-ml2 was down 1h58m** — power lost between **18:14:45 and 18:17:00 PDT**,
boot at **20:13:14** (the "~90 minutes" first recorded here was an estimate; the window is now
pinned by run 3c's training log and the hourly cron that never ran — see
[[2026-08-27-run3c-launch-count-reconstruction]]). **Operator-confirmed cause: a tripped power breaker**, not a
fault and not the tunnel. The discriminator that established scope: `ana-srv1`
(38.120.12.44:443, Anaheim's PUBLIC address) was dark **from the internet**, so it was not the
NH3↔ANA IPsec tunnel stranding NH3 — the site was not answering on any path. `ana-ml2` returned
with `up 1 min`, confirming a hard power event.

## ⚠ THE DURABLE FINDING — `onboot` was unset on four guests

pfi-pve came back and auto-started everything **except**:

    CT109  ana-nas        the storage SPOF
    CT113  ana-wg         the WireGuard remote-access path
    CT112  ana-filebot
    VM106  corviduo-dev

All four had `onboot` unset. **Recovery was manual and would have been manual every time** —
including for the NAS that postgres/PBS/cross-site-restic depend on, and the WireGuard host
that is the way in when the site misbehaves.

**FIXED, with dependency ordering** (operator-authorised):

    CT109  ana-nas          onboot=1  order=1,up=45   <- first; 45s for NFS to SERVE
    CT113  ana-wg           onboot=1  order=2         <- remote access before anything can fail
    VM104/105 Mongo/Postgres          order=3,up=60   (pre-existing)
    VM102  ANA-Docker                 order=4         (pre-existing)
    VM101  ANA-DC                     order=5,up=120  (pre-existing)
    CT112  ana-filebot      onboot=1  order=10
    VM106  corviduo-dev     onboot=1  order=10

Every guest on pfi-pve now auto-starts. ana-nas precedes the databases deliberately; the
`up=45` is for NFS to be *serving*, not merely for the container to be *running* — the exact
distinction that killed `rest-server` on ana-docker, which came up before the NAS existed,
found nothing to serve, and exited 255.

## ⚠ `/tank` came back DEGRADED — a disk is genuinely gone

    tank  DEGRADED, raidz2-0, 7 devices ONLINE
      9477159196657038377   FAULTED   was /dev/nvme4n1p1
    errors: No known data errors

**Only 7 physical NVMe present where the pool expects 8** — checked, so not renumbering. One
drive did not re-enumerate. raidz2 carries two disks of parity; one is spent. Operator taking
it; chassis is a Supermicro AS-4125GS-TNRT2 with PCIe hot-plug slots, so a swap should not
need a power-down.

## ⚠ `/mnt/smithy` is manual by design — it will be missing after EVERY reboot

Not in fstab, and **deliberately so**: a cross-site NFS entry can hang boot on a GPU host, and
it is `soft` rather than `hard` because ana-ml2 is cross-site from that NAS and a hard mount
turns a link blip into unkillable D-state. Remount with the recorded spec, do NOT "fix" it into
fstab:

    sudo mount -t nfs4 -o ro,soft,timeo=30,retrans=3,proto=tcp,vers=4.1 \
      10.100.50.50:/volume1/smithy /mnt/smithy

Full rationale: [[2026-08-23-smithy-mount-ana-ml2]].

## Power capacity is now the open item

Operator: *"we'll triage this weekend, probably shut down some seats."* ana-ml2 alone was
pulling ~600 W across both GPUs at their 300 W caps during training. `gen` stays up by
instruction; everything else on that box is idle.

    MEASURED      a breaker tripped (operator-confirmed)
    MEASURED      power lost between 18:14:45 and 18:17:00 PDT
    MEASURED      the WHOLE Anaheim site went dark from the internet
    TOPOLOGY      ONE CIRCUIT FEEDS THE ENTIRE RACK, including the firewall serving
                  the public IP  (operator, 2026-08-27)

    OPERATOR      "load tripped breaker, entire rack goes dark." Consistent with every
      DETERMINATION observation. Not a per-circuit measurement, and no longer needs to be —
                  the operator has the site knowledge and has ruled.

⚠ **SUPERSEDED — the blast-radius objection I raised here was WRONG, and it was committed.**
An earlier revision of this section (commit `3cc55b4`) argued that *"a site-wide blackout is a
larger blast radius than one GPU box's ~600 W accounts for on its own: if ana-ml2's draw were
the whole story, why did ana-nas, ana-wg and the public address go dark with it?"*

**One shared circuit inverts that completely.** Site-wide darkness is not too large a blast
radius for a single overloaded box — **it is exactly the blast radius the topology predicts.**
The fact I treated as unexplained was the signature of the mechanism.

Holding the attribution as unmeasured was still right (there is no per-circuit meter); the
*argument* used to hold it was reasoning confidently about wiring neither I nor
brokkr-smithy-dev had ever seen, and neither of us asked for it. Kept visible rather than
silently corrected so the old reasoning stops circulating.

## ⚠ THE FINDING THAT OUTLIVES THE INCIDENT — the recovery path shares the failure domain

One circuit powers, together:

    ana-ml2          the load most likely to trip it
    ana-gw           the firewall serving the public IP (38.120.12.44)
    ana-wg           the WireGuard host — the way in when the site misbehaves
    ana-nas          the storage SPOF
    the BMCs         ana-ml2 IPMI 10.250.250.50, pfi-pve iDRAC 10.250.250.30 — same rack

**So the thing that fails takes every path used to fix it with it.** Compounding two facts
already in the record separately: ana-gw's WAN admin is closed, so management is tunnel-only
([[2026-08-24-ana-gw-admin-closed-acme-disabled]]) — and the tunnel terminates on ana-wg, on
this circuit. **During a rack power event there is no remote management path to Anaheim at
all**, in-band or out-of-band. Recovery is: wait for power. Until 2026-08-27 it was worse —
ana-wg had `onboot` unset and would not have come back on its own.

Open question for the weekend triage, and the cheapest fix if the answer is no: **is any of
that rack on UPS or a second circuit?** Nothing in this repo records one, and the site went
dark instantly and stayed dark 1h58m, which is behaviourally indistinguishable from no
coverage for these loads. Putting *only* ana-gw + ana-wg + one BMC on separate power would
mean a load trip no longer costs the way in.

Load-shed on ana-ml2 is still the right first move and it is cheap. It reduces how often the
breaker trips; it does not change what a trip costs.

## Blast radius beyond us

heid lost **both gateway-routed arms of a four-arm panel** mid-dispatch and discovered the
outage by losing half a panel. That report produced the single most valuable artifact of the
incident — see [[2026-08-27-empty-response-as-a-datum]].
