vh
969a1b64a2
Written while the site is down so recovery does not have to be reconstructed later. Records what was measured rather than what is suspected: every FV address including the BMC is unreachable while all three other sites answer, the campaign's last log line was off_A rep 2 at 06:56:04Z, and the site was dark by 06:58:40Z. Names three candidate causes with the evidence that would distinguish them, because the instrument that could have settled it -- the per-card power log -- died with the box. The two-card-load hypothesis fits the timing and the two prior Anaheim breaker trips on this same chassis, but it is circumstantial and is recorded as such. Carries the recovery hazard that matters: every seat on the box is restart:unless-stopped, so resetting power alone brings ten vLLM containers up loading at once -- the largest transient the box can produce, into a circuit that may have just tripped. Staged sequence given, gen first and flash-next last. Also records the OOB gap the outage exposes: OPNsense-as-subnet-router protects against box-down/gateway-up, and not at all against the site-wide loss that actually happened, because the BMC's only path out is through that same gateway.
5.5 KiB
FV site dark — 2026-09-13 ~06:56Z
Status: UNRESOLVED, needs hands at Fountain Valley. Written while the site is down so recovery does not have to be reconstructed from memory.
What is down
Everything at Fountain Valley, measured 06:58:40Z from nh3-dev:
| target | result |
|---|---|
fv-ml1 10.251.50.54 |
100% loss |
fv-ml1 mesh 100.64.0.7 |
100% loss |
| FV gateway 10.251.50.1 | 100% loss |
| FV gateway mesh 100.64.0.8 | 100% loss |
| fv-ml1-bmc 10.251.250.50 | 100% loss |
fv.phasefinal.com (172.83.89.66) |
no answer |
My own path is healthy — ana-docker, nh3-docker, esh-docker-vm and 1.1.1.1 all answer with 0% loss, so this is FV-local, not an isolation of the observer.
Timeline
06:51:08 MTP campaign started on GPU 3 (:8023), operator-authorised, knowing
the two-card power risk. Production seat already live on GPU 2.
06:51:17 power sampler baseline: GPU0 3.80 W, GPU1 3.88 W, GPU2 14.27 W, GPU3 6.75 W
06:54:39 off_A arm healthy after 211 s
06:54:39 off_A rep 1 ran clean: 75.49 / 212.33 / 387.83 tok/s at conc 1/4/8
06:56:04 off_A rep 2 started <-- LAST LOG LINE
06:58:40 every FV address unreachable, including the BMC
So the site went dark inside a ~2.5 minute window, roughly five minutes into sustained bench load on GPU 3 with GPU 2 also resident/serving.
Three candidate causes, and what distinguishes them
Cannot be distinguished remotely, because every FV path — including the BMC — traverses the OPNsense gateway, and the gateway is also dark.
- Circuit tripped under two-card load. Fits the timing and is the predicted failure: the Anaheim rack breaker tripped twice (2026-08-26, 2026-09-11) on this same chassis, and the FV circuit was specced while every record still said the box had two GPUs rather than four. ⭐ Distinguishing evidence: the breaker is visibly tripped, and the OPNsense box is dark too (it draws ~20-30 W and would survive anything short of a circuit/utility loss).
- OPNsense gateway crashed or rebooted, taking all FV routing with it while the GPU box is fine. Distinguishing evidence: the GPU box's PSU fans/lights are on and the OPNsense box is the only dead thing.
- Upstream utility or colo-side power/network loss, unrelated to us. Distinguishing evidence: the breaker is NOT tripped and nothing at the rack has power; neighbouring equipment is also dark.
⚠ Do not record cause 1 as fact until someone has looked. The timing is strong circumstantial evidence, not a measurement — and the instrument that would have measured it (the power log on fv-ml1) died with the box.
Blast radius
19 of 30 LiteLLM aliases are dark — fv-ml1 backs most of the fleet's inference:
char-rp, char-rp-fast, char-rp-reasoning, chat-judge, classifier, coder-fast,
erp-tune-v2, gemma4-26b-a4b-it-base, gen, gen-large, gen-reasoning, image-judge,
qwen-image-bench, qwen3-embedding, reranker, sec, sec-reasoning, summarizer,
summarizer-large
Still served: the paid passthroughs (gen-frontier*, glm*, kimi*) and the irv-ml1
locals. The gateway itself on ana-docker is healthy — it is the backends that are gone.
⚠⚠ RECOVERY — do NOT just reset the breaker and walk away
Every vLLM seat on fv-ml1 carries restart: unless-stopped. On boot all of them
start loading simultaneously — gen, mog-sec, erp-seat, gemma4-charrp, flash-next,
embed, rerank, reward, coder, scriberr — which is the single largest power transient
the box can produce, fed straight into a circuit that may have just tripped. That is a
re-trip, and a re-trip during model load can leave a half-written page cache and a
much longer recovery.
Preferred sequence:
- Power the chassis on with Docker masked, so nothing auto-starts:
at the BMC/console, boot to the OS and before the network comes up run
systemctl mask docker containerd— or if the box is already up and loading,systemctl stop dockerimmediately. - Confirm
nvidia-smisees all four cards andzpool status tankis ONLINE. - Unmask, then bring seats up one at a time, waiting for each to report healthy:
genfirst (19 aliases depend on it), then embed/rerank/reward/coder, then mog-sec, then the rest.flash-nextLAST — it is the newest and least depended-on. - Do not restart the MTP campaign. It is the prime suspect.
- Watch power while seats come up:
nvidia-smi --query-gpu=index,power.draw --format=csv.
Do not reboot the OPNsense firewall (standing operator directive; its reboot API 403s anyway).
Afterwards
- ⭐ The OOB design gap this exposes. The cutover chose OPNsense-as-subnet-router specifically so the BMC stays reachable when the GPU box is down. That works for box-down/gateway-up. It does nothing for a site-wide power or gateway loss — exactly what happened — because the BMC's only path to the fleet is through that gateway. A genuine OOB path at FV needs something the FV circuit cannot take down: an LTE/cellular console, or the BMC on a separate circuit with its own uplink.
- Get the actual circuit rating and the box's real peak draw, now that it is known to have four cards and not two. Until then, treat concurrent multi-card load at FV as unproven rather than safe.
services/flash-next-mtp-bench/power.logon the box holds the per-card draw right up to the cut. Recover it after boot — it is the only measurement of what the load actually drew, and it survives on/tank, not in the container.