runbook: FV site dark 2026-09-13 — outage facts, blast radius, staged recovery, OOB design gap
Written while the site is down so recovery does not have to be reconstructed later. Records what was measured rather than what is suspected: every FV address including the BMC is unreachable while all three other sites answer, the campaign's last log line was off_A rep 2 at 06:56:04Z, and the site was dark by 06:58:40Z. Names three candidate causes with the evidence that would distinguish them, because the instrument that could have settled it -- the per-card power log -- died with the box. The two-card-load hypothesis fits the timing and the two prior Anaheim breaker trips on this same chassis, but it is circumstantial and is recorded as such. Carries the recovery hazard that matters: every seat on the box is restart:unless-stopped, so resetting power alone brings ten vLLM containers up loading at once -- the largest transient the box can produce, into a circuit that may have just tripped. Staged sequence given, gen first and flash-next last. Also records the OOB gap the outage exposes: OPNsense-as-subnet-router protects against box-down/gateway-up, and not at all against the site-wide loss that actually happened, because the BMC's only path out is through that same gateway.
This commit is contained in:
@@ -0,0 +1,107 @@
|
||||
# FV site dark — 2026-09-13 ~06:56Z
|
||||
|
||||
**Status: UNRESOLVED, needs hands at Fountain Valley.** Written while the site is
|
||||
down so recovery does not have to be reconstructed from memory.
|
||||
|
||||
## What is down
|
||||
|
||||
Everything at Fountain Valley, measured 06:58:40Z from nh3-dev:
|
||||
|
||||
| target | result |
|
||||
|---|---|
|
||||
| `fv-ml1` 10.251.50.54 | 100% loss |
|
||||
| `fv-ml1` mesh 100.64.0.7 | 100% loss |
|
||||
| FV gateway 10.251.50.1 | 100% loss |
|
||||
| FV gateway mesh 100.64.0.8 | 100% loss |
|
||||
| **fv-ml1-bmc 10.251.250.50** | **100% loss** |
|
||||
| `fv.phasefinal.com` (172.83.89.66) | no answer |
|
||||
|
||||
**My own path is healthy** — ana-docker, nh3-docker, esh-docker-vm and 1.1.1.1 all
|
||||
answer with 0% loss, so this is FV-local, not an isolation of the observer.
|
||||
|
||||
## Timeline
|
||||
|
||||
06:51:08 MTP campaign started on GPU 3 (:8023), operator-authorised, knowing
|
||||
the two-card power risk. Production seat already live on GPU 2.
|
||||
06:51:17 power sampler baseline: GPU0 3.80 W, GPU1 3.88 W, GPU2 14.27 W, GPU3 6.75 W
|
||||
06:54:39 off_A arm healthy after 211 s
|
||||
06:54:39 off_A rep 1 ran clean: 75.49 / 212.33 / 387.83 tok/s at conc 1/4/8
|
||||
06:56:04 off_A rep 2 started <-- LAST LOG LINE
|
||||
06:58:40 every FV address unreachable, including the BMC
|
||||
|
||||
So the site went dark inside a ~2.5 minute window, roughly five minutes into
|
||||
sustained bench load on GPU 3 with GPU 2 also resident/serving.
|
||||
|
||||
## Three candidate causes, and what distinguishes them
|
||||
|
||||
Cannot be distinguished remotely, because every FV path — including the BMC —
|
||||
traverses the OPNsense gateway, and the gateway is also dark.
|
||||
|
||||
1. **Circuit tripped under two-card load.** Fits the timing and is the predicted
|
||||
failure: the Anaheim rack breaker tripped twice (2026-08-26, 2026-09-11) on this
|
||||
same chassis, and the FV circuit was specced while every record still said the box
|
||||
had **two** GPUs rather than four. ⭐ Distinguishing evidence: **the breaker is
|
||||
visibly tripped**, and the OPNsense box is dark too (it draws ~20-30 W and would
|
||||
survive anything short of a circuit/utility loss).
|
||||
2. **OPNsense gateway crashed or rebooted**, taking all FV routing with it while the
|
||||
GPU box is fine. Distinguishing evidence: the GPU box's PSU fans/lights are on and
|
||||
the OPNsense box is the only dead thing.
|
||||
3. **Upstream utility or colo-side power/network loss**, unrelated to us.
|
||||
Distinguishing evidence: the breaker is NOT tripped and nothing at the rack has
|
||||
power; neighbouring equipment is also dark.
|
||||
|
||||
⚠ Do not record cause 1 as fact until someone has looked. The timing is strong
|
||||
circumstantial evidence, not a measurement — and the instrument that would have
|
||||
measured it (the power log on fv-ml1) died with the box.
|
||||
|
||||
## Blast radius
|
||||
|
||||
**19 of 30 LiteLLM aliases are dark** — fv-ml1 backs most of the fleet's inference:
|
||||
|
||||
char-rp, char-rp-fast, char-rp-reasoning, chat-judge, classifier, coder-fast,
|
||||
erp-tune-v2, gemma4-26b-a4b-it-base, gen, gen-large, gen-reasoning, image-judge,
|
||||
qwen-image-bench, qwen3-embedding, reranker, sec, sec-reasoning, summarizer,
|
||||
summarizer-large
|
||||
|
||||
Still served: the paid passthroughs (`gen-frontier*`, `glm*`, `kimi*`) and the irv-ml1
|
||||
locals. The gateway itself on ana-docker is healthy — it is the backends that are gone.
|
||||
|
||||
## ⚠⚠ RECOVERY — do NOT just reset the breaker and walk away
|
||||
|
||||
Every vLLM seat on fv-ml1 carries `restart: unless-stopped`. On boot **all of them
|
||||
start loading simultaneously** — gen, mog-sec, erp-seat, gemma4-charrp, flash-next,
|
||||
embed, rerank, reward, coder, scriberr — which is the single largest power transient
|
||||
the box can produce, fed straight into a circuit that may have just tripped. That is a
|
||||
re-trip, and a re-trip during model load can leave a half-written page cache and a
|
||||
much longer recovery.
|
||||
|
||||
**Preferred sequence:**
|
||||
|
||||
1. Power the chassis on with **Docker masked**, so nothing auto-starts:
|
||||
at the BMC/console, boot to the OS and before the network comes up run
|
||||
`systemctl mask docker containerd` — or if the box is already up and loading,
|
||||
`systemctl stop docker` immediately.
|
||||
2. Confirm `nvidia-smi` sees all four cards and `zpool status tank` is ONLINE.
|
||||
3. Unmask, then bring seats up **one at a time**, waiting for each to report healthy:
|
||||
`gen` first (19 aliases depend on it), then embed/rerank/reward/coder, then
|
||||
mog-sec, then the rest. `flash-next` LAST — it is the newest and least depended-on.
|
||||
4. **Do not restart the MTP campaign.** It is the prime suspect.
|
||||
5. Watch power while seats come up: `nvidia-smi --query-gpu=index,power.draw --format=csv`.
|
||||
|
||||
**Do not reboot the OPNsense firewall** (standing operator directive; its reboot API
|
||||
403s anyway).
|
||||
|
||||
## Afterwards
|
||||
|
||||
- ⭐ **The OOB design gap this exposes.** The cutover chose OPNsense-as-subnet-router
|
||||
specifically so the BMC stays reachable when the GPU box is down. That works for
|
||||
box-down/gateway-up. It does **nothing** for a site-wide power or gateway loss —
|
||||
exactly what happened — because the BMC's only path to the fleet is through that
|
||||
gateway. A genuine OOB path at FV needs something the FV circuit cannot take down:
|
||||
an LTE/cellular console, or the BMC on a separate circuit with its own uplink.
|
||||
- Get the actual circuit rating and the box's real peak draw, now that it is known to
|
||||
have four cards and not two. Until then, treat concurrent multi-card load at FV as
|
||||
unproven rather than safe.
|
||||
- `services/flash-next-mtp-bench/power.log` on the box holds the per-card draw right
|
||||
up to the cut. Recover it after boot — it is the only measurement of what the load
|
||||
actually drew, and it survives on `/tank`, not in the container.
|
||||
Reference in New Issue
Block a user