diff --git a/docs/runbooks/fv-site-dark-20260913.md b/docs/runbooks/fv-site-dark-20260913.md new file mode 100644 index 0000000..ed23efd --- /dev/null +++ b/docs/runbooks/fv-site-dark-20260913.md @@ -0,0 +1,107 @@ +# FV site dark — 2026-09-13 ~06:56Z + +**Status: UNRESOLVED, needs hands at Fountain Valley.** Written while the site is +down so recovery does not have to be reconstructed from memory. + +## What is down + +Everything at Fountain Valley, measured 06:58:40Z from nh3-dev: + +| target | result | +|---|---| +| `fv-ml1` 10.251.50.54 | 100% loss | +| `fv-ml1` mesh 100.64.0.7 | 100% loss | +| FV gateway 10.251.50.1 | 100% loss | +| FV gateway mesh 100.64.0.8 | 100% loss | +| **fv-ml1-bmc 10.251.250.50** | **100% loss** | +| `fv.phasefinal.com` (172.83.89.66) | no answer | + +**My own path is healthy** — ana-docker, nh3-docker, esh-docker-vm and 1.1.1.1 all +answer with 0% loss, so this is FV-local, not an isolation of the observer. + +## Timeline + + 06:51:08 MTP campaign started on GPU 3 (:8023), operator-authorised, knowing + the two-card power risk. Production seat already live on GPU 2. + 06:51:17 power sampler baseline: GPU0 3.80 W, GPU1 3.88 W, GPU2 14.27 W, GPU3 6.75 W + 06:54:39 off_A arm healthy after 211 s + 06:54:39 off_A rep 1 ran clean: 75.49 / 212.33 / 387.83 tok/s at conc 1/4/8 + 06:56:04 off_A rep 2 started <-- LAST LOG LINE + 06:58:40 every FV address unreachable, including the BMC + +So the site went dark inside a ~2.5 minute window, roughly five minutes into +sustained bench load on GPU 3 with GPU 2 also resident/serving. + +## Three candidate causes, and what distinguishes them + +Cannot be distinguished remotely, because every FV path — including the BMC — +traverses the OPNsense gateway, and the gateway is also dark. + +1. **Circuit tripped under two-card load.** Fits the timing and is the predicted + failure: the Anaheim rack breaker tripped twice (2026-08-26, 2026-09-11) on this + same chassis, and the FV circuit was specced while every record still said the box + had **two** GPUs rather than four. ⭐ Distinguishing evidence: **the breaker is + visibly tripped**, and the OPNsense box is dark too (it draws ~20-30 W and would + survive anything short of a circuit/utility loss). +2. **OPNsense gateway crashed or rebooted**, taking all FV routing with it while the + GPU box is fine. Distinguishing evidence: the GPU box's PSU fans/lights are on and + the OPNsense box is the only dead thing. +3. **Upstream utility or colo-side power/network loss**, unrelated to us. + Distinguishing evidence: the breaker is NOT tripped and nothing at the rack has + power; neighbouring equipment is also dark. + +⚠ Do not record cause 1 as fact until someone has looked. The timing is strong +circumstantial evidence, not a measurement — and the instrument that would have +measured it (the power log on fv-ml1) died with the box. + +## Blast radius + +**19 of 30 LiteLLM aliases are dark** — fv-ml1 backs most of the fleet's inference: + + char-rp, char-rp-fast, char-rp-reasoning, chat-judge, classifier, coder-fast, + erp-tune-v2, gemma4-26b-a4b-it-base, gen, gen-large, gen-reasoning, image-judge, + qwen-image-bench, qwen3-embedding, reranker, sec, sec-reasoning, summarizer, + summarizer-large + +Still served: the paid passthroughs (`gen-frontier*`, `glm*`, `kimi*`) and the irv-ml1 +locals. The gateway itself on ana-docker is healthy — it is the backends that are gone. + +## ⚠⚠ RECOVERY — do NOT just reset the breaker and walk away + +Every vLLM seat on fv-ml1 carries `restart: unless-stopped`. On boot **all of them +start loading simultaneously** — gen, mog-sec, erp-seat, gemma4-charrp, flash-next, +embed, rerank, reward, coder, scriberr — which is the single largest power transient +the box can produce, fed straight into a circuit that may have just tripped. That is a +re-trip, and a re-trip during model load can leave a half-written page cache and a +much longer recovery. + +**Preferred sequence:** + +1. Power the chassis on with **Docker masked**, so nothing auto-starts: + at the BMC/console, boot to the OS and before the network comes up run + `systemctl mask docker containerd` — or if the box is already up and loading, + `systemctl stop docker` immediately. +2. Confirm `nvidia-smi` sees all four cards and `zpool status tank` is ONLINE. +3. Unmask, then bring seats up **one at a time**, waiting for each to report healthy: + `gen` first (19 aliases depend on it), then embed/rerank/reward/coder, then + mog-sec, then the rest. `flash-next` LAST — it is the newest and least depended-on. +4. **Do not restart the MTP campaign.** It is the prime suspect. +5. Watch power while seats come up: `nvidia-smi --query-gpu=index,power.draw --format=csv`. + +**Do not reboot the OPNsense firewall** (standing operator directive; its reboot API +403s anyway). + +## Afterwards + +- ⭐ **The OOB design gap this exposes.** The cutover chose OPNsense-as-subnet-router + specifically so the BMC stays reachable when the GPU box is down. That works for + box-down/gateway-up. It does **nothing** for a site-wide power or gateway loss — + exactly what happened — because the BMC's only path to the fleet is through that + gateway. A genuine OOB path at FV needs something the FV circuit cannot take down: + an LTE/cellular console, or the BMC on a separate circuit with its own uplink. +- Get the actual circuit rating and the box's real peak draw, now that it is known to + have four cards and not two. Until then, treat concurrent multi-card load at FV as + unproven rather than safe. +- `services/flash-next-mtp-bench/power.log` on the box holds the per-card draw right + up to the cut. Recover it after boot — it is the only measurement of what the load + actually drew, and it survives on `/tank`, not in the container.