# FV site dark — 2026-09-13 ~06:56Z

**Status: UNRESOLVED, needs hands at Fountain Valley.** Written while the site is
down so recovery does not have to be reconstructed from memory.

## What is down

Everything at Fountain Valley, measured 06:58:40Z from nh3-dev:

| target | result |
|---|---|
| `fv-ml1` 10.251.50.54 | 100% loss |
| `fv-ml1` mesh 100.64.0.7 | 100% loss |
| FV gateway 10.251.50.1 | 100% loss |
| FV gateway mesh 100.64.0.8 | 100% loss |
| **fv-ml1-bmc 10.251.250.50** | **100% loss** |
| `fv.phasefinal.com` (172.83.89.66) | no answer |

**My own path is healthy** — ana-docker, nh3-docker, esh-docker-vm and 1.1.1.1 all
answer with 0% loss, so this is FV-local, not an isolation of the observer.

## Timeline

    06:51:08  MTP campaign started on GPU 3 (:8023), operator-authorised, knowing
              the two-card power risk. Production seat already live on GPU 2.
    06:51:17  power sampler baseline: GPU0 3.80 W, GPU1 3.88 W, GPU2 14.27 W, GPU3 6.75 W
    06:54:39  off_A arm healthy after 211 s
    06:54:39  off_A rep 1 ran clean: 75.49 / 212.33 / 387.83 tok/s at conc 1/4/8
    06:56:04  off_A rep 2 started  <-- LAST LOG LINE
    06:58:40  every FV address unreachable, including the BMC

So the site went dark inside a ~2.5 minute window, roughly five minutes into
sustained bench load on GPU 3 with GPU 2 also resident/serving.

## Three candidate causes, and what distinguishes them

Cannot be distinguished remotely, because every FV path — including the BMC —
traverses the OPNsense gateway, and the gateway is also dark.

1. **Circuit tripped under two-card load.** Fits the timing and is the predicted
   failure: the Anaheim rack breaker tripped twice (2026-08-26, 2026-09-11) on this
   same chassis, and the FV circuit was specced while every record still said the box
   had **two** GPUs rather than four. ⭐ Distinguishing evidence: **the breaker is
   visibly tripped**, and the OPNsense box is dark too (it draws ~20-30 W and would
   survive anything short of a circuit/utility loss).
2. **OPNsense gateway crashed or rebooted**, taking all FV routing with it while the
   GPU box is fine. Distinguishing evidence: the GPU box's PSU fans/lights are on and
   the OPNsense box is the only dead thing.
3. **Upstream utility or colo-side power/network loss**, unrelated to us.
   Distinguishing evidence: the breaker is NOT tripped and nothing at the rack has
   power; neighbouring equipment is also dark.

⚠ Do not record cause 1 as fact until someone has looked. The timing is strong
circumstantial evidence, not a measurement — and the instrument that would have
measured it (the power log on fv-ml1) died with the box.

## Blast radius

**19 of 30 LiteLLM aliases are dark** — fv-ml1 backs most of the fleet's inference:

    char-rp, char-rp-fast, char-rp-reasoning, chat-judge, classifier, coder-fast,
    erp-tune-v2, gemma4-26b-a4b-it-base, gen, gen-large, gen-reasoning, image-judge,
    qwen-image-bench, qwen3-embedding, reranker, sec, sec-reasoning, summarizer,
    summarizer-large

Still served: the paid passthroughs (`gen-frontier*`, `glm*`, `kimi*`) and the irv-ml1
locals. The gateway itself on ana-docker is healthy — it is the backends that are gone.

## ⚠⚠ RECOVERY — do NOT just reset the breaker and walk away

Every vLLM seat on fv-ml1 carries `restart: unless-stopped`. On boot **all of them
start loading simultaneously** — gen, mog-sec, erp-seat, gemma4-charrp, flash-next,
embed, rerank, reward, coder, scriberr — which is the single largest power transient
the box can produce, fed straight into a circuit that may have just tripped. That is a
re-trip, and a re-trip during model load can leave a half-written page cache and a
much longer recovery.

**Preferred sequence:**

1. Power the chassis on with **Docker masked**, so nothing auto-starts:
   at the BMC/console, boot to the OS and before the network comes up run
   `systemctl mask docker containerd` — or if the box is already up and loading,
   `systemctl stop docker` immediately.
2. Confirm `nvidia-smi` sees all four cards and `zpool status tank` is ONLINE.
3. Unmask, then bring seats up **one at a time**, waiting for each to report healthy:
   `gen` first (19 aliases depend on it), then embed/rerank/reward/coder, then
   mog-sec, then the rest. `flash-next` LAST — it is the newest and least depended-on.
4. **Do not restart the MTP campaign.** It is the prime suspect.
5. Watch power while seats come up: `nvidia-smi --query-gpu=index,power.draw --format=csv`.

**Do not reboot the OPNsense firewall** (standing operator directive; its reboot API
403s anyway).

## Afterwards

- ⭐ **The OOB design gap this exposes.** The cutover chose OPNsense-as-subnet-router
  specifically so the BMC stays reachable when the GPU box is down. That works for
  box-down/gateway-up. It does **nothing** for a site-wide power or gateway loss —
  exactly what happened — because the BMC's only path to the fleet is through that
  gateway. A genuine OOB path at FV needs something the FV circuit cannot take down:
  an LTE/cellular console, or the BMC on a separate circuit with its own uplink.
- Get the actual circuit rating and the box's real peak draw, now that it is known to
  have four cards and not two. Until then, treat concurrent multi-card load at FV as
  unproven rather than safe.
- `services/flash-next-mtp-bench/power.log` on the box holds the per-card draw right
  up to the cut. Recover it after boot — it is the only measurement of what the load
  actually drew, and it survives on `/tank`, not in the container.