esh-pfi-infrastructure/persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md

[2026-09-13] Flash-Next seat stood up on fv-ml1 — then the FV site went dark under two-card load

Two things happened in one session and they are causally linked: a 176B model was put on one GPU with its embedding table in host RAM, and the attempt to benchmark it on a second GPU took Fountain Valley off the air.

What was built and is real

stacks/flash-next-seat/ — Qwen3.8-Flash-Next (abliterated) on fv-ml1 GPU 2, :8022. The first seat on the fleet whose weights do not fit its card. 176B total = a 125B main model + a 51B n-gram (PLE) lookup table that lives in pinned host RAM and is read by the GPU over CUDA UVA. Measured, not predicted:

weights resident       74.36 GiB of 95.6    (load 97-242 s depending on page cache)
KV cache               14.00 GiB pinned in bytes -> 560,654 tokens
context                262,144 (native max), 2.14x concurrency
engine host RSS        67 GiB (47.7 GiB table + runtime)
throughput (n=1!)      75.5 / 212.3 / 387.8 tok/s at concurrency 1 / 4 / 8

Checkpoint dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b99, 126.0 GiB at /tank/aimodels/qwen38-flash-next-abliterated-nvfp4. Composition measured from the manifest: 63.3 GiB NVFP4 W4A4 routed experts + 47.7 GiB FP8 PLE table + 14.9 GiB BF16 — so it is NOT an all-4-bit quant, and the single largest item on disk is the FP8 table, 38% of the download.

gen-large on the LiteLLM gateway, one alias, verified registered for both the infra-ops admin key and the shared all-agents key. Sampling is the checkpoint's own declared set (temp 1.0 / top_p 0.95 / top_k 20); presence_penalty, min_p and repetition_penalty deliberately unset because the checkpoint declares no canonical value. Commits f964a47, 7e62a07.

⭐⭐ The five things worth carrying forward

1. The offload is upstream and merged — via a DIFFERENT PR than the park predicted. vLLM #54371 "UVA PLE-offload and Engram tensor parallelism" merged 2026-09-09T14:32Z: pinned host memory + direct UVA row reads, no worker process. It supersedes #53899 (worker-based, now explicitly paused), and that matters because #53899's entire bug family — the TP=1 startup deadlock #53960, the pidfd_getfd/yama ptrace gate, the shared-CUDA-event race, silently one-step-stale PLE outputs under CUDA graphs — all came from the separate worker the UVA path does not have. Flag: --engram-config '{"cpu_offload": true}'. In v0.29.1rc0 and main, NOT in v0.29.0 (cut ~6 h before the merge), and not in any nightly-<sha> dated 2026-09-09 or earlier — the nightly builds at ~06:16 UTC.
2. ⚠ text_config.ple_embedding_dtype is the load-or-fail discriminator. vLLM picks the PLE table's format from that field FIRST. A checkpoint with an FP8 table and no such declaration resolves through ModelOpt's *.ple.* exclude to the unquantized method and dies on the scale tensor it does ship. Check it before trying any other community build.
3. ⚠⚠ --kv-cache-memory makes vLLM SKIP MEMORY PROFILING and ignore --gpu-memory-utilization. This inverts the usual "pin bytes for reproducibility" advice. 16.00 GiB was tried and a 155,497-token prefill drove GPU 2 to 97,074 of 97,887 MiB with the allocator logging "OOM on device 0 while trying to allocate 488636416 bytes (free: 422117376)" — 466 MiB wanted, 403 free. The request still completed, so nothing failed visibly; that is one step before the shape that crashed mog-sec twice. 14.00 GiB re-probes clean. Do not take vLLM's own "17.46 GiB to fully utilize gpu memory" line: it is computed from a profile taken at max-num-batched-tokens depth and sits 3.5 GiB above what a 150K request survives. max-num-batched-tokens is what bounds the activation peak — doubling max_model_len left the profiled peak unchanged at 1.65 GiB.
4. ⚠ MTP is OFF but NOT written off, and the operator was right to push back. The seat initially defaulted MTP off citing vLLM's recipe (worse at every concurrency on 4xH100: 8-36% less throughput, 32-173% more latency, ~36% acceptance). That was a cross-harness comparison — 4xH100/TP=4/Hopper vs 1xBlackwell/TP=1 — and our own rule says those are invalid, not merely noisy. The real gap: the recipe tested k=3 only, and mtp_num_hidden_layers is 1, so the draft head is a single module run autoregressively for k>1. If throughput falls monotonically in k while acceptance rises, k=1 may win and nobody has published that cell. Campaign built at services/flash-next-mtp-bench/ with repeats, a boot-to-boot noise floor, and acceptance-greater-than-zero as a positive control. It completed exactly one off_A rep before the power failed.
5. ⚠ "Healthy" and "reachable" are on opposite sides of the container boundary. The seat once ran Up 4 minutes (healthy) with PORTS=[] — its port bind had lost a race with a dying container, and the healthcheck passed throughout because it runs inside the container against localhost:8000. Verify docker port / the docker-proxy listener, not the healthcheck.

⚠⚠ And then the site went dark — see docs/runbooks/fv-site-dark-20260913.md

Operator authorised running the MTP campaign on idle GPU 3 in parallel with live testing on GPU 2, with the two-card power risk explicitly raised and accepted.

06:51:08  campaign starts on GPU 3; idle draw GPU0 3.80 / GPU1 3.88 / GPU2 14.27 / GPU3 6.75 W
06:54:39  off_A healthy; rep 1 clean
06:56:04  off_A rep 2 starts   <-- last line ever written
06:58:40  EVERY FV address unreachable, BMC included; all other sites 0% loss

Leading hypothesis (operator): the 1500 VA Eaton UPS overloaded and died. It fits better than a breaker trip because a UPS's output rating sits far below the circuit's, making it the first protective device to give — which explains why the site let go at two cards loaded rather than four, and why the ~25 W firewall died with it.

• ⚠ Will not self-recover. An overload-tripped UPS needs a human; a dead one needs replacing. Do not poll FV addresses waiting for it.
• ⚠ Do NOT use a UPS's surge-only outlets to exceed its rating. Both banks are downstream of one input cord/inlet/breaker — for a 120 V 1500 VA unit, a single NEMA 5-15P rated 12 A at maximum load, total. The surge bank bypasses the inverter, not the current rating. Bypass the UPS entirely instead.
• ⭐ Recover /tank/aimodels/flash-next-mtp-bench/power.log before anything else. It sampled all four cards every 10 s to the cut and lives on /tank, not in a container. No load figure exists anywhere else — only idle. Nothing should put a wattage into a UPS purchase until it has been read.
• Blast radius: 19 of 30 gateway aliases, probed not inferred. ⚠ There is no local fallback — every free local model was on fv-ml1; irv-ml1 runs no chat seat at all (TTS/ComfyUI/arbo/clipper on two partly-occupied Ampere cards). The only non-fv chat backends are paid. Any paid coverage must go under a new opt-in alias name, never a silent repoint of summarizer/gen/classifier.
• ⭐ The OOB design gap. OPNsense-as-subnet-router protects box-down/gateway-up and does nothing for a site-wide loss, because the BMC's only route out is through that same gateway. Proposed-not-ratified fix in the runbook: firewall + BMC on the UPS, GPU chassis on raw PDU.

Also landed: the renumber the FV cutover missed

Every fv-ml1 link on the Homepage dashboard was dead — measured 16 entries pointing at 10.250.50.54 and zero at 10.251.50.54. The sweep script's allowlist was built from files that mention the HOST, and a homepage.href mentions only an IP, so every label-only stack fell outside it by construction. Fixed across 24 files plus the host copies; allowlist extended with how to derive it next time. Commit 3132a16.

Two bugs fell out: deploy-stack.sh rejected any stack name containing a dot (so qwen3.5-122b, qwopus3.5-122b, mistral-medium-3.5 could not be deployed at all), and scriberr's CORS allowlist contained only the dead IP and the dead scriberr.ana.internal — an allowlist with nothing reachable in it.

⚠ Still incomplete: the 10 running containers were never recreated, so the labels are still stale on the host — and a plain power-on will NOT fix them, because labels attach only at creation. The staged docker compose up -d <svc> recovery sequence applies them as a side effect. Eight stacks were deliberately NOT pushed from canonical because their host copies have genuine drift; three of those are untracked host-only stacks that should be brought into stacks/.