# `[2026-09-13]` Flash-Next seat stood up on fv-ml1 — then the FV site went dark under two-card load

Two things happened in one session and they are causally linked: a 176B model was put
on one GPU with its embedding table in host RAM, and the attempt to benchmark it on a
second GPU took Fountain Valley off the air.

## What was built and is real

**`stacks/flash-next-seat/` — Qwen3.8-Flash-Next (abliterated) on fv-ml1 GPU 2, `:8022`.**
The first seat on the fleet whose weights do not fit its card. 176B total = a 125B main
model + a **51B n-gram (PLE) lookup table** that lives in pinned host RAM and is read
by the GPU over **CUDA UVA**. Measured, not predicted:

    weights resident       74.36 GiB of 95.6    (load 97-242 s depending on page cache)
    KV cache               14.00 GiB pinned in bytes -> 560,654 tokens
    context                262,144 (native max), 2.14x concurrency
    engine host RSS        67 GiB (47.7 GiB table + runtime)
    throughput (n=1!)      75.5 / 212.3 / 387.8 tok/s at concurrency 1 / 4 / 8

Checkpoint `dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4` @ `be794b99`, 126.0 GiB at
`/tank/aimodels/qwen38-flash-next-abliterated-nvfp4`. Composition measured from the
manifest: **63.3 GiB NVFP4 W4A4 routed experts + 47.7 GiB FP8 PLE table + 14.9 GiB
BF16** — so it is NOT an all-4-bit quant, and the single largest item on disk is the
FP8 table, 38% of the download.

**`gen-large` on the LiteLLM gateway**, one alias, verified registered for both the
infra-ops admin key and the shared all-agents key. Sampling is the checkpoint's own
declared set (temp 1.0 / top_p 0.95 / top_k 20); presence_penalty, min_p and
repetition_penalty deliberately unset because the checkpoint declares no canonical
value. Commits `f964a47`, `7e62a07`.

## ⭐⭐ The five things worth carrying forward

1. **The offload is upstream and merged — via a DIFFERENT PR than the park predicted.**
   vLLM **#54371** "UVA PLE-offload and Engram tensor parallelism" merged
   2026-09-09T14:32Z: pinned host memory + direct UVA row reads, no worker process.
   It **supersedes #53899** (worker-based, now explicitly paused), and that matters
   because #53899's entire bug family — the TP=1 startup deadlock #53960, the
   `pidfd_getfd`/yama ptrace gate, the shared-CUDA-event race, silently one-step-stale
   PLE outputs under CUDA graphs — all came from the separate worker the UVA path does
   not have. Flag: `--engram-config '{"cpu_offload": true}'`. **In `v0.29.1rc0` and
   main, NOT in `v0.29.0`** (cut ~6 h before the merge), and not in any `nightly-<sha>`
   dated 2026-09-09 or earlier — the nightly builds at ~06:16 UTC.
2. **⚠ `text_config.ple_embedding_dtype` is the load-or-fail discriminator.** vLLM picks
   the PLE table's format from that field FIRST. A checkpoint with an FP8 table and no
   such declaration resolves through ModelOpt's `*.ple.*` exclude to the *unquantized*
   method and dies on the scale tensor it does ship. Check it before trying any other
   community build.
3. **⚠⚠ `--kv-cache-memory` makes vLLM SKIP MEMORY PROFILING and ignore
   `--gpu-memory-utilization`.** This inverts the usual "pin bytes for
   reproducibility" advice. 16.00 GiB was tried and a 155,497-token prefill drove GPU 2
   to 97,074 of 97,887 MiB with the allocator logging *"OOM on device 0 while trying to
   allocate 488636416 bytes (free: 422117376)"* — 466 MiB wanted, 403 free. The request
   still completed, so **nothing failed visibly**; that is one step before the shape
   that crashed mog-sec twice. 14.00 GiB re-probes clean. Do **not** take vLLM's own
   *"17.46 GiB to fully utilize gpu memory"* line: it is computed from a profile taken
   at `max-num-batched-tokens` depth and sits 3.5 GiB above what a 150K request
   survives. `max-num-batched-tokens` is what bounds the activation peak — doubling
   `max_model_len` left the profiled peak unchanged at 1.65 GiB.
4. **⚠ MTP is OFF but NOT written off, and the operator was right to push back.** The
   seat initially defaulted MTP off citing vLLM's recipe (worse at every concurrency on
   4xH100: 8-36% less throughput, 32-173% more latency, ~36% acceptance). **That was a
   cross-harness comparison** — 4xH100/TP=4/Hopper vs 1xBlackwell/TP=1 — and our own
   rule says those are invalid, not merely noisy. The real gap: the recipe tested
   **k=3 only**, and `mtp_num_hidden_layers` is **1**, so the draft head is a single
   module run autoregressively for k>1. If throughput falls monotonically in k while
   acceptance rises, **k=1 may win and nobody has published that cell.** Campaign built
   at `services/flash-next-mtp-bench/` with repeats, a boot-to-boot noise floor, and
   acceptance-greater-than-zero as a positive control. It completed exactly **one**
   `off_A` rep before the power failed.
5. **⚠ "Healthy" and "reachable" are on opposite sides of the container boundary.** The
   seat once ran `Up 4 minutes (healthy)` with **`PORTS=[]`** — its port bind had lost a
   race with a dying container, and the healthcheck passed throughout because it runs
   *inside* the container against `localhost:8000`. Verify `docker port` / the
   docker-proxy listener, not the healthcheck.

## ⚠⚠ And then the site went dark — see `docs/runbooks/fv-site-dark-20260913.md`

Operator authorised running the MTP campaign on idle GPU 3 in parallel with live
testing on GPU 2, with the two-card power risk explicitly raised and accepted.

    06:51:08  campaign starts on GPU 3; idle draw GPU0 3.80 / GPU1 3.88 / GPU2 14.27 / GPU3 6.75 W
    06:54:39  off_A healthy; rep 1 clean
    06:56:04  off_A rep 2 starts   <-- last line ever written
    06:58:40  EVERY FV address unreachable, BMC included; all other sites 0% loss

**Leading hypothesis (operator): the 1500 VA Eaton UPS overloaded and died.** It fits
better than a breaker trip because a UPS's output rating sits far below the circuit's,
making it the first protective device to give — which explains why the site let go at
**two** cards loaded rather than four, and why the ~25 W firewall died with it.

- ⚠ **Will not self-recover.** An overload-tripped UPS needs a human; a dead one needs
  replacing. Do not poll FV addresses waiting for it.
- ⚠ **Do NOT use a UPS's surge-only outlets to exceed its rating.** Both banks are
  downstream of one input cord/inlet/breaker — for a 120 V 1500 VA unit, a single
  NEMA 5-15P rated **12 A at maximum load**, total. The surge bank bypasses the
  inverter, not the current rating. Bypass the UPS entirely instead.
- ⭐ **Recover `/tank/aimodels/flash-next-mtp-bench/power.log` before anything else.**
  It sampled all four cards every 10 s to the cut and lives on `/tank`, not in a
  container. **No load figure exists anywhere else** — only idle. Nothing should put a
  wattage into a UPS purchase until it has been read.
- **Blast radius: 19 of 30 gateway aliases**, probed not inferred. ⚠ **There is no
  local fallback** — every free local model was on fv-ml1; irv-ml1 runs no chat seat at
  all (TTS/ComfyUI/arbo/clipper on two partly-occupied Ampere cards). The only non-fv
  chat backends are paid. Any paid coverage must go under a **new opt-in alias name**,
  never a silent repoint of `summarizer`/`gen`/`classifier`.
- ⭐ **The OOB design gap.** OPNsense-as-subnet-router protects box-down/gateway-up and
  does nothing for a site-wide loss, because the BMC's only route out is through that
  same gateway. Proposed-not-ratified fix in the runbook: firewall + BMC on the UPS,
  GPU chassis on raw PDU.

## Also landed: the renumber the FV cutover missed

Every fv-ml1 link on the Homepage dashboard was dead — measured **16 entries pointing
at 10.250.50.54 and zero at 10.251.50.54**. The sweep script's allowlist was built from
files that mention the HOST, and a `homepage.href` mentions only an IP, so every
label-only stack fell outside it by construction. Fixed across 24 files plus the host
copies; allowlist extended with how to derive it next time. Commit `3132a16`.

Two bugs fell out: `deploy-stack.sh` rejected any stack name containing a dot (so
`qwen3.5-122b`, `qwopus3.5-122b`, `mistral-medium-3.5` could not be deployed **at
all**), and scriberr's CORS allowlist contained only the dead IP and the dead
`scriberr.ana.internal` — an allowlist with nothing reachable in it.

⚠ **Still incomplete:** the 10 running containers were never recreated, so the labels
are still stale on the host — and a plain power-on will NOT fix them, because labels
attach only at creation. The staged `docker compose up -d <svc>` recovery sequence
applies them as a side effect. Eight stacks were deliberately NOT pushed from canonical
because their host copies have genuine drift; three of those are untracked host-only
stacks that should be brought into `stacks/`.