memory: snapshot — Flash-Next seat on one card, and the FV outage that followed
Durable capture so tomorrow's session does not have to reconstruct either half. Built and verified before the power failed: Qwen3.8-Flash-Next serving on a single RTX PRO 6000 with its 51B n-gram table pinned in host RAM and read over CUDA UVA -- 74.36 GiB weights resident, 14.00 GiB KV for 560,654 tokens at the full 262,144 context, 67 GiB host RSS -- plus a gen-large gateway alias verified end to end. The five findings worth carrying: the offload is #54371 (UVA, merged) which supersedes the paused worker-based #53899 and designs out its entire bug family; text_config.ple_embedding_dtype is the load-or-fail discriminator for any community build; --kv-cache-memory makes vLLM SKIP memory profiling and ignore gpu-memory-utilization, which inverts the usual pin-bytes advice and let a 16 GiB pin nearly OOM with no visible failure; MTP is off pending measurement here rather than written off, because the recipe's number is cross-harness and tested k=3 only while the head is one layer run autoregressively; and a container once reported (healthy) with no published port at all, because the healthcheck runs inside the boundary it was trusted to validate. Then the outage. Records it as will-not-self-recover, so no session wastes effort polling a dead site, and carries the three things that change the visit: bypass the UPS rather than using its surge-only bank (both banks share one 12 A inlet -- the surge bank bypasses the inverter, not the current rating), recover power.log before anything else because it is the only load measurement that exists anywhere, and bring seats up one at a time because ten restart:unless-stopped containers loading at once is the largest transient the box can make into whatever just failed. Also records what is still half-done: the stale homepage labels on the 10 containers that died before they could be recreated, which the staged bring-up fixes as a side effect, and the eight drifted stacks plus three untracked host-only stacks that were deliberately left for a deliberate reconciliation.
This commit is contained in:
@@ -0,0 +1,130 @@
|
||||
# `[2026-09-13]` Flash-Next seat stood up on fv-ml1 — then the FV site went dark under two-card load
|
||||
|
||||
Two things happened in one session and they are causally linked: a 176B model was put
|
||||
on one GPU with its embedding table in host RAM, and the attempt to benchmark it on a
|
||||
second GPU took Fountain Valley off the air.
|
||||
|
||||
## What was built and is real
|
||||
|
||||
**`stacks/flash-next-seat/` — Qwen3.8-Flash-Next (abliterated) on fv-ml1 GPU 2, `:8022`.**
|
||||
The first seat on the fleet whose weights do not fit its card. 176B total = a 125B main
|
||||
model + a **51B n-gram (PLE) lookup table** that lives in pinned host RAM and is read
|
||||
by the GPU over **CUDA UVA**. Measured, not predicted:
|
||||
|
||||
weights resident 74.36 GiB of 95.6 (load 97-242 s depending on page cache)
|
||||
KV cache 14.00 GiB pinned in bytes -> 560,654 tokens
|
||||
context 262,144 (native max), 2.14x concurrency
|
||||
engine host RSS 67 GiB (47.7 GiB table + runtime)
|
||||
throughput (n=1!) 75.5 / 212.3 / 387.8 tok/s at concurrency 1 / 4 / 8
|
||||
|
||||
Checkpoint `dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4` @ `be794b99`, 126.0 GiB at
|
||||
`/tank/aimodels/qwen38-flash-next-abliterated-nvfp4`. Composition measured from the
|
||||
manifest: **63.3 GiB NVFP4 W4A4 routed experts + 47.7 GiB FP8 PLE table + 14.9 GiB
|
||||
BF16** — so it is NOT an all-4-bit quant, and the single largest item on disk is the
|
||||
FP8 table, 38% of the download.
|
||||
|
||||
**`gen-large` on the LiteLLM gateway**, one alias, verified registered for both the
|
||||
infra-ops admin key and the shared all-agents key. Sampling is the checkpoint's own
|
||||
declared set (temp 1.0 / top_p 0.95 / top_k 20); presence_penalty, min_p and
|
||||
repetition_penalty deliberately unset because the checkpoint declares no canonical
|
||||
value. Commits `f964a47`, `7e62a07`.
|
||||
|
||||
## ⭐⭐ The five things worth carrying forward
|
||||
|
||||
1. **The offload is upstream and merged — via a DIFFERENT PR than the park predicted.**
|
||||
vLLM **#54371** "UVA PLE-offload and Engram tensor parallelism" merged
|
||||
2026-09-09T14:32Z: pinned host memory + direct UVA row reads, no worker process.
|
||||
It **supersedes #53899** (worker-based, now explicitly paused), and that matters
|
||||
because #53899's entire bug family — the TP=1 startup deadlock #53960, the
|
||||
`pidfd_getfd`/yama ptrace gate, the shared-CUDA-event race, silently one-step-stale
|
||||
PLE outputs under CUDA graphs — all came from the separate worker the UVA path does
|
||||
not have. Flag: `--engram-config '{"cpu_offload": true}'`. **In `v0.29.1rc0` and
|
||||
main, NOT in `v0.29.0`** (cut ~6 h before the merge), and not in any `nightly-<sha>`
|
||||
dated 2026-09-09 or earlier — the nightly builds at ~06:16 UTC.
|
||||
2. **⚠ `text_config.ple_embedding_dtype` is the load-or-fail discriminator.** vLLM picks
|
||||
the PLE table's format from that field FIRST. A checkpoint with an FP8 table and no
|
||||
such declaration resolves through ModelOpt's `*.ple.*` exclude to the *unquantized*
|
||||
method and dies on the scale tensor it does ship. Check it before trying any other
|
||||
community build.
|
||||
3. **⚠⚠ `--kv-cache-memory` makes vLLM SKIP MEMORY PROFILING and ignore
|
||||
`--gpu-memory-utilization`.** This inverts the usual "pin bytes for
|
||||
reproducibility" advice. 16.00 GiB was tried and a 155,497-token prefill drove GPU 2
|
||||
to 97,074 of 97,887 MiB with the allocator logging *"OOM on device 0 while trying to
|
||||
allocate 488636416 bytes (free: 422117376)"* — 466 MiB wanted, 403 free. The request
|
||||
still completed, so **nothing failed visibly**; that is one step before the shape
|
||||
that crashed mog-sec twice. 14.00 GiB re-probes clean. Do **not** take vLLM's own
|
||||
*"17.46 GiB to fully utilize gpu memory"* line: it is computed from a profile taken
|
||||
at `max-num-batched-tokens` depth and sits 3.5 GiB above what a 150K request
|
||||
survives. `max-num-batched-tokens` is what bounds the activation peak — doubling
|
||||
`max_model_len` left the profiled peak unchanged at 1.65 GiB.
|
||||
4. **⚠ MTP is OFF but NOT written off, and the operator was right to push back.** The
|
||||
seat initially defaulted MTP off citing vLLM's recipe (worse at every concurrency on
|
||||
4xH100: 8-36% less throughput, 32-173% more latency, ~36% acceptance). **That was a
|
||||
cross-harness comparison** — 4xH100/TP=4/Hopper vs 1xBlackwell/TP=1 — and our own
|
||||
rule says those are invalid, not merely noisy. The real gap: the recipe tested
|
||||
**k=3 only**, and `mtp_num_hidden_layers` is **1**, so the draft head is a single
|
||||
module run autoregressively for k>1. If throughput falls monotonically in k while
|
||||
acceptance rises, **k=1 may win and nobody has published that cell.** Campaign built
|
||||
at `services/flash-next-mtp-bench/` with repeats, a boot-to-boot noise floor, and
|
||||
acceptance-greater-than-zero as a positive control. It completed exactly **one**
|
||||
`off_A` rep before the power failed.
|
||||
5. **⚠ "Healthy" and "reachable" are on opposite sides of the container boundary.** The
|
||||
seat once ran `Up 4 minutes (healthy)` with **`PORTS=[]`** — its port bind had lost a
|
||||
race with a dying container, and the healthcheck passed throughout because it runs
|
||||
*inside* the container against `localhost:8000`. Verify `docker port` / the
|
||||
docker-proxy listener, not the healthcheck.
|
||||
|
||||
## ⚠⚠ And then the site went dark — see `docs/runbooks/fv-site-dark-20260913.md`
|
||||
|
||||
Operator authorised running the MTP campaign on idle GPU 3 in parallel with live
|
||||
testing on GPU 2, with the two-card power risk explicitly raised and accepted.
|
||||
|
||||
06:51:08 campaign starts on GPU 3; idle draw GPU0 3.80 / GPU1 3.88 / GPU2 14.27 / GPU3 6.75 W
|
||||
06:54:39 off_A healthy; rep 1 clean
|
||||
06:56:04 off_A rep 2 starts <-- last line ever written
|
||||
06:58:40 EVERY FV address unreachable, BMC included; all other sites 0% loss
|
||||
|
||||
**Leading hypothesis (operator): the 1500 VA Eaton UPS overloaded and died.** It fits
|
||||
better than a breaker trip because a UPS's output rating sits far below the circuit's,
|
||||
making it the first protective device to give — which explains why the site let go at
|
||||
**two** cards loaded rather than four, and why the ~25 W firewall died with it.
|
||||
|
||||
- ⚠ **Will not self-recover.** An overload-tripped UPS needs a human; a dead one needs
|
||||
replacing. Do not poll FV addresses waiting for it.
|
||||
- ⚠ **Do NOT use a UPS's surge-only outlets to exceed its rating.** Both banks are
|
||||
downstream of one input cord/inlet/breaker — for a 120 V 1500 VA unit, a single
|
||||
NEMA 5-15P rated **12 A at maximum load**, total. The surge bank bypasses the
|
||||
inverter, not the current rating. Bypass the UPS entirely instead.
|
||||
- ⭐ **Recover `/tank/aimodels/flash-next-mtp-bench/power.log` before anything else.**
|
||||
It sampled all four cards every 10 s to the cut and lives on `/tank`, not in a
|
||||
container. **No load figure exists anywhere else** — only idle. Nothing should put a
|
||||
wattage into a UPS purchase until it has been read.
|
||||
- **Blast radius: 19 of 30 gateway aliases**, probed not inferred. ⚠ **There is no
|
||||
local fallback** — every free local model was on fv-ml1; irv-ml1 runs no chat seat at
|
||||
all (TTS/ComfyUI/arbo/clipper on two partly-occupied Ampere cards). The only non-fv
|
||||
chat backends are paid. Any paid coverage must go under a **new opt-in alias name**,
|
||||
never a silent repoint of `summarizer`/`gen`/`classifier`.
|
||||
- ⭐ **The OOB design gap.** OPNsense-as-subnet-router protects box-down/gateway-up and
|
||||
does nothing for a site-wide loss, because the BMC's only route out is through that
|
||||
same gateway. Proposed-not-ratified fix in the runbook: firewall + BMC on the UPS,
|
||||
GPU chassis on raw PDU.
|
||||
|
||||
## Also landed: the renumber the FV cutover missed
|
||||
|
||||
Every fv-ml1 link on the Homepage dashboard was dead — measured **16 entries pointing
|
||||
at 10.250.50.54 and zero at 10.251.50.54**. The sweep script's allowlist was built from
|
||||
files that mention the HOST, and a `homepage.href` mentions only an IP, so every
|
||||
label-only stack fell outside it by construction. Fixed across 24 files plus the host
|
||||
copies; allowlist extended with how to derive it next time. Commit `3132a16`.
|
||||
|
||||
Two bugs fell out: `deploy-stack.sh` rejected any stack name containing a dot (so
|
||||
`qwen3.5-122b`, `qwopus3.5-122b`, `mistral-medium-3.5` could not be deployed **at
|
||||
all**), and scriberr's CORS allowlist contained only the dead IP and the dead
|
||||
`scriberr.ana.internal` — an allowlist with nothing reachable in it.
|
||||
|
||||
⚠ **Still incomplete:** the 10 running containers were never recreated, so the labels
|
||||
are still stale on the host — and a plain power-on will NOT fix them, because labels
|
||||
attach only at creation. The staged `docker compose up -d <svc>` recovery sequence
|
||||
applies them as a side effect. Eight stacks were deliberately NOT pushed from canonical
|
||||
because their host copies have genuine drift; three of those are untracked host-only
|
||||
stacks that should be brought into `stacks/`.
|
||||
Reference in New Issue
Block a user