vh
f964a47262
First seat whose weights do not fit its card. The model is 176B total -- a 125B main model plus a 51B n-gram (PLE) lookup table -- at ~6B active per token. The table is a pure embedding lookup, so it lives in pinned host RAM and the GPU reads rows directly over CUDA UVA: ~78 GiB resident on a 95.6 GiB card, 47.7 GiB pinned of 566 GB. GPU 2 and GPU 3 were both idle, so this displaced nothing. Checkpoint dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b99, pinned by revision: NVFP4 W4A4 routed experts, FP8 PLE table, everything else at source precision. Chosen over better-liked builds because its provenance states protocols and repeat counts -- AIME26 pass@1 98.75% over 30x8 repeats with a stated SEM, full-set GSM8K, and a byte-equality audit covering all 31 MTP tensors -- and because it declares text_config.ple_embedding_dtype, which is the field vLLM reads first when selecting the PLE weight format. Builds that ship an FP8 table without that declaration resolve to the unquantized path and fail on load; the README records the check. Requires vLLM #54371 (UVA PLE-offload, merged 2026-09-09T14:32Z), verified by ancestry: the pinned nightly is +150 commits / behind_by 0 from the merge commit. Not in v0.29.0, cut six hours earlier. The older worker-based offload (#53899) is paused upstream and is not the path here -- its deadlocks, ptrace gate and stale-output-under-graphs bugs all came from the separate worker process that UVA does not have. Five deliberate departures from the other seats on this box, each from a measurement rather than a preference, all annotated in place: - no MTP: the vLLM recipe measured it worse at every concurrency on 4xH100 (8-36% less throughput, 32-173% more latency, ~36% acceptance) - modelopt_fp4, not compressed-tensors: only the ModelOpt reader honours the ignore list keeping attention, shared experts, PLE and MTP out of W4A4 - KV left at auto: fp8 KV on this model's QSA path is an unmerged RFC (#54426) - mamba-cache-mode stated explicitly: the model raises on mode "all" - 128K context and 8192 batched tokens, not the native 262K: #54764 and #54919 make depth the risky axis, and sizing to the KV pool has never fixed a depth-driven crash on this hardware Nothing is wired into LiteLLM. Pointing an alias at this seat changes what existing callers receive and is a separate decision.
140 lines
7.6 KiB
Markdown
140 lines
7.6 KiB
Markdown
# flash-next-seat — Qwen3.8-Flash-Next (abliterated), fv-ml1 GPU 2, `:8022`
|
||
|
||
The first seat on the fleet whose weights do not fit its card and run anyway.
|
||
|
||
`Qwen3.8-Flash-Next` is 176B total — a 125B main model plus a **51B n-gram (PLE)
|
||
lookup table** — activating ~6B parameters per token. The n-gram table is a pure
|
||
embedding lookup with almost no compute per token, so it lives in **pinned host
|
||
RAM** and the GPU reads the rows it needs directly over **CUDA UVA** on a dedicated
|
||
stream with async prefetch.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Checkpoint | `dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4` @ `be794b99…` (126.0 GiB) |
|
||
| On the card | ~78 GiB of 95.6 GiB — routed experts NVFP4 W4A4, rest at source precision |
|
||
| In host RAM | 47.7 GiB pinned, FP8 E4M3, 10 `model-plefp8-*` shards + per-table scalar scale |
|
||
| Context | 131,072 to start (native ceiling 262,144) — see *Raising context* |
|
||
| Speculative decoding | **none** — see *Why MTP is off* |
|
||
| Gateway wiring | **none yet** — this seat is not in LiteLLM; `gen` is untouched |
|
||
|
||
## Deploy
|
||
|
||
```bash
|
||
scripts/deploy-stack.sh fv-ml1 flash-next-seat # diffs vs live, prompts y/N
|
||
# then on the host, first boot only:
|
||
ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/flash-next-seat && docker compose config >/dev/null && docker compose up -d'
|
||
```
|
||
|
||
The `.env` lives on the host and is never committed. Copy `.env.example`, set
|
||
`API_KEY`, and read the FIRST-BOOT annotations before changing anything else.
|
||
|
||
## Architecture, briefly
|
||
|
||
Four ideas, and three of them shape the serving config:
|
||
|
||
- **GDN + QSA.** 36 of 48 layers use Gated DeltaNet (linear attention) to compress
|
||
history; every fourth layer uses Qwen Sparse Attention for long-range retrieval.
|
||
This is why KV is cheap at depth and why `--mamba-cache-dtype float32` matters.
|
||
- **N-gram embedding.** The 51B lookup table that this seat offloads. Qwen's own
|
||
framing: capacity with almost no per-token compute.
|
||
- **Gated residual / hyper-connections.** Four residual branches; excluded from
|
||
quantization in this checkpoint.
|
||
- **MTP head.** Present and preserved byte-identically. Deliberately unused.
|
||
|
||
## Why this checkpoint, and the trap that disqualifies most others
|
||
|
||
vLLM selects the PLE table's weight format from **`text_config.ple_embedding_dtype`**,
|
||
as the *first* branch of `Qwen4ExpPLEEmbeddingMethod.from_quant_config`. This
|
||
checkpoint declares `"float8_e4m3fn"`.
|
||
|
||
A build that ships an FP8 PLE table **without** that declaration resolves instead
|
||
through ModelOpt's `*.ple.*` exclude to the *unquantized* method, never registers
|
||
the `weight_scale` parameter, and dies on load with `no module or parameter named
|
||
'ngram_embedding.weight_scale'`. `gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8`
|
||
is exactly this case. **Check that field before trying another build.**
|
||
|
||
Chosen over better-liked alternatives because its provenance states protocols and
|
||
repeat counts rather than adjectives. From its own `qualification-notes.md` and
|
||
metrics files, kept in the model directory:
|
||
|
||
- HarmBench, 240 genuinely-harmful behaviours, greedy: 100% compliance at reasoning
|
||
off / low / xhigh.
|
||
- MMLU 82.11% → 81.93% (−0.18 pp) on an identical harness, 2,280 questions.
|
||
- GSM8K 97.27% (1283/1319), full set, single-shot, temp 0.6 — inside the stated
|
||
BF16 reference band 97.12–97.50.
|
||
- AIME26 pass@1 98.75% (237/240, **SEM 0.61 pp, 30 problems × 8 repeats**),
|
||
majority@8 100%, `max_tokens` 130,000, 4.9M completion tokens, stop_rate 99.17%.
|
||
- Byte-equality audit of unchanged tensors: 1,562 tensors / 118.4 GB compared, all
|
||
passed, **including all 31 MTP tensors**.
|
||
|
||
⚠ Two honest gaps in that evidence. The routed experts are **NVFP4 W4A4**, and
|
||
nobody — including the publisher — has measured this checkpoint at the full 262K
|
||
context; AIME26's 130K-token generations are the deepest evidence that exists.
|
||
Separately, `validate_checkpoint_report.json` in the repo describes the *earlier
|
||
BF16-PLE revision* (204 shards / 173.6 GiB), not the published FP8-PLE one.
|
||
|
||
Rejected alternatives, for the record:
|
||
`orcarouter/…-Uncensored-NVFP4` is gated (access request pending nothing — not
|
||
requested); `nvidia/…-NVFP4` is the cleanest ModelOpt MIXED_PRECISION build but is
|
||
not abliterated; `lovedheart/…-Pruned-RTXPRO-6000` prunes to 448 of 512 experts.
|
||
|
||
## Why MTP is off
|
||
|
||
Against our house graft-MTP habit, and on purpose.
|
||
|
||
vLLM's own recipe for this model measured MTP on 4×H100 as **worse at every
|
||
concurrency tested** — 8–36% lower request throughput, 32–173% higher per-token
|
||
latency, driven by ~36% acceptance — and says do not enable it by default. Open
|
||
issue **#55357** reports episodic 0% draft acceptance with repetition collapse
|
||
inside thinking blocks. Open **#55496** reports ModelOpt `MIXED_PRECISION` failing
|
||
to load FP8_BLOCK_SCALES MTP experts.
|
||
|
||
Turning it on is two lines in `compose.yaml` (documented in place). If you do,
|
||
measure it **here**, with repeats, against this seat's own baseline — the numbers
|
||
above are someone else's hardware.
|
||
|
||
## The upstream situation, as of 2026-09-13
|
||
|
||
- **#53896** — model support. **Merged 2026-08-31.** In v0.29.0.
|
||
- **#54371** — *UVA PLE-offload and Engram tensor parallelism*. **Merged
|
||
2026-09-09T14:32Z.** This is the offload this seat uses. **Not in v0.29.0**,
|
||
which was cut ~6 h earlier; present in `v0.29.1rc0` and in any nightly from
|
||
2026-09-10 onward.
|
||
- **#53899** — the *older, worker-based* PLE offload. **Open and explicitly paused**
|
||
in favour of #54371. Do not go back to it. Its whole bug family — the TP=1
|
||
startup deadlock (#53960), the `pidfd_getfd` / `kernel.yama.ptrace_scope` gate,
|
||
the shared-CUDA-event race under async scheduling, and silently one-step-stale
|
||
PLE outputs under CUDA graphs — came from the separate worker process and the
|
||
CUDA-IPC row transfer that the UVA path does not have.
|
||
|
||
Open issues worth knowing about on SM120, none of them blocking:
|
||
|
||
| Issue | What it does | Our exposure |
|
||
|---|---|---|
|
||
| **#54173** | CUBLAS internal error / illegal memory access in the GDN path **with prefix caching** | We enable prefix caching. `FN_PREFIX_CACHING=` is the one-line rollback. |
|
||
| **#54764** | PLE short-conv batched prefill pads every request to the batch-max query length | Why `--max-num-batched-tokens` is 8192, not 16384 |
|
||
| **#54919** | Long prefill starves active decode for 3–7 minutes | Why context starts at 128K |
|
||
| **#54521** | Greedy decoding non-deterministic from `persistent_topk` in prefill | Affects any A/B on this seat — establish a noise floor before comparing |
|
||
| **#54426** | fp8_e4m3 KV on the QSA path is an unmerged RFC | Why `--kv-cache-dtype` is **not** set to fp8 here |
|
||
|
||
## Raising context
|
||
|
||
128K is a starting value, not a measured one. Before raising it, bisect with a
|
||
**non-repeating** prompt — a repeated one hashes to cached blocks and never
|
||
prefills deep, so it proves nothing. The `stacks/mog-sec` README records this the
|
||
hard way: three successive context cuts all sized the *KV pool* while the crashes
|
||
were governed by *processing depth*, which is a different number.
|
||
|
||
The point of a ceiling is the refusal. Below it the seat serves; above it vLLM
|
||
returns a clean 400 naming the limit, instead of the engine dying and taking every
|
||
in-flight request with it.
|
||
|
||
## Not done yet
|
||
|
||
- **Pin `--kv-cache-memory` in bytes** from the first boot's budget line, replacing
|
||
the 0.90 ratio. Same discipline as `stacks/mog-sec` and `stacks/erp-seat`.
|
||
- **Gateway wiring is deliberately absent.** Pointing any LiteLLM alias at this
|
||
seat — in particular displacing `gen` / `summarizer` / `classifier`, which is the
|
||
long-term intent recorded in henge item 49 — changes what every existing caller
|
||
receives and is the operator's call, not a deploy-time default.
|