First seat whose weights do not fit its card. The model is 176B total -- a 125B
main model plus a 51B n-gram (PLE) lookup table -- at ~6B active per token. The
table is a pure embedding lookup, so it lives in pinned host RAM and the GPU
reads rows directly over CUDA UVA: ~78 GiB resident on a 95.6 GiB card, 47.7 GiB
pinned of 566 GB. GPU 2 and GPU 3 were both idle, so this displaced nothing.
Checkpoint dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b99, pinned by
revision: NVFP4 W4A4 routed experts, FP8 PLE table, everything else at source
precision. Chosen over better-liked builds because its provenance states
protocols and repeat counts -- AIME26 pass@1 98.75% over 30x8 repeats with a
stated SEM, full-set GSM8K, and a byte-equality audit covering all 31 MTP
tensors -- and because it declares text_config.ple_embedding_dtype, which is the
field vLLM reads first when selecting the PLE weight format. Builds that ship an
FP8 table without that declaration resolve to the unquantized path and fail on
load; the README records the check.
Requires vLLM #54371 (UVA PLE-offload, merged 2026-09-09T14:32Z), verified by
ancestry: the pinned nightly is +150 commits / behind_by 0 from the merge commit.
Not in v0.29.0, cut six hours earlier. The older worker-based offload (#53899) is
paused upstream and is not the path here -- its deadlocks, ptrace gate and
stale-output-under-graphs bugs all came from the separate worker process that UVA
does not have.
Five deliberate departures from the other seats on this box, each from a
measurement rather than a preference, all annotated in place:
- no MTP: the vLLM recipe measured it worse at every concurrency on 4xH100
(8-36% less throughput, 32-173% more latency, ~36% acceptance)
- modelopt_fp4, not compressed-tensors: only the ModelOpt reader honours the
ignore list keeping attention, shared experts, PLE and MTP out of W4A4
- KV left at auto: fp8 KV on this model's QSA path is an unmerged RFC (#54426)
- mamba-cache-mode stated explicitly: the model raises on mode "all"
- 128K context and 8192 batched tokens, not the native 262K: #54764 and #54919
make depth the risky axis, and sizing to the KV pool has never fixed a
depth-driven crash on this hardware
Nothing is wired into LiteLLM. Pointing an alias at this seat changes what
existing callers receive and is a separate decision.
