esh-pfi-infrastructure/stacks/flash-next-seat

flash-next-seat — Qwen3.8-Flash-Next (abliterated), fv-ml1 GPU 2, :8022

The first seat on the fleet whose weights do not fit its card and run anyway.

Qwen3.8-Flash-Next is 176B total — a 125B main model plus a 51B n-gram (PLE) lookup table — activating ~6B parameters per token. The n-gram table is a pure embedding lookup with almost no compute per token, so it lives in pinned host RAM and the GPU reads the rows it needs directly over CUDA UVA on a dedicated stream with async prefetch.

Checkpoint	orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, PLE converted bf16→FP8 in-house 2026-09-14 (123.2 GiB)
On the card	~75 GiB of 95.6 GiB — weight-only on both axes: W4 float experts, W8 float attn, input_activations: null
In host RAM	47.7 GiB pinned, FP8 E4M3, 8 model-plefp8-* shards + one global BF16 scale
Context	262,144 (full native) — KV 344,155 tokens, 1.31x concurrency
Speculative decoding	MTP k=3 — 60.4% acceptance, mean acceptance length 2.81 (measured here, n=5)
Gateway wiring	8 aliases — gen, gen-reasoning, summarizer(-large), classifier, chat-judge, image-judge, qwen-image-bench

Deploy

scripts/deploy-stack.sh fv-ml1 flash-next-seat     # diffs vs live, prompts y/N
# then on the host, first boot only:
ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/flash-next-seat && docker compose config >/dev/null && docker compose up -d'

The .env lives on the host and is never committed. Copy .env.example, set API_KEY, and read the FIRST-BOOT annotations before changing anything else.

Architecture, briefly

Four ideas, and three of them shape the serving config:

• GDN + QSA. 36 of 48 layers use Gated DeltaNet (linear attention) to compress history; every fourth layer uses Qwen Sparse Attention for long-range retrieval. This is why KV is cheap at depth and why --mamba-cache-dtype float32 matters.
• N-gram embedding. The 51B lookup table that this seat offloads. Qwen's own framing: capacity with almost no per-token compute.
• Gated residual / hyper-connections. Four residual branches; excluded from quantization in this checkpoint.
• MTP head. Present, preserved byte-identically, and in use at k=3.

Why this checkpoint, and the two traps in front of it

Chosen for the activation axis: orcarouter's build is weight-only on both halves — config_groups gives W8 float for attention/dense and W4 float for the experts, with input_activations: null on each. The displaced dealignai build is ModelOpt W4A4 (4-bit activations), the long-context degradation mode. Same author as the gen seat.

It did not load out of the box, and there were two independent config-level blockers. Both are recorded here because each looks like a capability gap and neither is one.

Trap 1 — the PLE loader (and the claim we had wrong)

vLLM picks the PLE table's format in Qwen4ExpPLEEmbeddingMethod.from_quant_config:

1. ple_embedding_dtype == "float8_e4m3fn" -> FP8 method   <-- BEFORE any type check
2. quant_config is None                   -> unquantized
3. ModelOptMixedPrecisionConfig           -> FP8 / unquantized
4. ModelOptQuantConfigBase + excluded     -> unquantized
5. not isinstance(quant_config, Fp8Config)-> NotImplementedError

⚠ This README previously said an FP8 PLE without the declaration is disqualifying, and that compressed-tensors needs a vLLM source patch. Both were wrong (corrected 2026-09-14; see the quantization playbook's superseded-claims table). Branch 1 is unconditional, and the NotImplementedError is scoped to the PLE path only — experts and dense layers of a compressed-tensors build load through vLLM's ordinary compressed-tensors paths. So declaring an FP8 PLE bypasses the blocker on stock mainline.

orcarouter ships a bf16 PLE, so the honest fix was to make the declaration true: convert the table to FP8, then declare it. Its 128 PLE tensors sit in exactly one shard file with nothing else in it, which makes that a clean, cheap rewrite.

⚠ Declare only what is true. gorbatjovy/...-NVFP4-plefp8 ships an FP8 table with no declaration and dies on ngram_embedding.weight_scale; declaring FP8 over a bf16 table is that same failure in reverse. The declaration is a claim about the bytes, not a switch.

Trap 2 — Invalid layer_type qwen_sparse_attention

orcarouter labels its 12 QSA layers qwen_sparse_attention. vLLM accepts only linear_attention and full_attention, and selects QSA within full_attention when indexer_n_heads is present. The fix is renaming the 12 entries.

⚠⚠ Check indexer_n_heads before renaming. Without it the rename silently selects plain Qwen3NextAttention instead of Qwen4ExpQSAAttention — a subtly wrong model that loads, serves, and passes a healthcheck. Verified indexer_n_heads == 4 in both this checkpoint and the dealignai one, along with every other indexer/QSA key, before touching it.

The conversion, and what it cost

Global amax 0.0894 with a per-shard outlier ratio of only 1.66x, so the single global scale this method uses is well-conditioned here. The scale is chosen exactly representable in bf16 (2.002716e-04) so no scale-rounding error stacks on the quantization error; amax maps to 446.17 of 448, so nothing clips. Round-trip 2.655% RMS relative, 0.002% underflow, zero saturation — and the same FP8-PLE treatment dealignai already shipped, so it is not a regression against the seat it replaced. weight_scale is written BF16 [1] to match the published format. MTP head (31 tensors) and the vision tower carry through untouched.

⚠ Still unmeasured: a controlled quality A/B against dealignai — which is the entire reason for the swap — and a deep-prefill probe at 262K on this checkpoint. Rollback is two .env keys; the dealignai checkpoint is still on disk.

Rejected alternatives, for the record: nvidia/…-NVFP4 is the cleanest ModelOpt build but is not abliterated; lovedheart/…-Pruned-RTXPRO-6000 prunes to 448 of 512 experts; windowsxp811203/…-Abliterated-NVFP4 stores its 95 GiB PLE as a single malformed ple_embedding.shard_.weight instead of 128 ngram_embedding.shard_N.weight and has never been served by its own author.

Why MTP is ON at k=3 (reversing this seat's original default)

This seat shipped with speculative decoding off, citing vLLM's recipe: on 4xH100 at TP=4 that recipe measured MTP worse at every concurrency (8-36% lower throughput, 32-173% higher per-token latency, ~36% acceptance) and says do not default it on. Open #55357 reports episodic 0% acceptance with repetition collapse inside thinking blocks.

Measured here, that inverted. The campaign in services/flash-next-mtp-bench/ found MTP a win at every k and every concurrency tested on one Blackwell card (+29/41/27% at k=1, +42/52/38% at k=2, +52/51/34% at k=3 across conc 1/4/8). k=3 is deployed because this is a single-user fleet and conc=1 dominates.

On the current orcarouter checkpoint, measured 2026-09-14: 60.4% acceptance, mean acceptance length 2.81 (per-position 80.6 / 60.8 / 40.8%), warm decode median 167.5 tok/s at conc=1 (n=5, spread 12.2%).

⚠ MTP costs KV. The draft head adds ~5.08 GiB of weights and raises per-token KV cost ~16%; FN_KV_CACHE_MEMORY was cut 14 -> 10 GiB for it. At 14 GiB the engine OOMs at init with MTP on. If it OOMs, drop to 8589934592.

⚠ The recipe's numbers are someone else's hardware, and so are ours to anyone else. Re-measure on the seat, warm, with repeats — the first decode bench during the reorg read 39 tok/s and that was a cold-boot + contention artifact, not a result.

The upstream situation, as of 2026-09-13

• #53896 — model support. Merged 2026-08-31. In v0.29.0.
• #54371 — UVA PLE-offload and Engram tensor parallelism. Merged 2026-09-09T14:32Z. This is the offload this seat uses. Not in v0.29.0, which was cut ~6 h earlier; present in v0.29.1rc0 and in any nightly from 2026-09-10 onward.
• #53899 — the older, worker-based PLE offload. Open and explicitly paused in favour of #54371. Do not go back to it. Its whole bug family — the TP=1 startup deadlock (#53960), the pidfd_getfd / kernel.yama.ptrace_scope gate, the shared-CUDA-event race under async scheduling, and silently one-step-stale PLE outputs under CUDA graphs — came from the separate worker process and the CUDA-IPC row transfer that the UVA path does not have.

Open issues worth knowing about on SM120, none of them blocking:

Issue	What it does	Our exposure
#54173	CUBLAS internal error / illegal memory access in the GDN path with prefix caching	We enable prefix caching. FN_PREFIX_CACHING= is the one-line rollback.
#54764	PLE short-conv batched prefill pads every request to the batch-max query length	Why --max-num-batched-tokens is 8192, not 16384
#54919	Long prefill starves active decode for 3–7 minutes	Why context starts at 128K
#54521	Greedy decoding non-deterministic from persistent_topk in prefill	Affects any A/B on this seat — establish a noise floor before comparing
#54426	fp8_e4m3 KV on the QSA path is an unmerged RFC	Why --kv-cache-dtype is not set to fp8 here

Raising context

128K is a starting value, not a measured one. Before raising it, bisect with a non-repeating prompt — a repeated one hashes to cached blocks and never prefills deep, so it proves nothing. The stacks/mog-sec README records this the hard way: three successive context cuts all sized the KV pool while the crashes were governed by processing depth, which is a different number.

The point of a ceiling is the refusal. Below it the seat serves; above it vLLM returns a clean 400 naming the limit, instead of the engine dying and taking every in-flight request with it.

Not done yet

• Pin --kv-cache-memory in bytes from the first boot's budget line, replacing the 0.90 ratio. Same discipline as stacks/mog-sec and stacks/erp-seat.
• Gateway wiring is deliberately absent. Pointing any LiteLLM alias at this seat — in particular displacing gen / summarizer / classifier, which is the long-term intent recorded in henge item 49 — changes what every existing caller receives and is the operator's call, not a deploy-time default.