esh-pfi-infrastructure/stacks/flash-next-seat/README.md

flash-next-seat — Qwen3.8-Flash-Next (abliterated), fv-ml1 GPU 2, :8022

The first seat on the fleet whose weights do not fit its card and run anyway.

Qwen3.8-Flash-Next is 176B total — a 125B main model plus a 51B n-gram (PLE) lookup table — activating ~6B parameters per token. The n-gram table is a pure embedding lookup with almost no compute per token, so it lives in pinned host RAM and the GPU reads the rows it needs directly over CUDA UVA on a dedicated stream with async prefetch.

Checkpoint	orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, PLE converted bf16→FP8 in-house 2026-09-14 (123.2 GiB)
On the card	~75 GiB of 95.6 GiB — weight-only on both axes: W4 float experts, W8 float attn, input_activations: null
In host RAM	47.7 GiB pinned, FP8 E4M3, 8 model-plefp8-* shards + one global BF16 scale
Context	262,144 (full native) — KV 344,155 tokens, 1.31x concurrency
Speculative decoding	MTP k=3 — 60.4% acceptance, mean acceptance length 2.81 (measured here, n=5)
Gateway wiring	8 aliases — gen, gen-reasoning, summarizer(-large), classifier, chat-judge, image-judge, qwen-image-bench

Deploy

scripts/deploy-stack.sh fv-ml1 flash-next-seat     # diffs vs live, prompts y/N
# then on the host, first boot only:
ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/flash-next-seat && docker compose config >/dev/null && docker compose up -d'

The .env lives on the host and is never committed. Copy .env.example, set API_KEY, and read the FIRST-BOOT annotations before changing anything else.

Architecture, briefly

Four ideas, and three of them shape the serving config:

• GDN + QSA. 36 of 48 layers use Gated DeltaNet (linear attention) to compress history; every fourth layer uses Qwen Sparse Attention for long-range retrieval. This is why KV is cheap at depth and why --mamba-cache-dtype float32 matters.
• N-gram embedding. The 51B lookup table that this seat offloads. Qwen's own framing: capacity with almost no per-token compute.
• Gated residual / hyper-connections. Four residual branches; excluded from quantization in this checkpoint.
• MTP head. Present, preserved byte-identically, and in use at k=3.

Why this checkpoint, and the two traps in front of it

Chosen for the activation axis: orcarouter's build is weight-only on both halves — config_groups gives W8 float for attention/dense and W4 float for the experts, with input_activations: null on each. The displaced dealignai build is ModelOpt W4A4 (4-bit activations), the long-context degradation mode. Same author as the gen seat.

It did not load out of the box, and there were two independent config-level blockers. Both are recorded here because each looks like a capability gap and neither is one.

Trap 1 — the PLE loader (and the claim we had wrong)

vLLM picks the PLE table's format in Qwen4ExpPLEEmbeddingMethod.from_quant_config:

1. ple_embedding_dtype == "float8_e4m3fn" -> FP8 method   <-- BEFORE any type check
2. quant_config is None                   -> unquantized
3. ModelOptMixedPrecisionConfig           -> FP8 / unquantized
4. ModelOptQuantConfigBase + excluded     -> unquantized
5. not isinstance(quant_config, Fp8Config)-> NotImplementedError

⚠ This README previously said an FP8 PLE without the declaration is disqualifying, and that compressed-tensors needs a vLLM source patch. Both were wrong (corrected 2026-09-14; see the quantization playbook's superseded-claims table). Branch 1 is unconditional, and the NotImplementedError is scoped to the PLE path only — experts and dense layers of a compressed-tensors build load through vLLM's ordinary compressed-tensors paths. So declaring an FP8 PLE bypasses the blocker on stock mainline.

orcarouter ships a bf16 PLE, so the honest fix was to make the declaration true: convert the table to FP8, then declare it. Its 128 PLE tensors sit in exactly one shard file with nothing else in it, which makes that a clean, cheap rewrite.

⚠ Declare only what is true. gorbatjovy/...-NVFP4-plefp8 ships an FP8 table with no declaration and dies on ngram_embedding.weight_scale; declaring FP8 over a bf16 table is that same failure in reverse. The declaration is a claim about the bytes, not a switch.

Trap 2 — Invalid layer_type qwen_sparse_attention

orcarouter labels its 12 QSA layers qwen_sparse_attention. vLLM accepts only linear_attention and full_attention, and selects QSA within full_attention when indexer_n_heads is present. The fix is renaming the 12 entries.

⚠⚠ Check indexer_n_heads before renaming. Without it the rename silently selects plain Qwen3NextAttention instead of Qwen4ExpQSAAttention — a subtly wrong model that loads, serves, and passes a healthcheck. Verified indexer_n_heads == 4 in both this checkpoint and the dealignai one, along with every other indexer/QSA key, before touching it.

The conversion, and what it cost

Global amax 0.0894 with a per-shard outlier ratio of only 1.66x, so the single global scale this method uses is well-conditioned here. The scale is chosen exactly representable in bf16 (2.002716e-04) so no scale-rounding error stacks on the quantization error; amax maps to 446.17 of 448, so nothing clips. Round-trip 2.655% RMS relative, 0.002% underflow, zero saturation — and the same FP8-PLE treatment dealignai already shipped, so it is not a regression against the seat it replaced. weight_scale is written BF16 [1] to match the published format. MTP head (31 tensors) and the vision tower carry through untouched.

⚠⚠ THERE IS NO LOCAL ROLLBACK. The dealignai checkpoint was deleted on operator instruction 2026-09-14 (125 GiB reclaimed). Reverting this seat now means re-downloading 126 GiB from dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4, not flipping two .env keys. The .env backup (.env.bak-preorca-20260914-023408) still names the old paths, but those paths no longer exist — treat it as a record of the old settings, not a working revert.

⚠ Still unmeasured, and now unbacked: a controlled quality A/B against dealignai — which was the entire reason for the swap, and whose reference arm is gone — and a deep-prefill probe at 262K on this checkpoint. The 170 GiB pristine qwen38-flash-next-orcarouter-nvfp4 download is retained; it is what lets the PLE conversion be redone without re-fetching.

Rejected alternatives, for the record: nvidia/…-NVFP4 is the cleanest ModelOpt build but is not abliterated; lovedheart/…-Pruned-RTXPRO-6000 prunes to 448 of 512 experts; windowsxp811203/…-Abliterated-NVFP4 stores its 95 GiB PLE as a single malformed ple_embedding.shard_.weight instead of 128 ngram_embedding.shard_N.weight and has never been served by its own author.

Why MTP is ON at k=3 (reversing this seat's original default)

This seat shipped with speculative decoding off, citing vLLM's recipe: on 4xH100 at TP=4 that recipe measured MTP worse at every concurrency (8-36% lower throughput, 32-173% higher per-token latency, ~36% acceptance) and says do not default it on. Open #55357 reports episodic 0% acceptance with repetition collapse inside thinking blocks.

Measured here, that inverted. The campaign in services/flash-next-mtp-bench/ found MTP a win at every k and every concurrency tested on one Blackwell card (+29/41/27% at k=1, +42/52/38% at k=2, +52/51/34% at k=3 across conc 1/4/8). k=3 is deployed because this is a single-user fleet and conc=1 dominates.

On the current orcarouter checkpoint, measured 2026-09-14: 60.4% acceptance, mean acceptance length 2.81 (per-position 80.6 / 60.8 / 40.8%), warm decode median 167.5 tok/s at conc=1 (n=5, spread 12.2%).

⚠ MTP costs KV. The draft head adds ~5.08 GiB of weights and raises per-token KV cost ~16%; FN_KV_CACHE_MEMORY was cut 14 -> 10 GiB for it. At 14 GiB the engine OOMs at init with MTP on. If it OOMs, drop to 8589934592.

⚠ The recipe's numbers are someone else's hardware, and so are ours to anyone else. Re-measure on the seat, warm, with repeats — the first decode bench during the reorg read 39 tok/s and that was a cold-boot + contention artifact, not a result.

The upstream situation, as of 2026-09-13

• #53896 — model support. Merged 2026-08-31. In v0.29.0.
• #54371 — UVA PLE-offload and Engram tensor parallelism. Merged 2026-09-09T14:32Z. This is the offload this seat uses. Not in v0.29.0, which was cut ~6 h earlier; present in v0.29.1rc0 and in any nightly from 2026-09-10 onward.
• #53899 — the older, worker-based PLE offload. Open and explicitly paused in favour of #54371. Do not go back to it. Its whole bug family — the TP=1 startup deadlock (#53960), the pidfd_getfd / kernel.yama.ptrace_scope gate, the shared-CUDA-event race under async scheduling, and silently one-step-stale PLE outputs under CUDA graphs — came from the separate worker process and the CUDA-IPC row transfer that the UVA path does not have.

Open issues worth knowing about on SM120, none of them blocking:

Issue	What it does	Our exposure
#54173	CUBLAS internal error / illegal memory access in the GDN path with prefix caching	We enable prefix caching. FN_PREFIX_CACHING= is the one-line rollback.
#54764	PLE short-conv batched prefill pads every request to the batch-max query length	Why --max-num-batched-tokens is 8192, not 16384
#54919	Long prefill starves active decode for 3–7 minutes	Why context starts at 128K
#54521	Greedy decoding non-deterministic from persistent_topk in prefill	Affects any A/B on this seat — establish a noise floor before comparing
#54426	fp8_e4m3 KV on the QSA path is an unmerged RFC	Why --kv-cache-dtype is not set to fp8 here

Context depth — PROBED 2026-09-14, clean to 258,517 tokens

262,144 is the configured ceiling and it has now been bisected with a non-repeating prompt (unique random hex per probe, so prefix caching cannot short-circuit the prefill — a repeated prompt hashes to cached blocks and never prefills deep).

prompt tokens	31,978	64,154	128,191	196,172	240,290	258,517
result	ok	ok	ok	ok	ok	ok

The load-bearing evidence is the engine's own allocator log: zero OOM / CUBLAS / illegal-memory / traceback entries across the whole run — the same detector that caught the dealignai near-miss (OOM on device 0 ... 466 MiB wanted, 403 MiB free) at 155K on the previous checkpoint. It fired then; it is silent here. The run also happened under real concurrent operator load, which makes it a stricter test than a solo probe, not a weaker one.

Positive control passed. A mis-sized first attempt built a ~265K-token prompt and got a clean 400 naming the limit rather than killing the engine — so the probe could detect the failure it was looking for. (Calibration for anyone re-running it: random hex words tokenize at 7.9 tokens/word on this tokenizer.)

#54919 did not reproduce. That issue reports long prefill starving decode for 3-7 minutes; 258K prefilled in 28.9 s (~8,900 tok/s), scaling near-linearly from 32K.

⚠⚠ DO NOT reuse the memory-headroom half of that probe — the gauge was blind. It sampled nvidia-smi before and after each request and reported an identical 95,460 MiB / 2,427 MiB free on every row. Two reasons: --kv-cache-memory pins the pool and the engine log says it "skipped memory profiling", so GPU usage is constant regardless of depth; and the risk is a transient activation spike during prefill, which before/after bracketing structurally cannot see. Identical readings across a 8x range of depths are the tell. Real peak-activation headroom needs in-process sampling during the prefill. The pass/fail result stands on the allocator log, not on that column.

Not done yet

• Pin --kv-cache-memory in bytes from the first boot's budget line, replacing the 0.90 ratio. Same discipline as stacks/mog-sec and stacks/erp-seat.
• Gateway wiring is deliberately absent. Pointing any LiteLLM alias at this seat — in particular displacing gen / summarizer / classifier, which is the long-term intent recorded in henge item 49 — changes what every existing caller receives and is the operator's call, not a deploy-time default.