esh-pfi-infrastructure/stacks/flash-next-seat/README.md

# flash-next-seat — Qwen3.8-Flash-Next (abliterated), fv-ml1 GPU 2, `:8022`

The first seat on the fleet whose weights do not fit its card and run anyway.

`Qwen3.8-Flash-Next` is 176B total — a 125B main model plus a **51B n-gram (PLE)
lookup table** — activating ~6B parameters per token. The n-gram table is a pure
embedding lookup with almost no compute per token, so it lives in **pinned host
RAM** and the GPU reads the rows it needs directly over **CUDA UVA** on a dedicated
stream with async prefetch.

| | |
|---|---|
| Checkpoint | `orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4`, PLE converted bf16→FP8 in-house 2026-09-14 (123.2 GiB) |
| On the card | ~75 GiB of 95.6 GiB — **weight-only on both axes**: W4 float experts, W8 float attn, `input_activations: null` |
| In host RAM | 47.7 GiB pinned, FP8 E4M3, 8 `model-plefp8-*` shards + one global BF16 scale |
| Context | **262,144** (full native) — KV 344,155 tokens, 1.31x concurrency |
| Speculative decoding | **MTP k=3** — 60.4% acceptance, mean acceptance length 2.81 (measured here, n=5) |
| Gateway wiring | **8 aliases** — gen, gen-reasoning, summarizer(-large), classifier, chat-judge, image-judge, qwen-image-bench |

## Deploy

```bash
scripts/deploy-stack.sh fv-ml1 flash-next-seat     # diffs vs live, prompts y/N
# then on the host, first boot only:
ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/flash-next-seat && docker compose config >/dev/null && docker compose up -d'
```

The `.env` lives on the host and is never committed. Copy `.env.example`, set
`API_KEY`, and read the FIRST-BOOT annotations before changing anything else.

## Architecture, briefly

Four ideas, and three of them shape the serving config:

- **GDN + QSA.** 36 of 48 layers use Gated DeltaNet (linear attention) to compress
  history; every fourth layer uses Qwen Sparse Attention for long-range retrieval.
  This is why KV is cheap at depth and why `--mamba-cache-dtype float32` matters.
- **N-gram embedding.** The 51B lookup table that this seat offloads. Qwen's own
  framing: capacity with almost no per-token compute.
- **Gated residual / hyper-connections.** Four residual branches; excluded from
  quantization in this checkpoint.
- **MTP head.** Present, preserved byte-identically, and **in use at k=3**.

## Why this checkpoint, and the two traps in front of it

Chosen for the **activation axis**: orcarouter's build is weight-only on *both* halves —
`config_groups` gives W8 float for attention/dense and W4 float for the experts, with
`input_activations: null` on each. The displaced dealignai build is ModelOpt **W4A4**
(4-bit activations), the long-context degradation mode. Same author as the `gen` seat.

It did not load out of the box, and there were **two independent config-level blockers**.
Both are recorded here because each looks like a capability gap and neither is one.

### Trap 1 — the PLE loader (and the claim we had wrong)

vLLM picks the PLE table's format in `Qwen4ExpPLEEmbeddingMethod.from_quant_config`:

```
1. ple_embedding_dtype == "float8_e4m3fn" -> FP8 method   <-- BEFORE any type check
2. quant_config is None                   -> unquantized
3. ModelOptMixedPrecisionConfig           -> FP8 / unquantized
4. ModelOptQuantConfigBase + excluded     -> unquantized
5. not isinstance(quant_config, Fp8Config)-> NotImplementedError
```

⚠ **This README previously said an FP8 PLE without the declaration is disqualifying, and
that compressed-tensors needs a vLLM source patch. Both were wrong** (corrected 2026-09-14;
see the quantization playbook's superseded-claims table). Branch 1 is **unconditional**, and
the `NotImplementedError` is **scoped to the PLE path only** — experts and dense layers of a
compressed-tensors build load through vLLM's ordinary compressed-tensors paths. So declaring
an FP8 PLE bypasses the blocker on stock mainline.

orcarouter ships a **bf16** PLE, so the honest fix was to *make the declaration true*:
convert the table to FP8, then declare it. Its 128 PLE tensors sit in exactly one shard file
with nothing else in it, which makes that a clean, cheap rewrite.

⚠ **Declare only what is true.** `gorbatjovy/...-NVFP4-plefp8` ships an FP8 table with no
declaration and dies on `ngram_embedding.weight_scale`; declaring FP8 over a *bf16* table is
that same failure in reverse. The declaration is a claim about the bytes, not a switch.

### Trap 2 — `Invalid layer_type qwen_sparse_attention`

orcarouter labels its 12 QSA layers `qwen_sparse_attention`. vLLM accepts only
`linear_attention` and `full_attention`, and selects QSA *within* `full_attention` when
`indexer_n_heads` is present. The fix is renaming the 12 entries.

⚠⚠ **Check `indexer_n_heads` before renaming.** Without it the rename silently selects plain
`Qwen3NextAttention` instead of `Qwen4ExpQSAAttention` — a subtly wrong model that loads,
serves, and passes a healthcheck. Verified `indexer_n_heads == 4` in both this checkpoint and
the dealignai one, along with every other indexer/QSA key, before touching it.

### The conversion, and what it cost

Global amax 0.0894 with a per-shard outlier ratio of only **1.66x**, so the single global
scale this method uses is well-conditioned here. The scale is chosen **exactly representable
in bf16** (2.002716e-04) so no scale-rounding error stacks on the quantization error; amax
maps to 446.17 of 448, so nothing clips. Round-trip **2.655% RMS relative**, 0.002% underflow,
zero saturation — and the same FP8-PLE treatment dealignai already shipped, so it is not a
regression against the seat it replaced. `weight_scale` is written BF16 [1] to match the
published format. MTP head (31 tensors) and the vision tower carry through untouched.

⚠⚠ **THERE IS NO LOCAL ROLLBACK.** The dealignai checkpoint was deleted on operator
instruction 2026-09-14 (125 GiB reclaimed). Reverting this seat now means **re-downloading
126 GiB** from `dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4`, not flipping two `.env`
keys. The `.env` backup (`.env.bak-preorca-20260914-023408`) still names the old paths, but
those paths no longer exist — treat it as a record of the old settings, not a working revert.

⚠ Still unmeasured, and now unbacked: a controlled quality A/B against dealignai — which was
the entire reason for the swap, and whose reference arm is gone — and a deep-prefill probe at
262K on this checkpoint. The 170 GiB pristine `qwen38-flash-next-orcarouter-nvfp4` download is
retained; it is what lets the PLE conversion be redone without re-fetching.

Rejected alternatives, for the record: `nvidia/…-NVFP4` is the cleanest ModelOpt build but is
not abliterated; `lovedheart/…-Pruned-RTXPRO-6000` prunes to 448 of 512 experts;
`windowsxp811203/…-Abliterated-NVFP4` stores its 95 GiB PLE as a single malformed
`ple_embedding.shard_.weight` instead of 128 `ngram_embedding.shard_N.weight` and has never
been served by its own author.

## Why MTP is ON at k=3 (reversing this seat's original default)

This seat shipped with speculative decoding off, citing vLLM's recipe: on 4xH100 at TP=4
that recipe measured MTP **worse at every concurrency** (8-36% lower throughput, 32-173%
higher per-token latency, ~36% acceptance) and says do not default it on. Open #55357
reports episodic 0% acceptance with repetition collapse inside thinking blocks.

**Measured here, that inverted.** The campaign in `services/flash-next-mtp-bench/` found MTP
a win at every k and every concurrency tested on one Blackwell card (+29/41/27% at k=1,
+42/52/38% at k=2, +52/51/34% at k=3 across conc 1/4/8). k=3 is deployed because this is a
single-user fleet and conc=1 dominates.

On the current orcarouter checkpoint, measured 2026-09-14: **60.4% acceptance, mean
acceptance length 2.81** (per-position 80.6 / 60.8 / 40.8%), warm decode median **167.5
tok/s** at conc=1 (n=5, spread 12.2%).

⚠ **MTP costs KV.** The draft head adds ~5.08 GiB of weights and raises per-token KV cost
~16%; `FN_KV_CACHE_MEMORY` was cut 14 -> 10 GiB for it. At 14 GiB the engine OOMs at init
with MTP on. If it OOMs, drop to 8589934592.

⚠ The recipe's numbers are someone else's hardware, and so are ours to anyone else. Re-measure
on the seat, warm, with repeats — the first decode bench during the reorg read 39 tok/s and
that was a cold-boot + contention artifact, not a result.

## The upstream situation, as of 2026-09-13

- **#53896** — model support. **Merged 2026-08-31.** In v0.29.0.
- **#54371** — *UVA PLE-offload and Engram tensor parallelism*. **Merged
  2026-09-09T14:32Z.** This is the offload this seat uses. **Not in v0.29.0**,
  which was cut ~6 h earlier; present in `v0.29.1rc0` and in any nightly from
  2026-09-10 onward.
- **#53899** — the *older, worker-based* PLE offload. **Open and explicitly paused**
  in favour of #54371. Do not go back to it. Its whole bug family — the TP=1
  startup deadlock (#53960), the `pidfd_getfd` / `kernel.yama.ptrace_scope` gate,
  the shared-CUDA-event race under async scheduling, and silently one-step-stale
  PLE outputs under CUDA graphs — came from the separate worker process and the
  CUDA-IPC row transfer that the UVA path does not have.

Open issues worth knowing about on SM120, none of them blocking:

| Issue | What it does | Our exposure |
|---|---|---|
| **#54173** | CUBLAS internal error / illegal memory access in the GDN path **with prefix caching** | We enable prefix caching. `FN_PREFIX_CACHING=` is the one-line rollback. |
| **#54764** | PLE short-conv batched prefill pads every request to the batch-max query length | Why `--max-num-batched-tokens` is 8192, not 16384 |
| **#54919** | Long prefill starves active decode for 3–7 minutes | Why context starts at 128K |
| **#54521** | Greedy decoding non-deterministic from `persistent_topk` in prefill | Affects any A/B on this seat — establish a noise floor before comparing |
| **#54426** | fp8_e4m3 KV on the QSA path is an unmerged RFC | Why `--kv-cache-dtype` is **not** set to fp8 here |

## Context depth — PROBED 2026-09-14, clean to 258,517 tokens

262,144 is the configured ceiling and it has now been bisected with a **non-repeating**
prompt (unique random hex per probe, so prefix caching cannot short-circuit the prefill —
a repeated prompt hashes to cached blocks and never prefills deep).

| prompt tokens | 31,978 | 64,154 | 128,191 | 196,172 | 240,290 | **258,517** |
|---|---|---|---|---|---|---|
| result | ok | ok | ok | ok | ok | **ok** |

**The load-bearing evidence is the engine's own allocator log: zero OOM / CUBLAS /
illegal-memory / traceback entries across the whole run** — the same detector that caught the
dealignai near-miss (`OOM on device 0 ... 466 MiB wanted, 403 MiB free`) at 155K on the
previous checkpoint. It fired then; it is silent here. The run also happened **under real
concurrent operator load**, which makes it a stricter test than a solo probe, not a weaker one.

**Positive control passed.** A mis-sized first attempt built a ~265K-token prompt and got a
clean `400` naming the limit rather than killing the engine — so the probe could detect the
failure it was looking for. (Calibration for anyone re-running it: random hex words tokenize
at **7.9 tokens/word** on this tokenizer.)

**#54919 did not reproduce.** That issue reports long prefill starving decode for 3-7 minutes;
258K prefilled in **28.9 s** (~8,900 tok/s), scaling near-linearly from 32K.

⚠⚠ **DO NOT reuse the memory-headroom half of that probe — the gauge was blind.** It sampled
`nvidia-smi` before and after each request and reported an identical 95,460 MiB / 2,427 MiB
free on *every* row. Two reasons: `--kv-cache-memory` pins the pool and the engine log says it
**"skipped memory profiling"**, so GPU usage is constant regardless of depth; and the risk is a
*transient* activation spike **during** prefill, which before/after bracketing structurally
cannot see. Identical readings across a 8x range of depths are the tell. Real peak-activation
headroom needs in-process sampling during the prefill. The pass/fail result stands on the
allocator log, not on that column.

## Not done yet

- **Pin `--kv-cache-memory` in bytes** from the first boot's budget line, replacing
  the 0.90 ratio. Same discipline as `stacks/mog-sec` and `stacks/erp-seat`.
- **Gateway wiring is deliberately absent.** Pointing any LiteLLM alias at this
  seat — in particular displacing `gen` / `summarizer` / `classifier`, which is the
  long-term intent recorded in henge item 49 — changes what every existing caller
  receives and is the operator's call, not a deploy-time default.