# flash-next-seat — Qwen3.8-Flash-Next (abliterated), fv-ml1 GPU 2, `:8022`

The first seat on the fleet whose weights do not fit its card and run anyway.

`Qwen3.8-Flash-Next` is 176B total — a 125B main model plus a **51B n-gram (PLE)
lookup table** — activating ~6B parameters per token. The n-gram table is a pure
embedding lookup with almost no compute per token, so it lives in **pinned host
RAM** and the GPU reads the rows it needs directly over **CUDA UVA** on a dedicated
stream with async prefetch.

| | |
|---|---|
| Checkpoint | `dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4` @ `be794b99…` (126.0 GiB) |
| On the card | ~78 GiB of 95.6 GiB — routed experts NVFP4 W4A4, rest at source precision |
| In host RAM | 47.7 GiB pinned, FP8 E4M3, 10 `model-plefp8-*` shards + per-table scalar scale |
| Context | 131,072 to start (native ceiling 262,144) — see *Raising context* |
| Speculative decoding | **none** — see *Why MTP is off* |
| Gateway wiring | **none yet** — this seat is not in LiteLLM; `gen` is untouched |

## Deploy

```bash
scripts/deploy-stack.sh fv-ml1 flash-next-seat     # diffs vs live, prompts y/N
# then on the host, first boot only:
ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/flash-next-seat && docker compose config >/dev/null && docker compose up -d'
```

The `.env` lives on the host and is never committed. Copy `.env.example`, set
`API_KEY`, and read the FIRST-BOOT annotations before changing anything else.

## Architecture, briefly

Four ideas, and three of them shape the serving config:

- **GDN + QSA.** 36 of 48 layers use Gated DeltaNet (linear attention) to compress
  history; every fourth layer uses Qwen Sparse Attention for long-range retrieval.
  This is why KV is cheap at depth and why `--mamba-cache-dtype float32` matters.
- **N-gram embedding.** The 51B lookup table that this seat offloads. Qwen's own
  framing: capacity with almost no per-token compute.
- **Gated residual / hyper-connections.** Four residual branches; excluded from
  quantization in this checkpoint.
- **MTP head.** Present and preserved byte-identically. Deliberately unused.

## Why this checkpoint, and the trap that disqualifies most others

vLLM selects the PLE table's weight format from **`text_config.ple_embedding_dtype`**,
as the *first* branch of `Qwen4ExpPLEEmbeddingMethod.from_quant_config`. This
checkpoint declares `"float8_e4m3fn"`.

A build that ships an FP8 PLE table **without** that declaration resolves instead
through ModelOpt's `*.ple.*` exclude to the *unquantized* method, never registers
the `weight_scale` parameter, and dies on load with `no module or parameter named
'ngram_embedding.weight_scale'`. `gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8`
is exactly this case. **Check that field before trying another build.**

Chosen over better-liked alternatives because its provenance states protocols and
repeat counts rather than adjectives. From its own `qualification-notes.md` and
metrics files, kept in the model directory:

- HarmBench, 240 genuinely-harmful behaviours, greedy: 100% compliance at reasoning
  off / low / xhigh.
- MMLU 82.11% → 81.93% (−0.18 pp) on an identical harness, 2,280 questions.
- GSM8K 97.27% (1283/1319), full set, single-shot, temp 0.6 — inside the stated
  BF16 reference band 97.12–97.50.
- AIME26 pass@1 98.75% (237/240, **SEM 0.61 pp, 30 problems × 8 repeats**),
  majority@8 100%, `max_tokens` 130,000, 4.9M completion tokens, stop_rate 99.17%.
- Byte-equality audit of unchanged tensors: 1,562 tensors / 118.4 GB compared, all
  passed, **including all 31 MTP tensors**.

⚠ Two honest gaps in that evidence. The routed experts are **NVFP4 W4A4**, and
nobody — including the publisher — has measured this checkpoint at the full 262K
context; AIME26's 130K-token generations are the deepest evidence that exists.
Separately, `validate_checkpoint_report.json` in the repo describes the *earlier
BF16-PLE revision* (204 shards / 173.6 GiB), not the published FP8-PLE one.

Rejected alternatives, for the record:
`orcarouter/…-Uncensored-NVFP4` is gated (access request pending nothing — not
requested); `nvidia/…-NVFP4` is the cleanest ModelOpt MIXED_PRECISION build but is
not abliterated; `lovedheart/…-Pruned-RTXPRO-6000` prunes to 448 of 512 experts.

## Why MTP is off

Against our house graft-MTP habit, and on purpose.

vLLM's own recipe for this model measured MTP on 4×H100 as **worse at every
concurrency tested** — 8–36% lower request throughput, 32–173% higher per-token
latency, driven by ~36% acceptance — and says do not enable it by default. Open
issue **#55357** reports episodic 0% draft acceptance with repetition collapse
inside thinking blocks. Open **#55496** reports ModelOpt `MIXED_PRECISION` failing
to load FP8_BLOCK_SCALES MTP experts.

Turning it on is two lines in `compose.yaml` (documented in place). If you do,
measure it **here**, with repeats, against this seat's own baseline — the numbers
above are someone else's hardware.

## The upstream situation, as of 2026-09-13

- **#53896** — model support. **Merged 2026-08-31.** In v0.29.0.
- **#54371** — *UVA PLE-offload and Engram tensor parallelism*. **Merged
  2026-09-09T14:32Z.** This is the offload this seat uses. **Not in v0.29.0**,
  which was cut ~6 h earlier; present in `v0.29.1rc0` and in any nightly from
  2026-09-10 onward.
- **#53899** — the *older, worker-based* PLE offload. **Open and explicitly paused**
  in favour of #54371. Do not go back to it. Its whole bug family — the TP=1
  startup deadlock (#53960), the `pidfd_getfd` / `kernel.yama.ptrace_scope` gate,
  the shared-CUDA-event race under async scheduling, and silently one-step-stale
  PLE outputs under CUDA graphs — came from the separate worker process and the
  CUDA-IPC row transfer that the UVA path does not have.

Open issues worth knowing about on SM120, none of them blocking:

| Issue | What it does | Our exposure |
|---|---|---|
| **#54173** | CUBLAS internal error / illegal memory access in the GDN path **with prefix caching** | We enable prefix caching. `FN_PREFIX_CACHING=` is the one-line rollback. |
| **#54764** | PLE short-conv batched prefill pads every request to the batch-max query length | Why `--max-num-batched-tokens` is 8192, not 16384 |
| **#54919** | Long prefill starves active decode for 3–7 minutes | Why context starts at 128K |
| **#54521** | Greedy decoding non-deterministic from `persistent_topk` in prefill | Affects any A/B on this seat — establish a noise floor before comparing |
| **#54426** | fp8_e4m3 KV on the QSA path is an unmerged RFC | Why `--kv-cache-dtype` is **not** set to fp8 here |

## Raising context

128K is a starting value, not a measured one. Before raising it, bisect with a
**non-repeating** prompt — a repeated one hashes to cached blocks and never
prefills deep, so it proves nothing. The `stacks/mog-sec` README records this the
hard way: three successive context cuts all sized the *KV pool* while the crashes
were governed by *processing depth*, which is a different number.

The point of a ceiling is the refusal. Below it the seat serves; above it vLLM
returns a clean 400 naming the limit, instead of the engine dying and taking every
in-flight request with it.

## Not done yet

- **Pin `--kv-cache-memory` in bytes** from the first boot's budget line, replacing
  the 0.90 ratio. Same discipline as `stacks/mog-sec` and `stacks/erp-seat`.
- **Gateway wiring is deliberately absent.** Pointing any LiteLLM alias at this
  seat — in particular displacing `gen` / `summarizer` / `classifier`, which is the
  long-term intent recorded in henge item 49 — changes what every existing caller
  receives and is the operator's call, not a deploy-time default.