vh
4390be947d
Swaps gen-large from the dealignai ModelOpt W4A4 build to orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, which is weight-only on both axes (W8 float attn, W4 float experts, input_activations: null) and so avoids the 4-bit-activation long-context degradation mode. The checkpoint was previously recorded as unloadable on any mainline vLLM, requiring a from-source PLE-loader patch. That conclusion was wrong on cost. Qwen4ExpPLEEmbeddingMethod.from_quant_config checks ple_embedding_dtype as branch 1, before any quant-config type check, and its NotImplementedError for CompressedTensorsConfig is scoped to the PLE path only -- experts and dense load through the ordinary compressed-tensors paths. Verified by instantiating the real config and calling the selector both ways before doing any work. orcarouter ships a bf16 PLE, so the fix was to make the declaration true: convert the 51.2B-param table to FP8 and declare it. Its 128 PLE tensors sit in one shard file with nothing else in it. Global amax 0.0894, per-shard outlier ratio 1.66x, scale chosen exactly representable in bf16 so no scale-rounding error stacks on quantization; amax maps to 446.17/448, no clipping. Round-trip 2.655% RMS relative, 0.002% underflow, 0 saturation -- the same FP8-PLE treatment dealignai already shipped. MTP head (31 tensors) and vision tower carried through untouched. A second, independent blocker followed: orcarouter labels its 12 QSA layers qwen_sparse_attention, which vLLM rejects; it accepts full_attention and selects QSA via indexer_n_heads. Confirmed indexer_n_heads == 4 in both this and the dealignai checkpoint before renaming -- without that check the rename silently selects plain attention and serves a subtly wrong model that still passes a healthcheck. Measured on the live seat: healthy, coherent, KV 344,155 tokens @ 262,144 ctx, MTP k=3 at 60.4% acceptance / 2.81 mean acceptance length, warm decode median 167.5 tok/s at conc=1 (n=5, spread 12.2%). The reorg note's dealignai figure came from a different harness, so this is not claimed as a win over it; what it does establish is that weight-only experts did not cost decode speed. Still open: controlled quality A/B vs dealignai, and a deep-prefill probe at 262K. Rollback is two .env keys; dealignai remains on disk. Also corrects the README's MTP-is-off section, stale since k=3 was deployed, and adds a superseded-claims row to the quantization playbook.
180 lines
10 KiB
Markdown
180 lines
10 KiB
Markdown
# flash-next-seat — Qwen3.8-Flash-Next (abliterated), fv-ml1 GPU 2, `:8022`
|
||
|
||
The first seat on the fleet whose weights do not fit its card and run anyway.
|
||
|
||
`Qwen3.8-Flash-Next` is 176B total — a 125B main model plus a **51B n-gram (PLE)
|
||
lookup table** — activating ~6B parameters per token. The n-gram table is a pure
|
||
embedding lookup with almost no compute per token, so it lives in **pinned host
|
||
RAM** and the GPU reads the rows it needs directly over **CUDA UVA** on a dedicated
|
||
stream with async prefetch.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Checkpoint | `orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4`, PLE converted bf16→FP8 in-house 2026-09-14 (123.2 GiB) |
|
||
| On the card | ~75 GiB of 95.6 GiB — **weight-only on both axes**: W4 float experts, W8 float attn, `input_activations: null` |
|
||
| In host RAM | 47.7 GiB pinned, FP8 E4M3, 8 `model-plefp8-*` shards + one global BF16 scale |
|
||
| Context | **262,144** (full native) — KV 344,155 tokens, 1.31x concurrency |
|
||
| Speculative decoding | **MTP k=3** — 60.4% acceptance, mean acceptance length 2.81 (measured here, n=5) |
|
||
| Gateway wiring | **8 aliases** — gen, gen-reasoning, summarizer(-large), classifier, chat-judge, image-judge, qwen-image-bench |
|
||
|
||
## Deploy
|
||
|
||
```bash
|
||
scripts/deploy-stack.sh fv-ml1 flash-next-seat # diffs vs live, prompts y/N
|
||
# then on the host, first boot only:
|
||
ssh infra-ops@10.251.50.54 'cd /opt/docker/compose/flash-next-seat && docker compose config >/dev/null && docker compose up -d'
|
||
```
|
||
|
||
The `.env` lives on the host and is never committed. Copy `.env.example`, set
|
||
`API_KEY`, and read the FIRST-BOOT annotations before changing anything else.
|
||
|
||
## Architecture, briefly
|
||
|
||
Four ideas, and three of them shape the serving config:
|
||
|
||
- **GDN + QSA.** 36 of 48 layers use Gated DeltaNet (linear attention) to compress
|
||
history; every fourth layer uses Qwen Sparse Attention for long-range retrieval.
|
||
This is why KV is cheap at depth and why `--mamba-cache-dtype float32` matters.
|
||
- **N-gram embedding.** The 51B lookup table that this seat offloads. Qwen's own
|
||
framing: capacity with almost no per-token compute.
|
||
- **Gated residual / hyper-connections.** Four residual branches; excluded from
|
||
quantization in this checkpoint.
|
||
- **MTP head.** Present, preserved byte-identically, and **in use at k=3**.
|
||
|
||
## Why this checkpoint, and the two traps in front of it
|
||
|
||
Chosen for the **activation axis**: orcarouter's build is weight-only on *both* halves —
|
||
`config_groups` gives W8 float for attention/dense and W4 float for the experts, with
|
||
`input_activations: null` on each. The displaced dealignai build is ModelOpt **W4A4**
|
||
(4-bit activations), the long-context degradation mode. Same author as the `gen` seat.
|
||
|
||
It did not load out of the box, and there were **two independent config-level blockers**.
|
||
Both are recorded here because each looks like a capability gap and neither is one.
|
||
|
||
### Trap 1 — the PLE loader (and the claim we had wrong)
|
||
|
||
vLLM picks the PLE table's format in `Qwen4ExpPLEEmbeddingMethod.from_quant_config`:
|
||
|
||
```
|
||
1. ple_embedding_dtype == "float8_e4m3fn" -> FP8 method <-- BEFORE any type check
|
||
2. quant_config is None -> unquantized
|
||
3. ModelOptMixedPrecisionConfig -> FP8 / unquantized
|
||
4. ModelOptQuantConfigBase + excluded -> unquantized
|
||
5. not isinstance(quant_config, Fp8Config)-> NotImplementedError
|
||
```
|
||
|
||
⚠ **This README previously said an FP8 PLE without the declaration is disqualifying, and
|
||
that compressed-tensors needs a vLLM source patch. Both were wrong** (corrected 2026-09-14;
|
||
see the quantization playbook's superseded-claims table). Branch 1 is **unconditional**, and
|
||
the `NotImplementedError` is **scoped to the PLE path only** — experts and dense layers of a
|
||
compressed-tensors build load through vLLM's ordinary compressed-tensors paths. So declaring
|
||
an FP8 PLE bypasses the blocker on stock mainline.
|
||
|
||
orcarouter ships a **bf16** PLE, so the honest fix was to *make the declaration true*:
|
||
convert the table to FP8, then declare it. Its 128 PLE tensors sit in exactly one shard file
|
||
with nothing else in it, which makes that a clean, cheap rewrite.
|
||
|
||
⚠ **Declare only what is true.** `gorbatjovy/...-NVFP4-plefp8` ships an FP8 table with no
|
||
declaration and dies on `ngram_embedding.weight_scale`; declaring FP8 over a *bf16* table is
|
||
that same failure in reverse. The declaration is a claim about the bytes, not a switch.
|
||
|
||
### Trap 2 — `Invalid layer_type qwen_sparse_attention`
|
||
|
||
orcarouter labels its 12 QSA layers `qwen_sparse_attention`. vLLM accepts only
|
||
`linear_attention` and `full_attention`, and selects QSA *within* `full_attention` when
|
||
`indexer_n_heads` is present. The fix is renaming the 12 entries.
|
||
|
||
⚠⚠ **Check `indexer_n_heads` before renaming.** Without it the rename silently selects plain
|
||
`Qwen3NextAttention` instead of `Qwen4ExpQSAAttention` — a subtly wrong model that loads,
|
||
serves, and passes a healthcheck. Verified `indexer_n_heads == 4` in both this checkpoint and
|
||
the dealignai one, along with every other indexer/QSA key, before touching it.
|
||
|
||
### The conversion, and what it cost
|
||
|
||
Global amax 0.0894 with a per-shard outlier ratio of only **1.66x**, so the single global
|
||
scale this method uses is well-conditioned here. The scale is chosen **exactly representable
|
||
in bf16** (2.002716e-04) so no scale-rounding error stacks on the quantization error; amax
|
||
maps to 446.17 of 448, so nothing clips. Round-trip **2.655% RMS relative**, 0.002% underflow,
|
||
zero saturation — and the same FP8-PLE treatment dealignai already shipped, so it is not a
|
||
regression against the seat it replaced. `weight_scale` is written BF16 [1] to match the
|
||
published format. MTP head (31 tensors) and the vision tower carry through untouched.
|
||
|
||
⚠ Still unmeasured: a controlled quality A/B against dealignai — which is the entire reason
|
||
for the swap — and a deep-prefill probe at 262K on this checkpoint. Rollback is two `.env`
|
||
keys; the dealignai checkpoint is still on disk.
|
||
|
||
Rejected alternatives, for the record: `nvidia/…-NVFP4` is the cleanest ModelOpt build but is
|
||
not abliterated; `lovedheart/…-Pruned-RTXPRO-6000` prunes to 448 of 512 experts;
|
||
`windowsxp811203/…-Abliterated-NVFP4` stores its 95 GiB PLE as a single malformed
|
||
`ple_embedding.shard_.weight` instead of 128 `ngram_embedding.shard_N.weight` and has never
|
||
been served by its own author.
|
||
|
||
## Why MTP is ON at k=3 (reversing this seat's original default)
|
||
|
||
This seat shipped with speculative decoding off, citing vLLM's recipe: on 4xH100 at TP=4
|
||
that recipe measured MTP **worse at every concurrency** (8-36% lower throughput, 32-173%
|
||
higher per-token latency, ~36% acceptance) and says do not default it on. Open #55357
|
||
reports episodic 0% acceptance with repetition collapse inside thinking blocks.
|
||
|
||
**Measured here, that inverted.** The campaign in `services/flash-next-mtp-bench/` found MTP
|
||
a win at every k and every concurrency tested on one Blackwell card (+29/41/27% at k=1,
|
||
+42/52/38% at k=2, +52/51/34% at k=3 across conc 1/4/8). k=3 is deployed because this is a
|
||
single-user fleet and conc=1 dominates.
|
||
|
||
On the current orcarouter checkpoint, measured 2026-09-14: **60.4% acceptance, mean
|
||
acceptance length 2.81** (per-position 80.6 / 60.8 / 40.8%), warm decode median **167.5
|
||
tok/s** at conc=1 (n=5, spread 12.2%).
|
||
|
||
⚠ **MTP costs KV.** The draft head adds ~5.08 GiB of weights and raises per-token KV cost
|
||
~16%; `FN_KV_CACHE_MEMORY` was cut 14 -> 10 GiB for it. At 14 GiB the engine OOMs at init
|
||
with MTP on. If it OOMs, drop to 8589934592.
|
||
|
||
⚠ The recipe's numbers are someone else's hardware, and so are ours to anyone else. Re-measure
|
||
on the seat, warm, with repeats — the first decode bench during the reorg read 39 tok/s and
|
||
that was a cold-boot + contention artifact, not a result.
|
||
|
||
## The upstream situation, as of 2026-09-13
|
||
|
||
- **#53896** — model support. **Merged 2026-08-31.** In v0.29.0.
|
||
- **#54371** — *UVA PLE-offload and Engram tensor parallelism*. **Merged
|
||
2026-09-09T14:32Z.** This is the offload this seat uses. **Not in v0.29.0**,
|
||
which was cut ~6 h earlier; present in `v0.29.1rc0` and in any nightly from
|
||
2026-09-10 onward.
|
||
- **#53899** — the *older, worker-based* PLE offload. **Open and explicitly paused**
|
||
in favour of #54371. Do not go back to it. Its whole bug family — the TP=1
|
||
startup deadlock (#53960), the `pidfd_getfd` / `kernel.yama.ptrace_scope` gate,
|
||
the shared-CUDA-event race under async scheduling, and silently one-step-stale
|
||
PLE outputs under CUDA graphs — came from the separate worker process and the
|
||
CUDA-IPC row transfer that the UVA path does not have.
|
||
|
||
Open issues worth knowing about on SM120, none of them blocking:
|
||
|
||
| Issue | What it does | Our exposure |
|
||
|---|---|---|
|
||
| **#54173** | CUBLAS internal error / illegal memory access in the GDN path **with prefix caching** | We enable prefix caching. `FN_PREFIX_CACHING=` is the one-line rollback. |
|
||
| **#54764** | PLE short-conv batched prefill pads every request to the batch-max query length | Why `--max-num-batched-tokens` is 8192, not 16384 |
|
||
| **#54919** | Long prefill starves active decode for 3–7 minutes | Why context starts at 128K |
|
||
| **#54521** | Greedy decoding non-deterministic from `persistent_topk` in prefill | Affects any A/B on this seat — establish a noise floor before comparing |
|
||
| **#54426** | fp8_e4m3 KV on the QSA path is an unmerged RFC | Why `--kv-cache-dtype` is **not** set to fp8 here |
|
||
|
||
## Raising context
|
||
|
||
128K is a starting value, not a measured one. Before raising it, bisect with a
|
||
**non-repeating** prompt — a repeated one hashes to cached blocks and never
|
||
prefills deep, so it proves nothing. The `stacks/mog-sec` README records this the
|
||
hard way: three successive context cuts all sized the *KV pool* while the crashes
|
||
were governed by *processing depth*, which is a different number.
|
||
|
||
The point of a ceiling is the refusal. Below it the seat serves; above it vLLM
|
||
returns a clean 400 naming the limit, instead of the engine dying and taking every
|
||
in-flight request with it.
|
||
|
||
## Not done yet
|
||
|
||
- **Pin `--kv-cache-memory` in bytes** from the first boot's budget line, replacing
|
||
the 0.90 ratio. Same discipline as `stacks/mog-sec` and `stacks/erp-seat`.
|
||
- **Gateway wiring is deliberately absent.** Pointing any LiteLLM alias at this
|
||
seat — in particular displacing `gen` / `summarizer` / `classifier`, which is the
|
||
long-term intent recorded in henge item 49 — changes what every existing caller
|
||
receives and is the operator's call, not a deploy-time default.
|