vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(flash-next-seat): serve orcarouter weight-only NVFP4 on gen-large

Browse Source 
Swaps gen-large from the dealignai ModelOpt W4A4 build to
orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, which is weight-only on both
axes (W8 float attn, W4 float experts, input_activations: null) and so avoids
the 4-bit-activation long-context degradation mode.

The checkpoint was previously recorded as unloadable on any mainline vLLM,
requiring a from-source PLE-loader patch. That conclusion was wrong on cost.
Qwen4ExpPLEEmbeddingMethod.from_quant_config checks ple_embedding_dtype as
branch 1, before any quant-config type check, and its NotImplementedError for
CompressedTensorsConfig is scoped to the PLE path only -- experts and dense
load through the ordinary compressed-tensors paths. Verified by instantiating
the real config and calling the selector both ways before doing any work.

orcarouter ships a bf16 PLE, so the fix was to make the declaration true:
convert the 51.2B-param table to FP8 and declare it. Its 128 PLE tensors sit
in one shard file with nothing else in it. Global amax 0.0894, per-shard
outlier ratio 1.66x, scale chosen exactly representable in bf16 so no
scale-rounding error stacks on quantization; amax maps to 446.17/448, no
clipping. Round-trip 2.655% RMS relative, 0.002% underflow, 0 saturation --
the same FP8-PLE treatment dealignai already shipped. MTP head (31 tensors)
and vision tower carried through untouched.

A second, independent blocker followed: orcarouter labels its 12 QSA layers
qwen_sparse_attention, which vLLM rejects; it accepts full_attention and
selects QSA via indexer_n_heads. Confirmed indexer_n_heads == 4 in both this
and the dealignai checkpoint before renaming -- without that check the rename
silently selects plain attention and serves a subtly wrong model that still
passes a healthcheck.

Measured on the live seat: healthy, coherent, KV 344,155 tokens @ 262,144 ctx,
MTP k=3 at 60.4% acceptance / 2.81 mean acceptance length, warm decode median
167.5 tok/s at conc=1 (n=5, spread 12.2%). The reorg note's dealignai figure
came from a different harness, so this is not claimed as a win over it; what
it does establish is that weight-only experts did not cost decode speed.

Still open: controlled quality A/B vs dealignai, and a deep-prefill probe at
262K. Rollback is two .env keys; dealignai remains on disk.

Also corrects the README's MTP-is-off section, stale since k=3 was deployed,
and adds a superseded-claims row to the quantization playbook.
This commit is contained in:
vh
2026-09-14 02:48:42 -07:00
parent ed3f1ede3b
commit 4390be947d
5 changed files with 198 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/flash-next-seat/.env.example
+8 -3
View File
@@ -23,9 +23,14 @@ FN_PORT=8022
FN_CONTAINER_NAME=vllm-flash-next
# ── Model ───────────────────────────────────────────────────────────────────
# dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b990578ef3031eccf9f28e675a289a09ee9
FN_MODEL=/tank/aimodels/qwen38-flash-next-abliterated-nvfp4
FN_QUANT=modelopt_fp4
# orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, with its bf16 PLE converted to FP8
# in-house (2026-09-14) so vLLM's from_quant_config branch 1 selects the FP8 PLE method.
# ⚠ compressed-tensors, NOT modelopt_fp4 — this checkpoint is weight-only on BOTH axes
# (W8 float attn / W4 float experts, input_activations: null), which is why it was chosen
# over the dealignai W4A4 build. Rollback: the dealignai pair below, both still on disk.
# FN_MODEL=/tank/aimodels/qwen38-flash-next-abliterated-nvfp4 FN_QUANT=modelopt_fp4
FN_MODEL=/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4-plefp8
FN_QUANT=compressed-tensors
FN_SERVED_NAME=qwen3.8-flash-next-uncensored
FN_SERVED_NAME_THINK=qwen3.8-flash-next-uncensored-thinking
stacks/flash-next-seat/README.md
+88 -48
View File
@@ -10,12 +10,12 @@ stream with async prefetch.
| | |
|---|---|
| Checkpoint | `dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4` @ `be794b99…` (126.0 GiB) |
| On the card | ~78 GiB of 95.6 GiB — routed experts NVFP4 W4A4, rest at source precision |
| In host RAM | 47.7 GiB pinned, FP8 E4M3, 10 `model-plefp8-*` shards + per-table scalar scale |
| Context | 131,072 to start (native ceiling 262,144) — see *Raising context* |
| Speculative decoding | **none** — see *Why MTP is off* |
| Gateway wiring | **none yet** — this seat is not in LiteLLM; `gen` is untouched |
| Checkpoint | `orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4`, PLE converted bf16→FP8 in-house 2026-09-14 (123.2 GiB) |
| On the card | ~75 GiB of 95.6 GiB — **weight-only on both axes**: W4 float experts, W8 float attn, `input_activations: null` |
| In host RAM | 47.7 GiB pinned, FP8 E4M3, 8 `model-plefp8-*` shards + one global BF16 scale |
| Context | **262,144** (full native) — KV 344,155 tokens, 1.31x concurrency |
| Speculative decoding | **MTP k=3** — 60.4% acceptance, mean acceptance length 2.81 (measured here, n=5) |
| Gateway wiring | **8 aliases** — gen, gen-reasoning, summarizer(-large), classifier, chat-judge, image-judge, qwen-image-bench |
## Deploy
@@ -39,59 +39,99 @@ Four ideas, and three of them shape the serving config:
framing: capacity with almost no per-token compute.
- **Gated residual / hyper-connections.** Four residual branches; excluded from
quantization in this checkpoint.
- **MTP head.** Present and preserved byte-identically. Deliberately unused.
- **MTP head.** Present, preserved byte-identically, and **in use at k=3**.
## Why this checkpoint, and the trap that disqualifies most others
## Why this checkpoint, and the two traps in front of it
vLLM selects the PLE table's weight format from **`text_config.ple_embedding_dtype`**,
as the *first* branch of `Qwen4ExpPLEEmbeddingMethod.from_quant_config`. This
checkpoint declares `"float8_e4m3fn"`.
Chosen for the **activation axis**: orcarouter's build is weight-only on *both* halves —
`config_groups` gives W8 float for attention/dense and W4 float for the experts, with
`input_activations: null` on each. The displaced dealignai build is ModelOpt **W4A4**
(4-bit activations), the long-context degradation mode. Same author as the `gen` seat.
A build that ships an FP8 PLE table **without** that declaration resolves instead
through ModelOpt's `*.ple.*` exclude to the *unquantized* method, never registers
the `weight_scale` parameter, and dies on load with `no module or parameter named
'ngram_embedding.weight_scale'`. `gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8`
is exactly this case. **Check that field before trying another build.**
It did not load out of the box, and there were **two independent config-level blockers**.
Both are recorded here because each looks like a capability gap and neither is one.
Chosen over better-liked alternatives because its provenance states protocols and
repeat counts rather than adjectives. From its own `qualification-notes.md` and
metrics files, kept in the model directory:
### Trap 1 — the PLE loader (and the claim we had wrong)
- HarmBench, 240 genuinely-harmful behaviours, greedy: 100% compliance at reasoning
off / low / xhigh.
- MMLU 82.11% → 81.93% (−0.18 pp) on an identical harness, 2,280 questions.
- GSM8K 97.27% (1283/1319), full set, single-shot, temp 0.6 — inside the stated
BF16 reference band 97.12–97.50.
- AIME26 pass@1 98.75% (237/240, **SEM 0.61 pp, 30 problems × 8 repeats**),
majority@8 100%, `max_tokens` 130,000, 4.9M completion tokens, stop_rate 99.17%.
- Byte-equality audit of unchanged tensors: 1,562 tensors / 118.4 GB compared, all
passed, **including all 31 MTP tensors**.
vLLM picks the PLE table's format in `Qwen4ExpPLEEmbeddingMethod.from_quant_config`:
⚠ Two honest gaps in that evidence. The routed experts are **NVFP4 W4A4**, and
nobody — including the publisher — has measured this checkpoint at the full 262K
context; AIME26's 130K-token generations are the deepest evidence that exists.
Separately, `validate_checkpoint_report.json` in the repo describes the *earlier
BF16-PLE revision* (204 shards / 173.6 GiB), not the published FP8-PLE one.
```
1. ple_embedding_dtype == "float8_e4m3fn" -> FP8 method <-- BEFORE any type check
2. quant_config is None -> unquantized
3. ModelOptMixedPrecisionConfig -> FP8 / unquantized
4. ModelOptQuantConfigBase + excluded -> unquantized
5. not isinstance(quant_config, Fp8Config)-> NotImplementedError
```
Rejected alternatives, for the record:
`orcarouter/…-Uncensored-NVFP4` is gated (access request pending nothing — not
requested); `nvidia/…-NVFP4` is the cleanest ModelOpt MIXED_PRECISION build but is
not abliterated; `lovedheart/…-Pruned-RTXPRO-6000` prunes to 448 of 512 experts.
⚠ **This README previously said an FP8 PLE without the declaration is disqualifying, and
that compressed-tensors needs a vLLM source patch. Both were wrong** (corrected 2026-09-14;
see the quantization playbook's superseded-claims table). Branch 1 is **unconditional**, and
the `NotImplementedError` is **scoped to the PLE path only** — experts and dense layers of a
compressed-tensors build load through vLLM's ordinary compressed-tensors paths. So declaring
an FP8 PLE bypasses the blocker on stock mainline.
## Why MTP is off
orcarouter ships a **bf16** PLE, so the honest fix was to *make the declaration true*:
convert the table to FP8, then declare it. Its 128 PLE tensors sit in exactly one shard file
with nothing else in it, which makes that a clean, cheap rewrite.
Against our house graft-MTP habit, and on purpose.
⚠ **Declare only what is true.** `gorbatjovy/...-NVFP4-plefp8` ships an FP8 table with no
declaration and dies on `ngram_embedding.weight_scale`; declaring FP8 over a *bf16* table is
that same failure in reverse. The declaration is a claim about the bytes, not a switch.
vLLM's own recipe for this model measured MTP on 4×H100 as **worse at every
concurrency tested** — 8–36% lower request throughput, 32–173% higher per-token
latency, driven by ~36% acceptance — and says do not enable it by default. Open
issue **#55357** reports episodic 0% draft acceptance with repetition collapse
inside thinking blocks. Open **#55496** reports ModelOpt `MIXED_PRECISION` failing
to load FP8_BLOCK_SCALES MTP experts.
### Trap 2 — `Invalid layer_type qwen_sparse_attention`
Turning it on is two lines in `compose.yaml` (documented in place). If you do,
measure it **here**, with repeats, against this seat's own baseline — the numbers
above are someone else's hardware.
orcarouter labels its 12 QSA layers `qwen_sparse_attention`. vLLM accepts only
`linear_attention` and `full_attention`, and selects QSA *within* `full_attention` when
`indexer_n_heads` is present. The fix is renaming the 12 entries.
⚠⚠ **Check `indexer_n_heads` before renaming.** Without it the rename silently selects plain
`Qwen3NextAttention` instead of `Qwen4ExpQSAAttention` — a subtly wrong model that loads,
serves, and passes a healthcheck. Verified `indexer_n_heads == 4` in both this checkpoint and
the dealignai one, along with every other indexer/QSA key, before touching it.
### The conversion, and what it cost
Global amax 0.0894 with a per-shard outlier ratio of only **1.66x**, so the single global
scale this method uses is well-conditioned here. The scale is chosen **exactly representable
in bf16** (2.002716e-04) so no scale-rounding error stacks on the quantization error; amax
maps to 446.17 of 448, so nothing clips. Round-trip **2.655% RMS relative**, 0.002% underflow,
zero saturation — and the same FP8-PLE treatment dealignai already shipped, so it is not a
regression against the seat it replaced. `weight_scale` is written BF16 [1] to match the
published format. MTP head (31 tensors) and the vision tower carry through untouched.
⚠ Still unmeasured: a controlled quality A/B against dealignai — which is the entire reason
for the swap — and a deep-prefill probe at 262K on this checkpoint. Rollback is two `.env`
keys; the dealignai checkpoint is still on disk.
Rejected alternatives, for the record: `nvidia/…-NVFP4` is the cleanest ModelOpt build but is
not abliterated; `lovedheart/…-Pruned-RTXPRO-6000` prunes to 448 of 512 experts;
`windowsxp811203/…-Abliterated-NVFP4` stores its 95 GiB PLE as a single malformed
`ple_embedding.shard_.weight` instead of 128 `ngram_embedding.shard_N.weight` and has never
been served by its own author.
## Why MTP is ON at k=3 (reversing this seat's original default)
This seat shipped with speculative decoding off, citing vLLM's recipe: on 4xH100 at TP=4
that recipe measured MTP **worse at every concurrency** (8-36% lower throughput, 32-173%
higher per-token latency, ~36% acceptance) and says do not default it on. Open #55357
reports episodic 0% acceptance with repetition collapse inside thinking blocks.
**Measured here, that inverted.** The campaign in `services/flash-next-mtp-bench/` found MTP
a win at every k and every concurrency tested on one Blackwell card (+29/41/27% at k=1,
+42/52/38% at k=2, +52/51/34% at k=3 across conc 1/4/8). k=3 is deployed because this is a
single-user fleet and conc=1 dominates.
On the current orcarouter checkpoint, measured 2026-09-14: **60.4% acceptance, mean
acceptance length 2.81** (per-position 80.6 / 60.8 / 40.8%), warm decode median **167.5
tok/s** at conc=1 (n=5, spread 12.2%).
⚠ **MTP costs KV.** The draft head adds ~5.08 GiB of weights and raises per-token KV cost
~16%; `FN_KV_CACHE_MEMORY` was cut 14 -> 10 GiB for it. At 14 GiB the engine OOMs at init
with MTP on. If it OOMs, drop to 8589934592.
⚠ The recipe's numbers are someone else's hardware, and so are ours to anyone else. Re-measure
on the seat, warm, with repeats — the first decode bench during the reorg read 39 tok/s and
that was a cold-boot + contention artifact, not a result.
## The upstream situation, as of 2026-09-13