feat(flash-next-seat): serve orcarouter weight-only NVFP4 on gen-large
Swaps gen-large from the dealignai ModelOpt W4A4 build to orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, which is weight-only on both axes (W8 float attn, W4 float experts, input_activations: null) and so avoids the 4-bit-activation long-context degradation mode. The checkpoint was previously recorded as unloadable on any mainline vLLM, requiring a from-source PLE-loader patch. That conclusion was wrong on cost. Qwen4ExpPLEEmbeddingMethod.from_quant_config checks ple_embedding_dtype as branch 1, before any quant-config type check, and its NotImplementedError for CompressedTensorsConfig is scoped to the PLE path only -- experts and dense load through the ordinary compressed-tensors paths. Verified by instantiating the real config and calling the selector both ways before doing any work. orcarouter ships a bf16 PLE, so the fix was to make the declaration true: convert the 51.2B-param table to FP8 and declare it. Its 128 PLE tensors sit in one shard file with nothing else in it. Global amax 0.0894, per-shard outlier ratio 1.66x, scale chosen exactly representable in bf16 so no scale-rounding error stacks on quantization; amax maps to 446.17/448, no clipping. Round-trip 2.655% RMS relative, 0.002% underflow, 0 saturation -- the same FP8-PLE treatment dealignai already shipped. MTP head (31 tensors) and vision tower carried through untouched. A second, independent blocker followed: orcarouter labels its 12 QSA layers qwen_sparse_attention, which vLLM rejects; it accepts full_attention and selects QSA via indexer_n_heads. Confirmed indexer_n_heads == 4 in both this and the dealignai checkpoint before renaming -- without that check the rename silently selects plain attention and serves a subtly wrong model that still passes a healthcheck. Measured on the live seat: healthy, coherent, KV 344,155 tokens @ 262,144 ctx, MTP k=3 at 60.4% acceptance / 2.81 mean acceptance length, warm decode median 167.5 tok/s at conc=1 (n=5, spread 12.2%). The reorg note's dealignai figure came from a different harness, so this is not claimed as a win over it; what it does establish is that weight-only experts did not cost decode speed. Still open: controlled quality A/B vs dealignai, and a deep-prefill probe at 262K. Rollback is two .env keys; dealignai remains on disk. Also corrects the README's MTP-is-off section, stale since k=3 was deployed, and adds a superseded-claims row to the quantization playbook.
This commit is contained in:
@@ -777,6 +777,7 @@ day if followed:
|
||||
| "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). |
|
||||
| "The Qwen3.8 MTP corruption is fixed by disabling prefix caching while keeping MTP; the gen seat runs APC-off" | this playbook §3.7 (now marked), earlier auto-memory | **SUPERSEDED 2026-08-17, and the staleness was only caught 2026-08-26.** APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use; reverted the same day. The real cause was the **AEON W4A4 quant being defective** (~15-20% degenerate generations), with MTP / prefix-caching / gateway merely AMPLIFYING it (§3.8). The gen seat runs **MTP ON and prefix-caching ON** on the in-house mixed NVFP4+FP8 build — verified against the live container 2026-08-26. ⚠ The lesson inside the lesson: a *passing multi-turn probe* was not sufficient evidence either. |
|
||||
| "transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 without causal-conv1d; it is precision-driven cancellation and fp32 resolves it" | `services/coldfusion-abliteration/README.md`, `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` | **SUPERSEDED 2026-08-20.** Precision was never the variable. The NaN came from **multi-GPU sharding** and **`expandable_segments`** (§3.9, §3.10); fp32 only made it rarer, which is worse than failing. On one GPU with a plain allocator, **bf16 is exactly deterministic through all 64 layers and generates coherent prose** — at 50 GB and 4.3× the throughput of the 111 GB fp32 it replaced. |
|
||||
| "A checkpoint with an FP8 PLE table but no `ple_embedding_dtype` declaration cannot be loaded; compressed-tensors qwen4_exp needs a vLLM source patch" | `stacks/flash-next-seat/README.md`, `persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md` | **SUPERSEDED 2026-09-14.** `from_quant_config` checks `ple_embedding_dtype` as **branch 1, before any quant-config type check**, and its `NotImplementedError` is scoped to the **PLE path only**. Declaring the FP8 PLE bypasses it on stock mainline — a **one-key config fix**, not a source build. Proven live: orcarouter (compressed-tensors) serves on gen-large after converting its bf16 PLE to FP8 + declaring it. ⚠ Declare only what is TRUE — declaring FP8 over a bf16 table is the gorbatjovy failure in reverse. |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ here is on the running box; regenerate the authoritative view with
|
||||
|---|---|
|
||||
| 0 | `mog-sec` (`sec` :8019, dflash k=7) · **`sentinel-r3`** (`sentinel-r3` :8025, dflash k=7 — NEW) |
|
||||
| 1 | **`meromero-charrp`** (`char-rp` :8016, MeroMero-v2-31B dense — restored) · `erp-seat` (`char-rp-fast` :8021) · reward · coder · embed · rerank |
|
||||
| 2 | `flash-next` (`gen-large` :8022) — **DOWN** (orca swap failed; dealignai stopped) |
|
||||
| 2 | `flash-next` (`gen-large` :8022) — **UP on orcarouter** (PLE converted bf16→FP8; MTP k=3, 60.4% accept) |
|
||||
| 3 | **RESERVED scratch** — empty, operator directive; benches/quants/probes only |
|
||||
|
||||
## What changed tonight
|
||||
@@ -38,48 +38,72 @@ here is on the running box; regenerate the authoritative view with
|
||||
concurrent-contention artifact; warm+isolated it was 121. Operator caught it by
|
||||
testing the running `sec` (102 tok/s) as reference.
|
||||
|
||||
## THE BLOCKER — gen-large "orca" swap (resume here)
|
||||
## ✅ THE BLOCKER — RESOLVED 2026-09-14 (no source build needed)
|
||||
|
||||
Operator wants gen-large on **`orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4`**
|
||||
(weight-only compressed-tensors: NVFP4 experts + FP8 attn + bf16 PLE) instead of the
|
||||
incumbent **dealignai** build (modelopt_fp4 **W4A4** — 4-bit activations, the
|
||||
long-context degradation mode). orcarouter is the trusted author (= `gen`), better on
|
||||
the activation axis. Downloaded to `/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4`
|
||||
(170 GB, complete, 18 shards + MTP head, integrity-verified).
|
||||
gen-large now serves **`orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4`** from
|
||||
`/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4-plefp8`. Healthy, coherent, MTP k=3.
|
||||
|
||||
**IT DOES NOT LOAD.** Root cause (read the loader source, definitive):
|
||||
`Qwen4ExpPLEEmbeddingMethod.from_quant_config` in vLLM — even the LATEST mainline
|
||||
nightly — has branches only for **ModelOpt** and **FP8** configs; for a
|
||||
CompressedTensorsConfig it raises `NotImplementedError` BEFORE ever consulting the
|
||||
ignore list. So:
|
||||
- dealignai (modelopt_fp4) loads fine; orcarouter (compressed-tensors) cannot, on any
|
||||
mainline vLLM.
|
||||
- orcarouter's config also does NOT list its `ngram_embedding` in `ignore` (would need
|
||||
adding) — but that's moot until the loader has a compressed-tensors branch.
|
||||
- orcarouter's OWN card prescribes **TP=4 + `--enable-expert-parallel`** and
|
||||
`vllm>=<qwen4exp-supporting release>` — a LITERAL UNFILLED version placeholder. They
|
||||
shipped a build whose runtime they couldn't pin.
|
||||
**The earlier diagnosis was right about the symptom and wrong about the cost.** It said the
|
||||
only auditable path was cherry-picking a PLE-loader branch onto a from-source Blackwell
|
||||
build. Reading the loader in the running nightly showed otherwise:
|
||||
|
||||
**Backport vetted → UNFIT.** `wtdcode/vllm-backport` (docker `lazymio/vllm-backport`,
|
||||
212★, Apache-2.0, DCO, active) resolved this exact error (issue #72) BUT: its purpose is
|
||||
OLD GPUs (images sm86/sm80/sm89 only — **no Blackwell/sm_120**), it's built around its
|
||||
OWN `wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16` quant not orcarouter's NVFP4, and Blackwell
|
||||
has native FP4/FP8 so the whole backport premise is moot for us. Using it = source-build
|
||||
for an untested arch to load a quant it never tested.
|
||||
```
|
||||
from_quant_config (vllm/models/qwen4_exp/nvidia/ngram_embedding.py:168)
|
||||
1. ple_embedding_dtype == "float8_e4m3fn" -> FP8 method <-- BEFORE any type check
|
||||
2. quant_config is None -> unquantized
|
||||
3. ModelOptMixedPrecisionConfig -> FP8 / unquantized
|
||||
4. ModelOptQuantConfigBase + excluded -> unquantized
|
||||
5. not isinstance(quant_config, Fp8Config)-> NotImplementedError <-- the blocker
|
||||
```
|
||||
|
||||
**Options (operator decision pending):**
|
||||
- (a) **Restore dealignai** — works today, identical architecture, ~90% of the quality
|
||||
(loses only the weight-only-vs-W4A4 activation edge). The pragmatic interim.
|
||||
- (b) Cherry-pick the ~10-line compressed-tensors PLE-loader branch onto a mainline
|
||||
source build for Blackwell (mainline already has Blackwell FP4 compute). More
|
||||
auditable than the backport, still a from-source build + maintenance.
|
||||
- (c) Request orcarouter's own **W4A4** build (they offer it) — would load on the current
|
||||
mainline image like dealignai.
|
||||
- (d) Wait for mainline vLLM to merge compressed-tensors qwen4_exp.
|
||||
Branch 1 is unconditional, and the `NotImplementedError` is **scoped to the PLE embedding
|
||||
path only** — experts and dense layers of a compressed-tensors qwen4_exp build load through
|
||||
vLLM's ordinary compressed-tensors paths. Falsified directly before doing any work: built
|
||||
the real `CompressedTensorsConfig` from orca's own config and called `from_quant_config`
|
||||
both ways — as-shipped raises, with the declaration returns `Qwen4ExpPLEFp8EmbeddingMethod`.
|
||||
|
||||
⚠ Operator said "don't reload dealignai unprompted, priority orca" — but orca needs
|
||||
runtime work, so gen-large is intentionally DOWN pending the decision. Both configs
|
||||
backed up on host: `compose.yaml.bak-20260914-orcarouter`, `.env.bak-20260914-orcarouter`.
|
||||
### What was actually done
|
||||
|
||||
1. **Converted the PLE table bf16 → FP8.** orca's 128 PLE tensors live in exactly ONE shard
|
||||
(`model-00002-of-00017.safetensors`, 95.4 GiB) with **no other tensors in it** — a clean
|
||||
split. Converted on GPU3 (reserved scratch) in 139 s into 8 `model-plefp8-*` files.
|
||||
- global amax **0.0894**; per-shard outlier ratio only **1.66x**, so one global scale fits
|
||||
- scale chosen **exactly representable in bf16** (2.002716e-04) so no scale-rounding error
|
||||
stacks on the quantization error; amax maps to **446.17 / 448** → no clipping
|
||||
- round-trip **2.655 % RMS relative**, 0.002 % underflow, **0 saturation**
|
||||
- `weight_scale` written **BF16 [1]**, matching gorbatjovy's published format (read from
|
||||
its actual safetensors header, not guessed)
|
||||
- MTP head (31 tensors, BF16) and the 333 vision tensors carried through untouched
|
||||
2. **Declared it**: `text_config.ple_embedding_dtype = "float8_e4m3fn"`.
|
||||
3. **Renamed `layer_types`**: orca labels its 12 QSA layers `qwen_sparse_attention`; vLLM
|
||||
accepts only `linear_attention` / `full_attention` and picks QSA via `indexer_n_heads`.
|
||||
⚠ **Verified `indexer_n_heads == 4` in BOTH orca and dealignai before renaming** — without
|
||||
it the rename silently selects PLAIN attention and serves a subtly wrong model that still
|
||||
looks healthy. Every QSA/indexer key matches dealignai exactly.
|
||||
4. `.env`: `FN_MODEL` → the converted dir, `FN_QUANT` → `compressed-tensors`.
|
||||
|
||||
### Measured on the live seat
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| warm decode, conc=1, greedy 300 tok, **n=5** | median **167.5 tok/s** (min 150.0, max 170.4, spread 12.2 %) |
|
||||
| MTP k=3 | acceptance **60.4 %**, mean acceptance length **2.81** (per-pos 80.6/60.8/40.8 %) |
|
||||
| KV | 344,155 tokens @ 262,144 ctx, 1.31x concurrency, pinned 10 GiB |
|
||||
| on card | ~75 GiB; PLE 47.7 GiB pinned host RAM |
|
||||
|
||||
⚠ **Do NOT read 167.5 as a win over dealignai.** dealignai's ~121 tok/s in this file came
|
||||
from a different harness/prompt; cross-harness comparison is invalid. What IS established is
|
||||
that weight-only experts (orca is W8 weight-only attn + W4 weight-only experts) did **not**
|
||||
cost decode speed, which was the standing risk of giving up FP4 tensor-core compute.
|
||||
|
||||
### Still open on this seat
|
||||
|
||||
- Quality A/B orca vs dealignai (the actual reason for the swap — the W4A4 long-context
|
||||
degradation axis). Needs a controlled harness + noise floor.
|
||||
- Deep-prefill probe at 262K against THIS checkpoint. Startup is not a depth test.
|
||||
- Rollback is two `.env` keys; `.env.bak-preorca-20260914-023408` on the host.
|
||||
- Disk: the convert **copied** ~75 GiB of unchanged shards because hardlinks hit `EXDEV`
|
||||
(separate bind mounts of the same fs). Harmless; reclaimable by relinking if /tank tightens.
|
||||
|
||||
## Other open items
|
||||
|
||||
|
||||
+39
-13
File diff suppressed because one or more lines are too long
@@ -23,9 +23,14 @@ FN_PORT=8022
|
||||
FN_CONTAINER_NAME=vllm-flash-next
|
||||
|
||||
# ── Model ───────────────────────────────────────────────────────────────────
|
||||
# dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b990578ef3031eccf9f28e675a289a09ee9
|
||||
FN_MODEL=/tank/aimodels/qwen38-flash-next-abliterated-nvfp4
|
||||
FN_QUANT=modelopt_fp4
|
||||
# orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, with its bf16 PLE converted to FP8
|
||||
# in-house (2026-09-14) so vLLM's from_quant_config branch 1 selects the FP8 PLE method.
|
||||
# ⚠ compressed-tensors, NOT modelopt_fp4 — this checkpoint is weight-only on BOTH axes
|
||||
# (W8 float attn / W4 float experts, input_activations: null), which is why it was chosen
|
||||
# over the dealignai W4A4 build. Rollback: the dealignai pair below, both still on disk.
|
||||
# FN_MODEL=/tank/aimodels/qwen38-flash-next-abliterated-nvfp4 FN_QUANT=modelopt_fp4
|
||||
FN_MODEL=/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4-plefp8
|
||||
FN_QUANT=compressed-tensors
|
||||
FN_SERVED_NAME=qwen3.8-flash-next-uncensored
|
||||
FN_SERVED_NAME_THINK=qwen3.8-flash-next-uncensored-thinking
|
||||
|
||||
|
||||
@@ -10,12 +10,12 @@ stream with async prefetch.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Checkpoint | `dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4` @ `be794b99…` (126.0 GiB) |
|
||||
| On the card | ~78 GiB of 95.6 GiB — routed experts NVFP4 W4A4, rest at source precision |
|
||||
| In host RAM | 47.7 GiB pinned, FP8 E4M3, 10 `model-plefp8-*` shards + per-table scalar scale |
|
||||
| Context | 131,072 to start (native ceiling 262,144) — see *Raising context* |
|
||||
| Speculative decoding | **none** — see *Why MTP is off* |
|
||||
| Gateway wiring | **none yet** — this seat is not in LiteLLM; `gen` is untouched |
|
||||
| Checkpoint | `orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4`, PLE converted bf16→FP8 in-house 2026-09-14 (123.2 GiB) |
|
||||
| On the card | ~75 GiB of 95.6 GiB — **weight-only on both axes**: W4 float experts, W8 float attn, `input_activations: null` |
|
||||
| In host RAM | 47.7 GiB pinned, FP8 E4M3, 8 `model-plefp8-*` shards + one global BF16 scale |
|
||||
| Context | **262,144** (full native) — KV 344,155 tokens, 1.31x concurrency |
|
||||
| Speculative decoding | **MTP k=3** — 60.4% acceptance, mean acceptance length 2.81 (measured here, n=5) |
|
||||
| Gateway wiring | **8 aliases** — gen, gen-reasoning, summarizer(-large), classifier, chat-judge, image-judge, qwen-image-bench |
|
||||
|
||||
## Deploy
|
||||
|
||||
@@ -39,59 +39,99 @@ Four ideas, and three of them shape the serving config:
|
||||
framing: capacity with almost no per-token compute.
|
||||
- **Gated residual / hyper-connections.** Four residual branches; excluded from
|
||||
quantization in this checkpoint.
|
||||
- **MTP head.** Present and preserved byte-identically. Deliberately unused.
|
||||
- **MTP head.** Present, preserved byte-identically, and **in use at k=3**.
|
||||
|
||||
## Why this checkpoint, and the trap that disqualifies most others
|
||||
## Why this checkpoint, and the two traps in front of it
|
||||
|
||||
vLLM selects the PLE table's weight format from **`text_config.ple_embedding_dtype`**,
|
||||
as the *first* branch of `Qwen4ExpPLEEmbeddingMethod.from_quant_config`. This
|
||||
checkpoint declares `"float8_e4m3fn"`.
|
||||
Chosen for the **activation axis**: orcarouter's build is weight-only on *both* halves —
|
||||
`config_groups` gives W8 float for attention/dense and W4 float for the experts, with
|
||||
`input_activations: null` on each. The displaced dealignai build is ModelOpt **W4A4**
|
||||
(4-bit activations), the long-context degradation mode. Same author as the `gen` seat.
|
||||
|
||||
A build that ships an FP8 PLE table **without** that declaration resolves instead
|
||||
through ModelOpt's `*.ple.*` exclude to the *unquantized* method, never registers
|
||||
the `weight_scale` parameter, and dies on load with `no module or parameter named
|
||||
'ngram_embedding.weight_scale'`. `gorbatjovy/qwen3.8-flash-next-abliterated-NVFP4-plefp8`
|
||||
is exactly this case. **Check that field before trying another build.**
|
||||
It did not load out of the box, and there were **two independent config-level blockers**.
|
||||
Both are recorded here because each looks like a capability gap and neither is one.
|
||||
|
||||
Chosen over better-liked alternatives because its provenance states protocols and
|
||||
repeat counts rather than adjectives. From its own `qualification-notes.md` and
|
||||
metrics files, kept in the model directory:
|
||||
### Trap 1 — the PLE loader (and the claim we had wrong)
|
||||
|
||||
- HarmBench, 240 genuinely-harmful behaviours, greedy: 100% compliance at reasoning
|
||||
off / low / xhigh.
|
||||
- MMLU 82.11% → 81.93% (−0.18 pp) on an identical harness, 2,280 questions.
|
||||
- GSM8K 97.27% (1283/1319), full set, single-shot, temp 0.6 — inside the stated
|
||||
BF16 reference band 97.12–97.50.
|
||||
- AIME26 pass@1 98.75% (237/240, **SEM 0.61 pp, 30 problems × 8 repeats**),
|
||||
majority@8 100%, `max_tokens` 130,000, 4.9M completion tokens, stop_rate 99.17%.
|
||||
- Byte-equality audit of unchanged tensors: 1,562 tensors / 118.4 GB compared, all
|
||||
passed, **including all 31 MTP tensors**.
|
||||
vLLM picks the PLE table's format in `Qwen4ExpPLEEmbeddingMethod.from_quant_config`:
|
||||
|
||||
⚠ Two honest gaps in that evidence. The routed experts are **NVFP4 W4A4**, and
|
||||
nobody — including the publisher — has measured this checkpoint at the full 262K
|
||||
context; AIME26's 130K-token generations are the deepest evidence that exists.
|
||||
Separately, `validate_checkpoint_report.json` in the repo describes the *earlier
|
||||
BF16-PLE revision* (204 shards / 173.6 GiB), not the published FP8-PLE one.
|
||||
```
|
||||
1. ple_embedding_dtype == "float8_e4m3fn" -> FP8 method <-- BEFORE any type check
|
||||
2. quant_config is None -> unquantized
|
||||
3. ModelOptMixedPrecisionConfig -> FP8 / unquantized
|
||||
4. ModelOptQuantConfigBase + excluded -> unquantized
|
||||
5. not isinstance(quant_config, Fp8Config)-> NotImplementedError
|
||||
```
|
||||
|
||||
Rejected alternatives, for the record:
|
||||
`orcarouter/…-Uncensored-NVFP4` is gated (access request pending nothing — not
|
||||
requested); `nvidia/…-NVFP4` is the cleanest ModelOpt MIXED_PRECISION build but is
|
||||
not abliterated; `lovedheart/…-Pruned-RTXPRO-6000` prunes to 448 of 512 experts.
|
||||
⚠ **This README previously said an FP8 PLE without the declaration is disqualifying, and
|
||||
that compressed-tensors needs a vLLM source patch. Both were wrong** (corrected 2026-09-14;
|
||||
see the quantization playbook's superseded-claims table). Branch 1 is **unconditional**, and
|
||||
the `NotImplementedError` is **scoped to the PLE path only** — experts and dense layers of a
|
||||
compressed-tensors build load through vLLM's ordinary compressed-tensors paths. So declaring
|
||||
an FP8 PLE bypasses the blocker on stock mainline.
|
||||
|
||||
## Why MTP is off
|
||||
orcarouter ships a **bf16** PLE, so the honest fix was to *make the declaration true*:
|
||||
convert the table to FP8, then declare it. Its 128 PLE tensors sit in exactly one shard file
|
||||
with nothing else in it, which makes that a clean, cheap rewrite.
|
||||
|
||||
Against our house graft-MTP habit, and on purpose.
|
||||
⚠ **Declare only what is true.** `gorbatjovy/...-NVFP4-plefp8` ships an FP8 table with no
|
||||
declaration and dies on `ngram_embedding.weight_scale`; declaring FP8 over a *bf16* table is
|
||||
that same failure in reverse. The declaration is a claim about the bytes, not a switch.
|
||||
|
||||
vLLM's own recipe for this model measured MTP on 4×H100 as **worse at every
|
||||
concurrency tested** — 8–36% lower request throughput, 32–173% higher per-token
|
||||
latency, driven by ~36% acceptance — and says do not enable it by default. Open
|
||||
issue **#55357** reports episodic 0% draft acceptance with repetition collapse
|
||||
inside thinking blocks. Open **#55496** reports ModelOpt `MIXED_PRECISION` failing
|
||||
to load FP8_BLOCK_SCALES MTP experts.
|
||||
### Trap 2 — `Invalid layer_type qwen_sparse_attention`
|
||||
|
||||
Turning it on is two lines in `compose.yaml` (documented in place). If you do,
|
||||
measure it **here**, with repeats, against this seat's own baseline — the numbers
|
||||
above are someone else's hardware.
|
||||
orcarouter labels its 12 QSA layers `qwen_sparse_attention`. vLLM accepts only
|
||||
`linear_attention` and `full_attention`, and selects QSA *within* `full_attention` when
|
||||
`indexer_n_heads` is present. The fix is renaming the 12 entries.
|
||||
|
||||
⚠⚠ **Check `indexer_n_heads` before renaming.** Without it the rename silently selects plain
|
||||
`Qwen3NextAttention` instead of `Qwen4ExpQSAAttention` — a subtly wrong model that loads,
|
||||
serves, and passes a healthcheck. Verified `indexer_n_heads == 4` in both this checkpoint and
|
||||
the dealignai one, along with every other indexer/QSA key, before touching it.
|
||||
|
||||
### The conversion, and what it cost
|
||||
|
||||
Global amax 0.0894 with a per-shard outlier ratio of only **1.66x**, so the single global
|
||||
scale this method uses is well-conditioned here. The scale is chosen **exactly representable
|
||||
in bf16** (2.002716e-04) so no scale-rounding error stacks on the quantization error; amax
|
||||
maps to 446.17 of 448, so nothing clips. Round-trip **2.655% RMS relative**, 0.002% underflow,
|
||||
zero saturation — and the same FP8-PLE treatment dealignai already shipped, so it is not a
|
||||
regression against the seat it replaced. `weight_scale` is written BF16 [1] to match the
|
||||
published format. MTP head (31 tensors) and the vision tower carry through untouched.
|
||||
|
||||
⚠ Still unmeasured: a controlled quality A/B against dealignai — which is the entire reason
|
||||
for the swap — and a deep-prefill probe at 262K on this checkpoint. Rollback is two `.env`
|
||||
keys; the dealignai checkpoint is still on disk.
|
||||
|
||||
Rejected alternatives, for the record: `nvidia/…-NVFP4` is the cleanest ModelOpt build but is
|
||||
not abliterated; `lovedheart/…-Pruned-RTXPRO-6000` prunes to 448 of 512 experts;
|
||||
`windowsxp811203/…-Abliterated-NVFP4` stores its 95 GiB PLE as a single malformed
|
||||
`ple_embedding.shard_.weight` instead of 128 `ngram_embedding.shard_N.weight` and has never
|
||||
been served by its own author.
|
||||
|
||||
## Why MTP is ON at k=3 (reversing this seat's original default)
|
||||
|
||||
This seat shipped with speculative decoding off, citing vLLM's recipe: on 4xH100 at TP=4
|
||||
that recipe measured MTP **worse at every concurrency** (8-36% lower throughput, 32-173%
|
||||
higher per-token latency, ~36% acceptance) and says do not default it on. Open #55357
|
||||
reports episodic 0% acceptance with repetition collapse inside thinking blocks.
|
||||
|
||||
**Measured here, that inverted.** The campaign in `services/flash-next-mtp-bench/` found MTP
|
||||
a win at every k and every concurrency tested on one Blackwell card (+29/41/27% at k=1,
|
||||
+42/52/38% at k=2, +52/51/34% at k=3 across conc 1/4/8). k=3 is deployed because this is a
|
||||
single-user fleet and conc=1 dominates.
|
||||
|
||||
On the current orcarouter checkpoint, measured 2026-09-14: **60.4% acceptance, mean
|
||||
acceptance length 2.81** (per-position 80.6 / 60.8 / 40.8%), warm decode median **167.5
|
||||
tok/s** at conc=1 (n=5, spread 12.2%).
|
||||
|
||||
⚠ **MTP costs KV.** The draft head adds ~5.08 GiB of weights and raises per-token KV cost
|
||||
~16%; `FN_KV_CACHE_MEMORY` was cut 14 -> 10 GiB for it. At 14 GiB the engine OOMs at init
|
||||
with MTP on. If it OOMs, drop to 8589934592.
|
||||
|
||||
⚠ The recipe's numbers are someone else's hardware, and so are ours to anyone else. Re-measure
|
||||
on the seat, warm, with repeats — the first decode bench during the reorg read 39 tok/s and
|
||||
that was a cold-boot + contention artifact, not a result.
|
||||
|
||||
## The upstream situation, as of 2026-09-13
|
||||
|
||||
|
||||
Reference in New Issue
Block a user