feat(flash-next-seat): serve orcarouter weight-only NVFP4 on gen-large

Swaps gen-large from the dealignai ModelOpt W4A4 build to
orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, which is weight-only on both
axes (W8 float attn, W4 float experts, input_activations: null) and so avoids
the 4-bit-activation long-context degradation mode.

The checkpoint was previously recorded as unloadable on any mainline vLLM,
requiring a from-source PLE-loader patch. That conclusion was wrong on cost.
Qwen4ExpPLEEmbeddingMethod.from_quant_config checks ple_embedding_dtype as
branch 1, before any quant-config type check, and its NotImplementedError for
CompressedTensorsConfig is scoped to the PLE path only -- experts and dense
load through the ordinary compressed-tensors paths. Verified by instantiating
the real config and calling the selector both ways before doing any work.

orcarouter ships a bf16 PLE, so the fix was to make the declaration true:
convert the 51.2B-param table to FP8 and declare it. Its 128 PLE tensors sit
in one shard file with nothing else in it. Global amax 0.0894, per-shard
outlier ratio 1.66x, scale chosen exactly representable in bf16 so no
scale-rounding error stacks on quantization; amax maps to 446.17/448, no
clipping. Round-trip 2.655% RMS relative, 0.002% underflow, 0 saturation --
the same FP8-PLE treatment dealignai already shipped. MTP head (31 tensors)
and vision tower carried through untouched.

A second, independent blocker followed: orcarouter labels its 12 QSA layers
qwen_sparse_attention, which vLLM rejects; it accepts full_attention and
selects QSA via indexer_n_heads. Confirmed indexer_n_heads == 4 in both this
and the dealignai checkpoint before renaming -- without that check the rename
silently selects plain attention and serves a subtly wrong model that still
passes a healthcheck.

Measured on the live seat: healthy, coherent, KV 344,155 tokens @ 262,144 ctx,
MTP k=3 at 60.4% acceptance / 2.81 mean acceptance length, warm decode median
167.5 tok/s at conc=1 (n=5, spread 12.2%). The reorg note's dealignai figure
came from a different harness, so this is not claimed as a win over it; what
it does establish is that weight-only experts did not cost decode speed.

Still open: controlled quality A/B vs dealignai, and a deep-prefill probe at
262K. Rollback is two .env keys; dealignai remains on disk.

Also corrects the README's MTP-is-off section, stale since k=3 was deployed,
and adds a superseded-claims row to the quantization playbook.
This commit is contained in:
vh
2026-09-14 02:48:42 -07:00
parent ed3f1ede3b
commit 4390be947d
5 changed files with 198 additions and 102 deletions
+1
View File
@@ -777,6 +777,7 @@ day if followed:
| "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). |
| "The Qwen3.8 MTP corruption is fixed by disabling prefix caching while keeping MTP; the gen seat runs APC-off" | this playbook §3.7 (now marked), earlier auto-memory | **SUPERSEDED 2026-08-17, and the staleness was only caught 2026-08-26.** APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use; reverted the same day. The real cause was the **AEON W4A4 quant being defective** (~15-20% degenerate generations), with MTP / prefix-caching / gateway merely AMPLIFYING it (§3.8). The gen seat runs **MTP ON and prefix-caching ON** on the in-house mixed NVFP4+FP8 build — verified against the live container 2026-08-26. ⚠ The lesson inside the lesson: a *passing multi-turn probe* was not sufficient evidence either. |
| "transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 without causal-conv1d; it is precision-driven cancellation and fp32 resolves it" | `services/coldfusion-abliteration/README.md`, `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` | **SUPERSEDED 2026-08-20.** Precision was never the variable. The NaN came from **multi-GPU sharding** and **`expandable_segments`** (§3.9, §3.10); fp32 only made it rarer, which is worse than failing. On one GPU with a plain allocator, **bf16 is exactly deterministic through all 64 layers and generates coherent prose** — at 50 GB and 4.3× the throughput of the 111 GB fp32 it replaced. |
| "A checkpoint with an FP8 PLE table but no `ple_embedding_dtype` declaration cannot be loaded; compressed-tensors qwen4_exp needs a vLLM source patch" | `stacks/flash-next-seat/README.md`, `persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md` | **SUPERSEDED 2026-09-14.** `from_quant_config` checks `ple_embedding_dtype` as **branch 1, before any quant-config type check**, and its `NotImplementedError` is scoped to the **PLE path only**. Declaring the FP8 PLE bypasses it on stock mainline — a **one-key config fix**, not a source build. Proven live: orcarouter (compressed-tensors) serves on gen-large after converting its bf16 PLE to FP8 + declaring it. ⚠ Declare only what is TRUE — declaring FP8 over a bf16 table is the gorbatjovy failure in reverse. |
---