Swaps gen-large from the dealignai ModelOpt W4A4 build to
orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, which is weight-only on both
axes (W8 float attn, W4 float experts, input_activations: null) and so avoids
the 4-bit-activation long-context degradation mode.
The checkpoint was previously recorded as unloadable on any mainline vLLM,
requiring a from-source PLE-loader patch. That conclusion was wrong on cost.
Qwen4ExpPLEEmbeddingMethod.from_quant_config checks ple_embedding_dtype as
branch 1, before any quant-config type check, and its NotImplementedError for
CompressedTensorsConfig is scoped to the PLE path only -- experts and dense
load through the ordinary compressed-tensors paths. Verified by instantiating
the real config and calling the selector both ways before doing any work.
orcarouter ships a bf16 PLE, so the fix was to make the declaration true:
convert the 51.2B-param table to FP8 and declare it. Its 128 PLE tensors sit
in one shard file with nothing else in it. Global amax 0.0894, per-shard
outlier ratio 1.66x, scale chosen exactly representable in bf16 so no
scale-rounding error stacks on quantization; amax maps to 446.17/448, no
clipping. Round-trip 2.655% RMS relative, 0.002% underflow, 0 saturation --
the same FP8-PLE treatment dealignai already shipped. MTP head (31 tensors)
and vision tower carried through untouched.
A second, independent blocker followed: orcarouter labels its 12 QSA layers
qwen_sparse_attention, which vLLM rejects; it accepts full_attention and
selects QSA via indexer_n_heads. Confirmed indexer_n_heads == 4 in both this
and the dealignai checkpoint before renaming -- without that check the rename
silently selects plain attention and serves a subtly wrong model that still
passes a healthcheck.
Measured on the live seat: healthy, coherent, KV 344,155 tokens @ 262,144 ctx,
MTP k=3 at 60.4% acceptance / 2.81 mean acceptance length, warm decode median
167.5 tok/s at conc=1 (n=5, spread 12.2%). The reorg note's dealignai figure
came from a different harness, so this is not claimed as a win over it; what
it does establish is that weight-only experts did not cost decode speed.
Still open: controlled quality A/B vs dealignai, and a deep-prefill probe at
262K. Rollback is two .env keys; dealignai remains on disk.
Also corrects the README's MTP-is-off section, stale since k=3 was deployed,
and adds a superseded-claims row to the quantization playbook.
vh
4390be947d