feat(flash-next-seat): serve orcarouter weight-only NVFP4 on gen-large
Swaps gen-large from the dealignai ModelOpt W4A4 build to orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, which is weight-only on both axes (W8 float attn, W4 float experts, input_activations: null) and so avoids the 4-bit-activation long-context degradation mode. The checkpoint was previously recorded as unloadable on any mainline vLLM, requiring a from-source PLE-loader patch. That conclusion was wrong on cost. Qwen4ExpPLEEmbeddingMethod.from_quant_config checks ple_embedding_dtype as branch 1, before any quant-config type check, and its NotImplementedError for CompressedTensorsConfig is scoped to the PLE path only -- experts and dense load through the ordinary compressed-tensors paths. Verified by instantiating the real config and calling the selector both ways before doing any work. orcarouter ships a bf16 PLE, so the fix was to make the declaration true: convert the 51.2B-param table to FP8 and declare it. Its 128 PLE tensors sit in one shard file with nothing else in it. Global amax 0.0894, per-shard outlier ratio 1.66x, scale chosen exactly representable in bf16 so no scale-rounding error stacks on quantization; amax maps to 446.17/448, no clipping. Round-trip 2.655% RMS relative, 0.002% underflow, 0 saturation -- the same FP8-PLE treatment dealignai already shipped. MTP head (31 tensors) and vision tower carried through untouched. A second, independent blocker followed: orcarouter labels its 12 QSA layers qwen_sparse_attention, which vLLM rejects; it accepts full_attention and selects QSA via indexer_n_heads. Confirmed indexer_n_heads == 4 in both this and the dealignai checkpoint before renaming -- without that check the rename silently selects plain attention and serves a subtly wrong model that still passes a healthcheck. Measured on the live seat: healthy, coherent, KV 344,155 tokens @ 262,144 ctx, MTP k=3 at 60.4% acceptance / 2.81 mean acceptance length, warm decode median 167.5 tok/s at conc=1 (n=5, spread 12.2%). The reorg note's dealignai figure came from a different harness, so this is not claimed as a win over it; what it does establish is that weight-only experts did not cost decode speed. Still open: controlled quality A/B vs dealignai, and a deep-prefill probe at 262K. Rollback is two .env keys; dealignai remains on disk. Also corrects the README's MTP-is-off section, stale since k=3 was deployed, and adds a superseded-claims row to the quantization playbook.
This commit is contained in:
@@ -23,9 +23,14 @@ FN_PORT=8022
|
||||
FN_CONTAINER_NAME=vllm-flash-next
|
||||
|
||||
# ── Model ───────────────────────────────────────────────────────────────────
|
||||
# dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4 @ be794b990578ef3031eccf9f28e675a289a09ee9
|
||||
FN_MODEL=/tank/aimodels/qwen38-flash-next-abliterated-nvfp4
|
||||
FN_QUANT=modelopt_fp4
|
||||
# orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, with its bf16 PLE converted to FP8
|
||||
# in-house (2026-09-14) so vLLM's from_quant_config branch 1 selects the FP8 PLE method.
|
||||
# ⚠ compressed-tensors, NOT modelopt_fp4 — this checkpoint is weight-only on BOTH axes
|
||||
# (W8 float attn / W4 float experts, input_activations: null), which is why it was chosen
|
||||
# over the dealignai W4A4 build. Rollback: the dealignai pair below, both still on disk.
|
||||
# FN_MODEL=/tank/aimodels/qwen38-flash-next-abliterated-nvfp4 FN_QUANT=modelopt_fp4
|
||||
FN_MODEL=/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4-plefp8
|
||||
FN_QUANT=compressed-tensors
|
||||
FN_SERVED_NAME=qwen3.8-flash-next-uncensored
|
||||
FN_SERVED_NAME_THINK=qwen3.8-flash-next-uncensored-thinking
|
||||
|
||||
|
||||
Reference in New Issue
Block a user