Swaps gen-large from the dealignai ModelOpt W4A4 build to
orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4, which is weight-only on both
axes (W8 float attn, W4 float experts, input_activations: null) and so avoids
the 4-bit-activation long-context degradation mode.
The checkpoint was previously recorded as unloadable on any mainline vLLM,
requiring a from-source PLE-loader patch. That conclusion was wrong on cost.
Qwen4ExpPLEEmbeddingMethod.from_quant_config checks ple_embedding_dtype as
branch 1, before any quant-config type check, and its NotImplementedError for
CompressedTensorsConfig is scoped to the PLE path only -- experts and dense
load through the ordinary compressed-tensors paths. Verified by instantiating
the real config and calling the selector both ways before doing any work.
orcarouter ships a bf16 PLE, so the fix was to make the declaration true:
convert the 51.2B-param table to FP8 and declare it. Its 128 PLE tensors sit
in one shard file with nothing else in it. Global amax 0.0894, per-shard
outlier ratio 1.66x, scale chosen exactly representable in bf16 so no
scale-rounding error stacks on quantization; amax maps to 446.17/448, no
clipping. Round-trip 2.655% RMS relative, 0.002% underflow, 0 saturation --
the same FP8-PLE treatment dealignai already shipped. MTP head (31 tensors)
and vision tower carried through untouched.
A second, independent blocker followed: orcarouter labels its 12 QSA layers
qwen_sparse_attention, which vLLM rejects; it accepts full_attention and
selects QSA via indexer_n_heads. Confirmed indexer_n_heads == 4 in both this
and the dealignai checkpoint before renaming -- without that check the rename
silently selects plain attention and serves a subtly wrong model that still
passes a healthcheck.
Measured on the live seat: healthy, coherent, KV 344,155 tokens @ 262,144 ctx,
MTP k=3 at 60.4% acceptance / 2.81 mean acceptance length, warm decode median
167.5 tok/s at conc=1 (n=5, spread 12.2%). The reorg note's dealignai figure
came from a different harness, so this is not claimed as a win over it; what
it does establish is that weight-only experts did not cost decode speed.
Still open: controlled quality A/B vs dealignai, and a deep-prefill probe at
262K. Rollback is two .env keys; dealignai remains on disk.
Also corrects the README's MTP-is-off section, stale since k=3 was deployed,
and adds a superseded-claims row to the quantization playbook.
FV colo recovered 2026-09-13 midday (chassis on PDU, firewall on the Eaton 5P1000,
GPU caps 275W/card). All-night fv-ml1 seat reorganization:
- flash-next gained MTP k=3 (campaign measured it a win here, +52% at conc=1),
inverting vLLM's 4xH100 recipe; KV 14->10 GiB.
- gen consolidated onto flash-next (all 8 gen/summarizer/classifier/judge aliases
repointed); 27B dense gen seat retired, 38 GB freed on GPU0.
- char-rp restored to the in-house MeroMero-v2-31B dense heretic (was serving a
leftover-test RedHatAI 26B); char-rp-fast is the deliberate speed tier.
- Sentinel-R3 (SFT pentest finetune) served for an A/B vs mog-sec, then dflash k=7
cut over after measuring it beat MTP (2.40 vs 2.18 acceptance, ~121 tok/s warm).
gen-large is intentionally DOWN: the orcarouter weight-only NVFP4 build downloaded
(170 GB, verified) but no mainline vLLM loads its compressed-tensors qwen4_exp PLE;
the third-party backport was vetted and is unfit (old-hardware fork, no Blackwell
image). Runtime decision pending -- this is the resume point.
Also this session: vh/infra-reference repo, scripts/seat-inventory.py + daily drift
alarm, OPNsense API reference vendored, secrets shed from a prior scratchpad.
Leaves the fv-to-ana-nat files (another session's) and graphify-out untouched.
vh