esh-pfi-infrastructure/services/gen-seat-mixed-quant

gen-seat mixed-precision quant — NVFP4 W4A4 MLP + FP8 W8A8 attention

The pipeline that produced /tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed, the current fleet gen seat on ana-ml2 GPU0 :8015. +18% decode over the previous weight-only NVFP4A16 build, at equal MTP acceptance and +1.7% perplexity.

The headline correction: "W4A8" is not a thing you can serve

The queued task was "re-quant to NVFP4 weights + FP8 activations (W4A8) for ~20% more decode". That checkpoint cannot load. vLLM 0.24's compressed-tensors dispatcher (compressed_tensors.py:704-713) allows NVFP4 weights with exactly two activation options:

input_activations	scheme	kernel
None	W4A16	Marlin (forced — kernels/linear/__init__.py:881-883)
NVFP4	W4A4	native Blackwell FP4

anything else — FP8 included — raises

ValueError: For NVFP4 weights, input quantization must also be NVFP4 format,
            None for NVFP4A16

CompressedTensorsW4A8Fp8 exists but is INT4 weights (W4A8_SUPPORTED_TYPES_MAP = {4: int4}) gated on _check_scheme_supported(90, match_exact=True) — Hopper only. ana-ml2 is Blackwell (sm_120), so that path is doubly closed.

The ~20% intuition was right; the scheme name was wrong. The servable way to get FP8 into the mix is per-layer-group, which is exactly what unsloth/Qwen3.8-27B-NVFP4 does — and that build, measured on-box, ran +19.1% faster than ours at identical MTP acceptance. This pipeline replicates its recipe on the abliterated weights.

The recipe

group	scheme	targets
group_0	FP8 W8A8 — channel weights (static), per-token dynamic activations	self_attn.{q,k,v,o}_proj, linear_attn.{in_proj_qkv,in_proj_z,out_proj}, lm_head, layers 56-63 MLPs
group_1	NVFP4 W4A4 — tensor_group gsize16, fp8 scales, imatrix_mse weights, dynamic:"local" activations	layers 0-55 MLP {gate,up,down}_proj
kv cache	FP8 static tensor	—
ignored	vision tower, linear_attn.{norm,in_proj_a,in_proj_b}, re:^mtp.*	—

Keeping the last 8 layers' MLPs at FP8 is the accuracy-preservation trick — late layers are the sensitive ones. conv1d in linear_attn is not a Linear and stays BF16 in both our build and unsloth's.

Targets are deliberately non-overlapping (group_1 enumerates layers 0-55 rather than matching all MLPs) instead of relying on group precedence to resolve the 56-63 collision. validate_targets.py proves this against the real module names before any GPU time is spent — run it first.

Running it

# 0. prove the regexes hit what you think (free, no GPU)
python3 validate_targets.py /tank/aimodels/qwen38-27b-uncensored-bf16
#    -> expect OVERLAP 0, MLP layer union covers 0-63 True

# 1. quant (~20 min on one Blackwell; needs llmcompressor, NOT modelopt)
python3 quant_mixed_nvfp4.py \
  --model /tank/aimodels/qwen38-27b-uncensored-bf16 \
  --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
  --out   /tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed \
  --num-samples 256 --seqlen 2048

# 2. MANDATORY post-steps — graft MTP, restore preprocessor, repair the mtp ignore
python3 post_quant.py <bf16-source> <out-dir>

pip install llmcompressor into the stock vllm/vllm-openai:latest image gives llmcompressor 0.13.0 + compressed-tensors 0.18.0 without disturbing torch or transformers. Do not use modelopt 0.43 — dependency hell on qwen3_5.

The foot-gun that has now cost three rounds

llm-compressor prunes ignore entries that matched no module at quant time. The wrapper class (Qwen3_5ForConditionalGeneration) never loads the MTP head, so re:^mtp.* matches nothing and is silently dropped from the saved config. vLLM then treats the freshly grafted BF16 MTP head as quantized, brings it up uninitialised, and speculative decoding runs at 0% acceptance.

post_quant.py re-injects the entry after the graft and re-verifies. It is not optional, and it verifies rather than assumes — that check fired on this very run.

Acceptance gate (bench/)

Speed alone does not justify cutting over a seat backing 7 LiteLLM aliases.

metric	W4A16 (old)	mixed (new)	delta
decode tok/s, bs=1, cache-busted	80.12	94.53	+18.0%
MTP acceptance	47.8%	47.7%	unchanged
perplexity, 6 held-out passages	6.941	7.059	+1.7% worse
abliteration compliance	4/4	4/4	preserved
weights on disk	27.7 GB	22.5 GB	−19%

• quickbench.py — cache-busted bs=1 decode + MTP acceptance. Bust the cache: with a fixed prompt, prefix caching returns byte-identical timings and you measure nothing.
• eval_quality.py — perplexity, deterministic generations, abliteration survival. PPL must be measured with --speculative-config OFF: under MTP, vLLM's prompt_logprobs come back ~uniform over the vocab (median rank ~10^5, logprob ≈ log(1/vocab)). The harness raises rather than reporting the garbage.
• surface_test.py — the real gate: plain chat, vision, tool calling, thinking split, 36K-token needle retrieval, streaming. All six must pass before a cutover.
• serve_probe.sh <model-dir> [nospec] — serve a candidate on :8017 without touching the live seat.

GPU0 budget

The mixed build's weights are 5.2 GB smaller. At the old GEN_GPU_MEM_UTIL=0.45 the seat absorbed that slack as extra KV (17.0 GiB / 477K tokens) and left meromero-charrp 0.18 GiB short of its 0.52 budget — it crash-looped on startup. Fixed by handing the space back: GEN_GPU_MEM_UTIL=0.43 → 15.1 GiB / 422K tokens, still 1.6× the 262K context. Both seats co-resident at 89.8 / 97.9 GB.

Levers already measured — do not re-chase

GEN_SPEC_TOKENS swept on this seat: n=2 → 77.1, n=3 → 80.1, n=4 → 78.7, n=5 → 75.9 tok/s. Three is the optimum; higher n trades acceptance for draft width and loses.

Rollback

The previous build is untouched at /tank/aimodels/qwen38-27b-uncensored-nvfp4.

ssh infra-ops@10.250.50.54
sudo cp /opt/docker/compose/gen-seat/.env.bak-w4a16-20260815 /opt/docker/compose/gen-seat/.env
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen

⚠ gen-seat/.env is mode 0600 and lkraven-owned — every docker compose call against it needs sudo, or compose fails with permission denied reading .env, leaves the old container running, and the change silently does not take.