vh
a91cc3fb38
Quants are hard-fought and we keep re-paying for the same lessons. A survey found quant knowledge scattered across 18 files in four trees, with three documents having independently discovered and recorded overlapping "landmines" sections — and one of them now actively misleading. Adds docs/pfi/model-quantization-playbook.md as the single home for the TRANSFERABLE lessons, with per-model artifacts demoted to worked examples that link up to it. Contents: - scheme decision table, incl. that a literal "W4A8" NVFP4 checkpoint is unservable on vLLM (two legal activation settings, FP8 is not one) - the reference mixed-precision recipe and the three parts of it that are load-bearing and easy to drop - the recurring landmines, ordered by cost: the loader-class trap (rediscovered THREE times), the three separate ways to lose the MTP head, toolchain deadlocks, vision configs, memory/device placement - pipeline shape: prove targets before spending GPU time; mandatory post-steps that verify rather than assume - the acceptance gate, and the three ways measurement has lied to us — prefix caching faking both speed metrics, prompt_logprobs going uniform under speculative decoding, and a 0600 .env making compose silently no-op - hardware/co-residency, including that a SMALLER model can starve its neighbour because gpu-memory-utilization is a fraction of the whole card - a superseded-claims table, and measured negatives not to re-chase The superseded table earns its place immediately: the heretic2 runbook tells readers to use modelopt because "compressed-tensors can't load the BF16 MTP head, 0% acceptance". That symptom was real but the cause was not the format -- it was the missing re:^mtp.* ignore entry. compressed-tensors gives 47.7-83.2% acceptance in production. A fresh session following that doc would be sent down the modelopt path that current memory calls dependency hell, so the runbook now carries a stale-warning header pointing here. Wires discovery: an orientation.md "Where to look for what" row, pointers from the gen-seat / heretic2 / mistral artifacts, and a CLAUDE.md maintenance rule so the playbook gets fed instead of going stale -- model-agnostic lessons land in the playbook, model-specific ones stay put, and a wrong claim earns a dated superseded row rather than a silent edit. Motivated by Qwen3.8 having just released: the next model swap will need a requant, and this is what that session should read first.
7.5 KiB
gen-seat mixed-precision quant — NVFP4 W4A4 MLP + FP8 W8A8 attention
The pipeline that produced /tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed, the
current fleet gen seat on ana-ml2 GPU0 :8015. +18% decode over the previous
weight-only NVFP4A16 build, at equal MTP acceptance and +1.7% perplexity.
General lessons live in
docs/pfi/model-quantization-playbook.md— read that before starting a quant on a different model. This file is the worked example for Qwen3.8-27B-Uncensored specifically. Where the two disagree, the playbook wins.
The headline correction: "W4A8" is not a thing you can serve
The queued task was "re-quant to NVFP4 weights + FP8 activations (W4A8) for ~20%
more decode". That checkpoint cannot load. vLLM 0.24's compressed-tensors
dispatcher (compressed_tensors.py:704-713) allows NVFP4 weights with exactly two
activation options:
| input_activations | scheme | kernel |
|---|---|---|
None |
W4A16 | Marlin (forced — kernels/linear/__init__.py:881-883) |
| NVFP4 | W4A4 | native Blackwell FP4 |
anything else — FP8 included — raises
ValueError: For NVFP4 weights, input quantization must also be NVFP4 format,
None for NVFP4A16
CompressedTensorsW4A8Fp8 exists but is INT4 weights (W4A8_SUPPORTED_TYPES_MAP = {4: int4}) gated on _check_scheme_supported(90, match_exact=True) — Hopper only.
ana-ml2 is Blackwell (sm_120), so that path is doubly closed.
The ~20% intuition was right; the scheme name was wrong. The servable way to get FP8
into the mix is per-layer-group, which is exactly what unsloth/Qwen3.8-27B-NVFP4
does — and that build, measured on-box, ran +19.1% faster than ours at identical MTP
acceptance. This pipeline replicates its recipe on the abliterated weights.
The recipe
| group | scheme | targets |
|---|---|---|
group_0 |
FP8 W8A8 — channel weights (static), per-token dynamic activations | self_attn.{q,k,v,o}_proj, linear_attn.{in_proj_qkv,in_proj_z,out_proj}, lm_head, layers 56-63 MLPs |
group_1 |
NVFP4 W4A4 — tensor_group gsize16, fp8 scales, imatrix_mse weights, dynamic:"local" activations |
layers 0-55 MLP {gate,up,down}_proj |
| kv cache | FP8 static tensor | — |
| ignored | vision tower, linear_attn.{norm,in_proj_a,in_proj_b}, re:^mtp.* |
— |
Keeping the last 8 layers' MLPs at FP8 is the accuracy-preservation trick — late
layers are the sensitive ones. conv1d in linear_attn is not a Linear and stays BF16
in both our build and unsloth's.
Targets are deliberately non-overlapping (group_1 enumerates layers 0-55 rather
than matching all MLPs) instead of relying on group precedence to resolve the 56-63
collision. validate_targets.py proves this against the real module names before any
GPU time is spent — run it first.
Running it
# 0. prove the regexes hit what you think (free, no GPU)
python3 validate_targets.py /tank/aimodels/qwen38-27b-uncensored-bf16
# -> expect OVERLAP 0, MLP layer union covers 0-63 True
# 1. quant (~20 min on one Blackwell; needs llmcompressor, NOT modelopt)
python3 quant_mixed_nvfp4.py \
--model /tank/aimodels/qwen38-27b-uncensored-bf16 \
--calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
--out /tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed \
--num-samples 256 --seqlen 2048
# 2. MANDATORY post-steps — graft MTP, restore preprocessor, repair the mtp ignore
python3 post_quant.py <bf16-source> <out-dir>
pip install llmcompressor into the stock vllm/vllm-openai:latest image gives
llmcompressor 0.13.0 + compressed-tensors 0.18.0 without disturbing torch or
transformers. Do not use modelopt 0.43 — dependency hell on qwen3_5.
The foot-gun that has now cost three rounds
llm-compressor prunes ignore entries that matched no module at quant time.
The wrapper class (Qwen3_5ForConditionalGeneration) never loads the MTP head, so
re:^mtp.* matches nothing and is silently dropped from the saved config. vLLM then
treats the freshly grafted BF16 MTP head as quantized, brings it up uninitialised,
and speculative decoding runs at 0% acceptance.
post_quant.py re-injects the entry after the graft and re-verifies. It is not
optional, and it verifies rather than assumes — that check fired on this very run.
Acceptance gate (bench/)
Speed alone does not justify cutting over a seat backing 7 LiteLLM aliases.
| metric | W4A16 (old) | mixed (new) | delta |
|---|---|---|---|
| decode tok/s, bs=1, cache-busted | 80.12 | 94.53 | +18.0% |
| prefill tok/s, ~6.7k prompt | 3,206 | 6,334 | +98% |
| prefill tok/s, ~27k prompt | 2,862 | 5,085 | +78% |
| TTFT on a ~27k-token doc | 9.43 s | 5.31 s | −44% |
| MTP acceptance | 47.8% | 47.7% | unchanged |
| perplexity, 6 held-out passages | 6.941 | 7.059 | +1.7% worse |
| abliteration compliance | 4/4 | 4/4 | preserved |
| weights on disk | 27.7 GB | 22.5 GB | −19% |
Prefill roughly doubled — the bigger practical win, and exactly what theory predicts:
decode at bs=1 is memory-bandwidth-bound (weights are 4-bit either way, so little changes),
while prefill is compute-bound and is where Blackwell's native FP4 tensor cores replace the
Marlin dequant-to-BF16 path. This is what the summarizer / summarizer-large aliases feel
on long documents.
quickbench.py— cache-busted bs=1 decode + MTP acceptance. Bust the cache: with a fixed prompt, prefix caching returns byte-identical timings and you measure nothing.prefill_bench.py— TTFT on long prompts. Same trap, worse: a seeded nonce reproduces the previous run's prompts verbatim, so prefix caching serves them and you read ~41k tok/s of cache-hit instead of ~5k of real prefill. UsesSystemRandom; never seed it.eval_quality.py— perplexity, deterministic generations, abliteration survival. PPL must be measured with--speculative-configOFF: under MTP, vLLM'sprompt_logprobscome back ~uniform over the vocab (median rank ~10^5, logprob ≈ log(1/vocab)). The harness raises rather than reporting the garbage.surface_test.py— the real gate: plain chat, vision, tool calling, thinking split, 36K-token needle retrieval, streaming. All six must pass before a cutover.serve_probe.sh <model-dir> [nospec]— serve a candidate on:8017without touching the live seat.
GPU0 budget
The mixed build's weights are 5.2 GB smaller. At the old GEN_GPU_MEM_UTIL=0.45 the
seat absorbed that slack as extra KV (17.0 GiB / 477K tokens) and left meromero-charrp
0.18 GiB short of its 0.52 budget — it crash-looped on startup. Fixed by handing the
space back: GEN_GPU_MEM_UTIL=0.43 → 15.1 GiB / 422K tokens, still 1.6× the 262K
context. Both seats co-resident at 89.8 / 97.9 GB.
Levers already measured — do not re-chase
GEN_SPEC_TOKENS swept on this seat: n=2 → 77.1, n=3 → 80.1, n=4 → 78.7,
n=5 → 75.9 tok/s. Three is the optimum; higher n trades acceptance for draft width and
loses.
Rollback
The previous build is untouched at /tank/aimodels/qwen38-27b-uncensored-nvfp4.
ssh infra-ops@10.250.50.54
sudo cp /opt/docker/compose/gen-seat/.env.bak-w4a16-20260815 /opt/docker/compose/gen-seat/.env
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen
⚠ gen-seat/.env is mode 0600 and lkraven-owned — every docker compose call
against it needs sudo, or compose fails with permission denied reading .env,
leaves the old container running, and the change silently does not take.