esh-pfi-infrastructure/docs/pfi/model-quantization-playbook.md

Model quantization playbook — the lessons that keep costing us hours

Read this before starting any new quant. Not the per-model runbooks — those are worked examples of a specific model at a specific point in time, and several carry claims that are now false (see §7). This file owns the transferable part: what recurs regardless of which model dropped this week.

Written 2026-08-15, after the fourth quant in five weeks re-discovered the third-known instance of the same loader-class bug. Scope: NVFP4 / FP8 / mixed-precision on the Blackwell boxes (ana-ml2), vLLM-served. Ampere (irv-ml1) has no native FP4/FP8 — see §6.

Maintenance rule. When a quant teaches you something model-agnostic, it lands here and the per-model README links up. When it's model-specific (this checkpoint's odd tensor names, this finetune's missing config), it stays in the per-model artifact. If you find yourself writing a "Gotchas" section that repeats §3, you are re-litigating — add the delta here instead.

---

1. The 60-second decision: which scheme

On Blackwell + vLLM, for a dense-or-hybrid VL model you intend to serve at long context:

want	scheme	notes
default, best speed/accuracy	mixed: NVFP4 W4A4 bulk MLPs + FP8 W8A8 attention/lm_head/last-8-layer MLPs	the current answer. §2.
max fidelity, don't care about prefill	NVFP4 W4A16 (weight-only)	forces the Marlin kernel — ~half the prefill of native FP4
small model, VRAM is free	FP8 W8A8	safe and simple; 2× the weight bytes of 4-bit
—	"W4A8" = NVFP4 weights + FP8 activations	DOES NOT EXIST. §3.1

Measured on Qwen3.8-27B (2026-08-15), W4A16 → mixed: decode +18%, prefill +78–98%, MTP acceptance unchanged, perplexity +1.7%, weights −19%.

Note the shape of that: decode barely moves, prefill nearly doubles. Decode at batch-1 is memory-bandwidth-bound and the weights are 4-bit under either scheme, so there is little to win; prefill is compute-bound, which is where native FP4 tensor cores replace the Marlin dequantize-to-BF16 path. If someone promises you a big decode win from a scheme change, be skeptical — and go measure §5 before believing it.

The accuracy cost is real and is paid on purpose. Operator ruling 2026-08-15: the ~1.7% perplexity is an acceptable price for the speed. Settled — don't re-litigate. For correct attribution: it is the activation-quantization cost (A4/A8 vs BF16 activations), not an MTP cost. Turning MTP off does not recover it; only reverting the quant does.

---

2. The reference recipe (mixed-precision)

Lifted from unsloth/Qwen3.8-27B-NVFP4 and replicated in-house. Prefer replicating a published recipe from a reputable quantizer over inventing one — they have already paid for the sensitivity analysis.

group	scheme	targets
group_0	FP8 W8A8 — channel weights (static) + per-token dynamic activations	self_attn.{q,k,v,o}_proj, linear_attn.{in_proj_qkv,in_proj_z,out_proj}, lm_head, the last 8 layers' MLPs
group_1	NVFP4 W4A4 — tensor_group gsize 16, fp8 scales, imatrix_mse weights, dynamic:"local" activations	all remaining MLP {gate,up,down}_proj
kv cache	FP8 static tensor
ignore	vision tower, linear_attn.{norm,in_proj_a,in_proj_b}, re:^mtp.*

Three things in there are load-bearing and easy to drop:

• Late layers stay FP8. Holding the last ~8 layers' MLPs (and lm_head) at 8-bit is the accuracy-preservation trick — late layers are the sensitive ones. Uniform W4A4 is what collapses.
• imatrix_mse on the W4A4 weights, not memoryless_minmax. Importance-weighted; needs calibration data.
• Group targets must be non-overlapping. Do not let group_1's .*mlp\..* also match the late layers and rely on group precedence to sort it out. Enumerate the early layers explicitly (re:.*layers\.([0-9]|[1-4][0-9]|5[0-5])\.mlp\.…) and prove it with a dry run (§4.1).

Toolchain: pip install llmcompressor into stock vllm/vllm-openai:latest gives llmcompressor 0.13 + compressed-tensors 0.18 without disturbing torch/transformers. Avoid nvidia-modelopt — see §3.4.

---

3. The recurring landmines

Ordered by how much time each has cost. Every one of these has bitten more than once.

3.1 "W4A8" is not a servable shape

vLLM's compressed-tensors dispatcher (compressed_tensors.py:704-713) accepts NVFP4 weights with exactly two activation settings:

input_activations	result
None	W4A16 — and it forces the Marlin kernel (kernels/linear/__init__.py:881-883)
NVFP4	W4A4, native

Anything else — FP8 included — raises at load:

ValueError: For NVFP4 weights, input quantization must also be NVFP4 format, None for NVFP4A16

CompressedTensorsW4A8Fp8 exists but is INT4 weights (W4A8_SUPPORTED_TYPES_MAP = {4: int4}) gated on _check_scheme_supported(90, match_exact=True) — Hopper-exact, so on Blackwell (sm_120) it is closed twice over. FP8 enters per-layer-group, never as activations on NVFP4 weights.

Cost: one queued task written against an impossible scheme.

3.2 Wrong loader class → silent weight-load failure

Rediscovered three times. Load the model through the class vLLM actually serves — the …ForConditionalGeneration / …ForImageTextToText wrapper, never AutoModelForCausalLM.

AutoModelForCausalLM resolves a VL config to the text-only inner class and saves a flat config with model.layers.* keys. vLLM's weight mapper wants model.language_model.* (+ model.visual.*). The mismatch does not error — every layer silently fails to load and you get !!!! gibberish, or an engine that rejects the checkpoint outright.

Bit: heretic2 (gibberish), Dark-Scarlett (both vLLM and SGLang refused the checkpoint), and the 2026-08 rounds.

3.3 The MTP head — three separate ways to lose it

Speculative decoding is a large fraction of the seat's throughput. It fails silently: the model serves fine, just at 0% acceptance.

1. The wrapper class does not instantiate mtp.*, so the quant drops it. Post-quant you must graft the BF16 model-mtp.safetensors back and register its tensors in the output index.
2. re:^mtp.* must be in quantization_config.ignore — else vLLM loads the grafted BF16 head as though quantized, it comes up uninitialised, and acceptance is 0%.
3. ⭐ llm-compressor PRUNES ignore entries that matched no module at quant time. Since the wrapper never loaded mtp.*, the entry matches nothing and is silently deleted from the saved config — even though you put it in the recipe. So it must be re-injected after the graft, and then verified, not assumed.

Cost: three rounds. The verify step caught it live on the third.

There is also a modelopt-format-specific version of this: vLLM 0.24 does not propagate modelopt exclude_modules to the spec-decode draft model, which no checkpoint config can fix (needs a sitecustomize runtime patch). Using compressed-tensors avoids it entirely — §3.4.

3.4 Toolchain version deadlocks

Both directions have burned us, so the resolution is: use llm-compressor / compressed-tensors, not nvidia-modelopt.

• modelopt 0.45 ↔ transformers 5.12: mtq.quantize dies TypeError: issubclass() arg 2 must be a class (modelopt registers transformers' FusedMoE, a function in 5.x, as an nn class).
• modelopt 0.43 doesn't fix it — it drags transformers back to 4.57, which cannot load qwen3_5 at all.
• modelopt's config API also trails the current model families by a version.

3.5 Vision tower and its configs

• Keep the vision tower in ignore (BF16). Only the LLM backbone gets quantized.
• The wrapper-class save drops preprocessor_config.json (and the video one). Without it the seat crash-loops Can't load image processor. Restore from the source — and if the upstream repo omits it, reconstruct it from processor_config.json's image_processor sub-dict.

3.6 Memory and device placement (large models)

• device_map=None/"cpu", never "auto". auto fills GPU0 and OOMs during un-fusing; constraining with max_memory then offloads to the meta device, which cannot be .copy_()d. CPU-resident keeps every tensor real; the sequential pipeline still onloads per-layer to GPU.
• Avoid mmap on /tank. safetensors.safe_open() mmaps a whole shard; on ZFS a 50 GB shard ENOMEMs regardless of free RAM (MAP_SHARED never consults the commit limit). Read with plain read() + load(bytes), one shard cached at a time.
• vm.overcommit_memory=1 on ana-ml2 (durable via playbooks/ana-ml2-overcommit-memory.yaml).

---

4. Pipeline shape

4.1 Prove the targets before spending GPU time

Enumerate module names from the safetensors index and check your regexes against them: zero overlap between groups, and the union covers every layer you intended. This is free, takes seconds, and catches a mis-scoped regex that would otherwise surface as a mystery quality regression hours later. Reference: services/gen-seat-mixed-quant/validate_targets.py.

4.2 Quantize

Calibration data matters for imatrix_mse + static activation observers. We use /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl (512 chat samples, RP/GM-flavoured — appropriate for our seats). 256 samples @ 2048 tokens ≈ 20 min for a 27B on one Blackwell.

4.3 The mandatory post-steps

Never optional, always in this order, and the last one verifies rather than assumes:

1. Graft model-mtp.safetensors + register its tensors in the output index.
2. Restore preprocessor_config.json / processor_config.json / video_preprocessor_config.json.
3. Re-inject re:^mtp.* into quantization_config.ignore and confirm it is there (§3.3).

Reference implementation: services/gen-seat-mixed-quant/post_quant.py.

4.4 Test on a temp port, never on the live seat

Serve the candidate on an alt port with the live seat's exact flags, run the gate (§5), and only then flip .env. Keep the previous build on disk; rollback is one .env line.

---

5. The acceptance gate — and how measurement lies to you

Speed alone does not justify cutting over a shared seat. Gate on all of: decode tok/s, MTP acceptance, perplexity, a behavioural surface test, and — for an abliterated model — that the abliteration survived.

Three ways the numbers have lied to us. All three produced confident, wrong results.

1. Prefix caching fakes both speed metrics. A fixed prompt returns byte-identical timings run after run; you are measuring cache, not compute. Worse for prefill: a seeded nonce regenerates the previous run's prompts verbatim and reads ~41k tok/s of cache-hit instead of ~5k of real prefill. Use a fresh unseeded nonce per request; never seed a cache-buster.
2. prompt_logprobs are garbage while speculative decoding is on — ~uniform over the vocab (median rank ~10⁵; " Paris" after "The capital of France is" ranked 69698). Perplexity must be measured on a seat served without --speculative-config, on both sides of the comparison.
3. A 0600 .env makes docker compose silently no-op. Without sudo it fails permission denied reading .env, leaves the old container running, and reports success — producing a full page of "benchmark results" that were just the unchanged baseline. Hard-verify the change landed against docker inspect …Config.Cmd.

Re-measure the baseline before believing a target. The 2026-08-15 handoff quoted ~68 tok/s; cache-busted, the incumbent was already doing 80.1 — essentially the target of the work queued against it. Had that not been re-measured, doing nothing would have looked like a 20% win.

Cheap shortcut worth taking first: if a reputable published quant of the same architecture is already on-box (or is a small pull), serve it as a probe and measure it before committing hours to your own. It answers "is this gain even real?" in ten minutes and hands you the recipe.

Harness: services/gen-seat-mixed-quant/bench/ — quickbench.py (decode + acceptance), prefill_bench.py, eval_quality.py (PPL + abliteration), surface_test.py (chat, vision, tools, thinking split, long-context needle, streaming), serve_probe.sh.

---

6. Hardware and co-residency

• ana-ml2 = Blackwell (sm_120), 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths (match_exact=True on sm90) are closed here — do not plan around them.
• irv-ml1 = Ampere (sm_86), 3090 + A6000. No native FP8/FP4 — 4-bit there is a VRAM saving only, not a speed win. Don't port a Blackwell scheme over and expect the throughput.
• GPU co-residency is a zero-sum budget, and a smaller model can break its neighbour. gpu-memory-utilization is a fraction of the whole card, so when new weights are smaller the seat absorbs the slack as extra KV rather than releasing it. That is exactly how a −5.2 GB requant left the co-resident seat 0.18 GiB short and crash-looping. After any requant, re-check both seats' budgets and hand the space back explicitly.

---

7. Superseded claims — do not follow these

Old docs stay for their history, but these specific claims are false now and will cost you a day if followed:

claim	where	status
"Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance"	docs/runbooks/heretic2-nvfp4-mtp-seat.md §landmine 2	SUPERSEDED 2026-08-14. The 0% was the missing re:^mtp.* ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.7–83.2% acceptance, live. Use compressed-tensors.
"Abliteration desyncs the MTP head → uncensored models can't do MTP"	earlier auto-memory	SUPERSEDED 2026-08-14. A modest abliteration preserves MTP (83.7% at bf16). Test MTP on bf16 first to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source.
"NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer"	reference_nvfp4_w4a4_granite_infeasible	NARROWED. True for uniform W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs with FP8 on attention and late layers is fine and is the current default (§2).

---

8. Measured negatives — don't re-chase

• num_speculative_tokens = 3 is optimal on the Qwen3.8-27B seat. Swept: n=2 → 77.1, n=3 → 80.1, n=4 → 78.7, n=5 → 75.9 tok/s. Higher n trades acceptance for draft width and loses. Re-sweep only if the drafter architecture changes.
• Uniform W4A4 — see §7 row 3.
• Dense-VL as the anatomy judge — A/B'd, MoE retained. Don't re-propose.

---

9. Worked examples

Per-model artifacts. Read for how a specific model went, not for the general lessons — those are above, and where the two disagree, this file wins.

artifact	what it is
services/gen-seat-mixed-quant/	current reference. Mixed NVFP4+FP8 on Qwen3.8-27B-Uncensored: scripts, acceptance harness, raw measurements.
stacks/gen-seat/README.md	the live gen seat (7 LiteLLM aliases)
stacks/meromero-charrp/README.md	Gemma-4 seat — the tool-call/reasoning-parser trap (a parser default that returns null content for all prose)
services/heretic2-nvfp4-quant/	modelopt-format MTP seat — historical; see §7 before following it
tools/mistral-small4-nvfp4/	MoE + native-convert path; source of §3.6
docs/pfi/recommended-model-settings.md	serve-time sampler/flag defaults (not quant)

A new model just dropped and needs requanting? §1 → §2 → §4 → §5. Skim §3 first; it is the part that costs hours.