esh-pfi-infrastructure/docs/pfi/model-quantization-playbook.md

Model quantization playbook — the lessons that keep costing us hours

Read this before starting any new quant. Not the per-model runbooks — those are worked examples of a specific model at a specific point in time, and several carry claims that are now false (see §7). This file owns the transferable part: what recurs regardless of which model dropped this week.

Written 2026-08-15, after the fourth quant in five weeks re-discovered the third-known instance of the same loader-class bug. Scope: NVFP4 / FP8 / mixed-precision on the Blackwell boxes (ana-ml2), vLLM-served. Ampere (irv-ml1) has no native FP4/FP8 — see §6.

Maintenance rule. When a quant teaches you something model-agnostic, it lands here and the per-model README links up. When it's model-specific (this checkpoint's odd tensor names, this finetune's missing config), it stays in the per-model artifact. If you find yourself writing a "Gotchas" section that repeats §3, you are re-litigating — add the delta here instead.

---

1. The 60-second decision: which scheme

On Blackwell + vLLM, for a dense-or-hybrid VL model you intend to serve at long context:

want	scheme	notes
default, best speed/accuracy	mixed: NVFP4 W4A4 bulk MLPs + FP8 W8A8 attention/lm_head/last-8-layer MLPs	the current answer. §2.
max fidelity, don't care about prefill	NVFP4 W4A16 (weight-only)	forces the Marlin kernel — ~half the prefill of native FP4
small model, VRAM is free	FP8 W8A8	safe and simple; 2× the weight bytes of 4-bit
—	"W4A8" = NVFP4 weights + FP8 activations	DOES NOT EXIST. §3.1

Measured on Qwen3.8-27B (2026-08-15), W4A16 → mixed: decode +18%, prefill +78–98%, MTP acceptance unchanged, perplexity +1.7%, weights −19%.

Note the shape of that: decode barely moves, prefill nearly doubles. Decode at batch-1 is memory-bandwidth-bound and the weights are 4-bit under either scheme, so there is little to win; prefill is compute-bound, which is where native FP4 tensor cores replace the Marlin dequantize-to-BF16 path. If someone promises you a big decode win from a scheme change, be skeptical — and go measure §5 before believing it.

The accuracy cost is real and is paid on purpose. Operator ruling 2026-08-15: the ~1.7% perplexity is an acceptable price for the speed. Settled — don't re-litigate. For correct attribution: it is the activation-quantization cost (A4/A8 vs BF16 activations), not an MTP cost. Turning MTP off does not recover it; only reverting the quant does.

---

2. The reference recipe (mixed-precision)

Lifted from unsloth/Qwen3.8-27B-NVFP4 and replicated in-house. Prefer replicating a published recipe from a reputable quantizer over inventing one — they have already paid for the sensitivity analysis.

group	scheme	targets
group_0	FP8 W8A8 — channel weights (static) + per-token dynamic activations	self_attn.{q,k,v,o}_proj, linear_attn.{in_proj_qkv,in_proj_z,out_proj}, lm_head, the last 8 layers' MLPs
group_1	NVFP4 W4A4 — tensor_group gsize 16, fp8 scales, imatrix_mse weights, dynamic:"local" activations	all remaining MLP {gate,up,down}_proj
kv cache	FP8 static tensor
ignore	vision tower, linear_attn.{norm,in_proj_a,in_proj_b}, re:^mtp.*

Three things in there are load-bearing and easy to drop:

• Late layers stay FP8. Holding the last ~8 layers' MLPs (and lm_head) at 8-bit is the accuracy-preservation trick — late layers are the sensitive ones. Uniform W4A4 is what collapses.
• imatrix_mse on the W4A4 weights, not memoryless_minmax. Importance-weighted; needs calibration data.
• Group targets must be non-overlapping. Do not let group_1's .*mlp\..* also match the late layers and rely on group precedence to sort it out. Enumerate the early layers explicitly (re:.*layers\.([0-9]|[1-4][0-9]|5[0-5])\.mlp\.…) and prove it with a dry run (§4.1).

Toolchain: pip install llmcompressor into stock vllm/vllm-openai:latest gives llmcompressor 0.13 + compressed-tensors 0.18 without disturbing torch/transformers. Avoid nvidia-modelopt — see §3.4.

---

3. The recurring landmines

Ordered by how much time each has cost. Every one of these has bitten more than once.

3.1 "W4A8" is not a servable shape

vLLM's compressed-tensors dispatcher (compressed_tensors.py:704-713) accepts NVFP4 weights with exactly two activation settings:

input_activations	result
None	W4A16 — and it forces the Marlin kernel (kernels/linear/__init__.py:881-883)
NVFP4	W4A4, native

Anything else — FP8 included — raises at load:

ValueError: For NVFP4 weights, input quantization must also be NVFP4 format, None for NVFP4A16

CompressedTensorsW4A8Fp8 exists but is INT4 weights (W4A8_SUPPORTED_TYPES_MAP = {4: int4}) gated on _check_scheme_supported(90, match_exact=True) — Hopper-exact, so on Blackwell (sm_120) it is closed twice over. FP8 enters per-layer-group, never as activations on NVFP4 weights.

Cost: one queued task written against an impossible scheme.

3.2 Wrong loader class → silent weight-load failure

Rediscovered three times. Load the model through the class vLLM actually serves — the …ForConditionalGeneration / …ForImageTextToText wrapper, never AutoModelForCausalLM.

AutoModelForCausalLM resolves a VL config to the text-only inner class and saves a flat config with model.layers.* keys. vLLM's weight mapper wants model.language_model.* (+ model.visual.*). The mismatch does not error — every layer silently fails to load and you get !!!! gibberish, or an engine that rejects the checkpoint outright.

Bit: heretic2 (gibberish), Dark-Scarlett (both vLLM and SGLang refused the checkpoint), and the 2026-08 rounds.

3.3 The MTP head — three separate ways to lose it

Speculative decoding is a large fraction of the seat's throughput. It fails silently: the model serves fine, just at 0% acceptance.

1. The wrapper class does not instantiate mtp.*, so the quant drops it. Post-quant you must graft the BF16 model-mtp.safetensors back and register its tensors in the output index.
2. re:^mtp.* must be in quantization_config.ignore — else vLLM loads the grafted BF16 head as though quantized, it comes up uninitialised, and acceptance is 0%.
3. ⭐ llm-compressor PRUNES ignore entries that matched no module at quant time. Since the wrapper never loaded mtp.*, the entry matches nothing and is silently deleted from the saved config — even though you put it in the recipe. So it must be re-injected after the graft, and then verified, not assumed.

Cost: three rounds. The verify step caught it live on the third.

There is also a modelopt-format-specific version of this: vLLM 0.24 does not propagate modelopt exclude_modules to the spec-decode draft model, which no checkpoint config can fix (needs a sitecustomize runtime patch). Using compressed-tensors avoids it entirely — §3.4.

3.8 ⭐⭐ Multi-turn degeneration from TWO real compounding causes — how they masked each other

The most expensive diagnosis this project has had, because there were two real causes at once and each partial fix moved the needle enough to look like the answer. Recorded precisely because the first write-up of this section over-attributed it to the quant alone; that was wrong.

Cause 1 (real, upstream): the vLLM qwen3_5_mtp × Gated-DeltaNet bug. Confirmed by two cross-frontier peers and the tracker (vllm#47087 symptom-twin, #43559 fix lineage, #51113 fix): the GDN recurrent state cannot roll back on a partial draft-accept, so speculative decoding corrupts it, worse with context. Architectural — vLLM/SGLang/llama.cpp mainline all shared it. Genuinely fixed enough by moving to vLLM nightly (v0.27.2rc1.dev150+, carries #51113): the operator reported it "significantly better" — this was a real bug, not just an amplifier.

Cause 2 (real, quant): full W4A4 is mildly subpar, per the known gradient. sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4 is full W4A4 — 4-bit activations on attention too, the bottom of the activation-precision ordering already in §1: W4A4 (A4) < W4+FP8 (A8) < W4+bf16 (A16). Not "defective," just lowest-fidelity; on top of Cause 1 it degenerated ~15-20% of real multi-turn generations. The FP8-attention mixed build (qwen38-27b-uncensored-nvfp4-mixed, same base, same MTP, same nightly) sits a rung up that gradient and is coherent. AEON was purged 2026-08-17 (operator ruled it no-good; re-pullable from HF).

Why it cost days — and the process lessons that stand:

1. Two real causes compound and mask each other. Each mitigation (MTP-off, APC-off, the nightly #51113 fix) partially helped, so each looked like the fix and then failed in real use. When a mitigation "helps but doesn't fix," suspect a second cause rather than a wrong one.
2. Stochastic degeneration (~15-20%) is nearly invisible to a small synthetic probe — a 7-turn run passes ~4 in 5. n=1 "clean" proves nothing; this class needs many runs or the operator's real high-volume use. Three non-fixes were "validated" by a single clean probe here.
3. Isolate the WEIGHTS in parallel with the serving flags, not after. Swapping to a different quant of the same base (AEON→mixed) is what finally separated Cause 2 from Cause 1; doing it earlier would have shortened the hunt. But note it would NOT have found Cause 1 — the vLLM bug was real and needed the nightly.
4. Prefer FP8 attention (the §2 mixed recipe) over full W4A4 for a coherence- sensitive seat. AEON passed every static gate (abliteration 4/4, surface 6/6, a 36k needle, 52% acceptance) and was still the lower-fidelity of the two.

Current primary gen: the mixed FP8-attention build on pinned vLLM nightly with MTP, until the DavidAU Qwen3.8 lands. A W4+bf16 (W4A16) build would be higher fidelity still (§1) at a prefill cost — an option if the mixed build ever proves marginal.

3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn

§3.3 is about losing the head (0% acceptance, silent). This is the opposite and worse failure: the head loads, acceptance looks healthy, single-turn output is perfect — and then it corrupts multi-turn conversations once cumulative context passes ~2,000 tokens. The reply collapses in length and bleeds earlier turns into the current answer (a "describe durian" reply that contained the Krebs-cycle and winter answers from three turns back). Single-turn probes and the acceptance gate (§5) do not catch it — it only appears as accumulated context grows.

Isolated 2026-08-16 (operator-confirmed), each step measured on a fixed 7-turn probe:

• Not the serving gateway, not sampling, not repetition/template. Identical input gateway-vs-direct behaves the same; presence_penalty 1.5/0.5/0.0 all collapse; higher temperature collapses harder; a conversation of unrelated topics collapses at the same ~2k tokens as a repetitive one → it is context- length-driven, not template lock-in.
• Model-independent across every Qwen3.8-27B quant (AEON W4A4, unsloth FP8-attn, our in-house mixed) — so not a quant-brand or scheme artifact.
• DECISIVE: same model + same conversation, MTP OFF → coherent through 4k+ tokens, zero bleed. Toggle it back on → collapse returns. MTP is the cause.

Qwen3.6-27B running the same qwen3_5_mtp method is CLEAN. So the 3.6 MTP head/graft is fine and the 3.8 one is not — suspects: the bf16 graft being subtly wrong for the 3.8 head, or the vLLM qwen3_5_mtp impl diverging at num_speculative_tokens=3. Open upstream question (queried dvalin/bil-smithy 2026-08-17).

Rule: gate MTP on a MULTI-TURN coherence probe, not just single-shot acceptance. Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens for length-collapse and cross-turn bleed.

THE MITIGATION (resolved 2026-08-17): disable prefix caching, keep MTP. The corruption is gated on MTP × prefix-caching together (vllm#43559 / #47194) — with --no-enable-prefix-caching the GDN cache runs in a mode where the buggy partial-accept align-path is inert. Confirmed on our stack: AEON W4A4, MTP on + prefix-caching off → the 7-turn varied series stays coherent through 3.9k tokens, zero bleed, at 104.6 tok/s / 53.6% acceptance — i.e. the FULL MTP speedup back (vs ~half with MTP off), losing only prefix-cache reuse. The gen seat runs this config as of 2026-08-17.

Things that do not work, ruled out: num_speculative_tokens=1 (corruption is depth-independent — reproduces at n=1 and n=2, deterministically probed upstream); switching engine (vLLM / SGLang / llama.cpp mainline all share the GDN-rollback bug — it is architectural). The proper upstream fix (vllm#51113) is in main / v0.27.2rc0 only — not in a stable release, so we hold at APC-off until it lands. Two cross-frontier peers (dvalin/bil-smithy) confirmed the bug class and pointed at the open symptom-twin issue #47087.

3.4 Toolchain version deadlocks

Both directions have burned us, so the resolution is: use llm-compressor / compressed-tensors, not nvidia-modelopt.

• modelopt 0.45 ↔ transformers 5.12: mtq.quantize dies TypeError: issubclass() arg 2 must be a class (modelopt registers transformers' FusedMoE, a function in 5.x, as an nn class).
• modelopt 0.43 doesn't fix it — it drags transformers back to 4.57, which cannot load qwen3_5 at all.
• modelopt's config API also trails the current model families by a version.

3.5 Vision tower and its configs

• Keep the vision tower in ignore (BF16). Only the LLM backbone gets quantized.
• The wrapper-class save drops preprocessor_config.json (and the video one). Without it the seat crash-loops Can't load image processor. Restore from the source — and if the upstream repo omits it, reconstruct it from processor_config.json's image_processor sub-dict.

3.6 Memory and device placement (large models)

• device_map=None/"cpu", never "auto". auto fills GPU0 and OOMs during un-fusing; constraining with max_memory then offloads to the meta device, which cannot be .copy_()d. CPU-resident keeps every tensor real; the sequential pipeline still onloads per-layer to GPU.
• Avoid mmap on /tank. safetensors.safe_open() mmaps a whole shard; on ZFS a 50 GB shard ENOMEMs regardless of free RAM (MAP_SHARED never consults the commit limit). Read with plain read() + load(bytes), one shard cached at a time.
• vm.overcommit_memory=1 on ana-ml2 (durable via playbooks/ana-ml2-overcommit-memory.yaml).

3.9 ⭐⭐ A sharded forward can be silently WRONG — never trust device_map="auto" for activations

Splitting Qwen3.8-27B (Qwen3_5 hybrid) across the two Blackwells with device_map="auto" produces a model that loads clean, reports no error, and computes garbage: the residual stream collapses to exactly zero a couple of layers past the GPU0→GPU1 boundary, and the logits decode to rubbish ('8', '�', 'b'). Every layer below the boundary stays healthy, deterministic, and bit-identical to a single-GPU run — which is what makes it so dangerous. A capture that reads a low layer looks perfectly plausible and is fine; one that reads a high layer is reading zeros, and nothing in the pipeline says so. Measured 2026-08-20 (§9 Cold-Fusion).

Rule: any workload that reads activations — refusal-direction capture, calibration, activation statistics, PPL — must run on ONE device. Sharding is for storage, and it is only safe when you consume the model's final output through an engine that was built for it (vLLM does TP correctly; device_map="auto" in transformers is not the same thing). If it does not fit on one card, shrink the model, not the guarantee: truncating the decoder to N layers is exact for any activation read at a layer < N (a causal stack's layer-N state cannot depend on layers above N), and it is cheap — verified by reproducing the full model's layers 18/20/22/26 bit-for-bit.

Gate it, don't remember it. Assert single-device residency and zero offload before the forward:

dmap = getattr(model, "hf_device_map", {}) or {}
gpus = {str(v) for v in dmap.values()} - {"cpu", "disk"}
offloaded = [k for k, v in dmap.items() if str(v) in ("cpu", "disk")]
if len(gpus) > 1 or offloaded:
    sys.exit("residency gate FAILED — sharded/offloaded forward reads garbage")

3.10 ⭐⭐ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True corrupts retained tensors

On torch 2.12+cu130 / Blackwell, tensors that outlive their allocation come back corrupted with this flag set: captured hidden states carried Inf / NaN / zeros that moved between bit-identical forwards (same input, same weights → a different layer corrupted each time). Unset, the identical forwards are exactly reproducible. Several runbooks recommend this flag for headroom on large loads; for anything that keeps activations it buys corruption.

Two tells that distinguish this from a real numerical blowup, both worth knowing because they generalise: a genuine blowup propagates to later layers and is deterministic. Corruption does neither — downstream layers were finite and consistent, and the affected layer moved run to run. If a "NaN" fails to propagate, stop debugging the math and start debugging memory.

Corollary: do not read output_hidden_states=True off a returned object on a large multi-device load. Take what you need during the forward with a register_forward_pre_hook that clones to CPU immediately — it closes the reuse window and never retains a [B, seq, hidden] tensor per layer, so it is cheaper than the thing it replaces.

3.11 Determinism is a necessary check, not a sufficient one

Both defects above were found by the cheapest possible test — run the same input twice and diff — which no amount of eyeballing plausible-looking numbers would have caught. Add it to any activation-reading pipeline. But note the trap that followed: after fixing the allocator, the run went perfectly "deterministic" because the corrupted layers were now stably zero. Pair the determinism check with a magnitude check (residual norms should grow smoothly with depth; an exact 0.0 mid-stack is impossible) and, where you can, a coherence check (generate 40 tokens and read them).

---

4. Pipeline shape

4.1 Prove the targets before spending GPU time

Enumerate module names from the safetensors index and check your regexes against them: zero overlap between groups, and the union covers every layer you intended. This is free, takes seconds, and catches a mis-scoped regex that would otherwise surface as a mystery quality regression hours later. Reference: services/gen-seat-mixed-quant/validate_targets.py.

4.2 Quantize

Calibration data matters for imatrix_mse + static activation observers. We use /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl (512 chat samples, RP/GM-flavoured — appropriate for our seats). 256 samples @ 2048 tokens ≈ 20 min for a 27B on one Blackwell.

4.3 The mandatory post-steps

Never optional, always in this order, and the last one verifies rather than assumes:

1. Graft model-mtp.safetensors + register its tensors in the output index.
2. Restore preprocessor_config.json / processor_config.json / video_preprocessor_config.json.
3. Re-inject re:^mtp.* into quantization_config.ignore and confirm it is there (§3.3).

Reference implementation: services/gen-seat-mixed-quant/post_quant.py.

4.4 Test on a temp port, never on the live seat

Serve the candidate on an alt port with the live seat's exact flags, run the gate (§5), and only then flip .env. Keep the previous build on disk; rollback is one .env line.

---

5. The acceptance gate — and how measurement lies to you

Speed alone does not justify cutting over a shared seat. Gate on all of: decode tok/s, MTP acceptance, perplexity, a behavioural surface test, and — for an abliterated model — that the abliteration survived.

Three ways the numbers have lied to us. All three produced confident, wrong results.

1. Prefix caching fakes both speed metrics. A fixed prompt returns byte-identical timings run after run; you are measuring cache, not compute. Worse for prefill: a seeded nonce regenerates the previous run's prompts verbatim and reads ~41k tok/s of cache-hit instead of ~5k of real prefill. Use a fresh unseeded nonce per request; never seed a cache-buster.
2. prompt_logprobs are garbage while speculative decoding is on — ~uniform over the vocab (median rank ~10⁵; " Paris" after "The capital of France is" ranked 69698). Perplexity must be measured on a seat served without --speculative-config, on both sides of the comparison.
3. A 0600 .env makes docker compose silently no-op. Without sudo it fails permission denied reading .env, leaves the old container running, and reports success — producing a full page of "benchmark results" that were just the unchanged baseline. Hard-verify the change landed against docker inspect …Config.Cmd.

Re-measure the baseline before believing a target. The 2026-08-15 handoff quoted ~68 tok/s; cache-busted, the incumbent was already doing 80.1 — essentially the target of the work queued against it. Had that not been re-measured, doing nothing would have looked like a 20% win.

Cheap shortcut worth taking first: if a reputable published quant of the same architecture is already on-box (or is a small pull), serve it as a probe and measure it before committing hours to your own. It answers "is this gain even real?" in ten minutes and hands you the recipe.

Harness: services/gen-seat-mixed-quant/bench/ — quickbench.py (decode + acceptance), prefill_bench.py, eval_quality.py (PPL + abliteration), surface_test.py (chat, vision, tools, thinking split, long-context needle, streaming), serve_probe.sh.

---

6. Hardware and co-residency

• ana-ml2 = Blackwell (sm_120), 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths (match_exact=True on sm90) are closed here — do not plan around them.
• irv-ml1 = Ampere (sm_86), 3090 + A6000. No native FP8/FP4 — 4-bit there is a VRAM saving only, not a speed win. Don't port a Blackwell scheme over and expect the throughput.
• GPU co-residency is a zero-sum budget, and a smaller model can break its neighbour. gpu-memory-utilization is a fraction of the whole card, so when new weights are smaller the seat absorbs the slack as extra KV rather than releasing it. That is exactly how a −5.2 GB requant left the co-resident seat 0.18 GiB short and crash-looping. After any requant, re-check both seats' budgets and hand the space back explicitly.

---

7. Superseded claims — do not follow these

Old docs stay for their history, but these specific claims are false now and will cost you a day if followed:

claim	where	status
"Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance"	docs/runbooks/heretic2-nvfp4-mtp-seat.md §landmine 2	SUPERSEDED 2026-08-14. The 0% was the missing re:^mtp.* ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.7–83.2% acceptance, live. Use compressed-tensors.
"Abliteration desyncs the MTP head → uncensored models can't do MTP"	earlier auto-memory	SUPERSEDED 2026-08-14. A modest abliteration preserves MTP (83.7% at bf16). Test MTP on bf16 first to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source.
"NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer"	reference_nvfp4_w4a4_granite_infeasible	NARROWED. True for uniform W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs with FP8 on attention and late layers is fine and is the current default (§2).
"transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 without causal-conv1d; it is precision-driven cancellation and fp32 resolves it"	services/coldfusion-abliteration/README.md, persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md	SUPERSEDED 2026-08-20. Precision was never the variable. The NaN came from multi-GPU sharding and expandable_segments (§3.9, §3.10); fp32 only made it rarer, which is worse than failing. On one GPU with a plain allocator, bf16 is exactly deterministic through all 64 layers and generates coherent prose — at 50 GB and 4.3× the throughput of the 111 GB fp32 it replaced.

---

8. Measured negatives — don't re-chase

• num_speculative_tokens = 3 is optimal on the Qwen3.8-27B seat. Swept: n=2 → 77.1, n=3 → 80.1, n=4 → 78.7, n=5 → 75.9 tok/s. Higher n trades acceptance for draft width and loses. Re-sweep only if the drafter architecture changes.
• Uniform W4A4 — see §7 row 3.
• Dense-VL as the anatomy judge — A/B'd, MoE retained. Don't re-propose.

---

9. Worked examples

Per-model artifacts. Read for how a specific model went, not for the general lessons — those are above, and where the two disagree, this file wins.

artifact	what it is
services/gen-seat-mixed-quant/	current reference. Mixed NVFP4+FP8 on Qwen3.8-27B-Uncensored: scripts, acceptance harness, raw measurements.
stacks/gen-seat/README.md	the live gen seat (7 LiteLLM aliases)
stacks/meromero-charrp/README.md	Gemma-4 seat — the tool-call/reasoning-parser trap (a parser default that returns null content for all prose)
services/heretic2-nvfp4-quant/	modelopt-format MTP seat — historical; see §7 before following it
tools/mistral-small4-nvfp4/	MoE + native-convert path; source of §3.6
docs/pfi/recommended-model-settings.md	serve-time sampler/flag defaults (not quant)

A new model just dropped and needs requanting? §1 → §2 → §4 → §5. Skim §3 first; it is the part that costs hours.