# Model quantization playbook — the lessons that keep costing us hours **Read this before starting any new quant.** Not the per-model runbooks — those are worked examples of a *specific* model at a *specific* point in time, and several carry claims that are now false (see §7). This file owns the **transferable** part: what recurs regardless of which model dropped this week. Written 2026-08-15, after the fourth quant in five weeks re-discovered the third-known instance of the same loader-class bug. Scope: NVFP4 / FP8 / mixed-precision on the Blackwell boxes (ana-ml2), vLLM-served. Ampere (irv-ml1) has no native FP4/FP8 — see §6. **Maintenance rule.** When a quant teaches you something *model-agnostic*, it lands here and the per-model README links up. When it's model-specific (this checkpoint's odd tensor names, this finetune's missing config), it stays in the per-model artifact. If you find yourself writing a "Gotchas" section that repeats §3, you are re-litigating — add the delta here instead. --- ## 1. The 60-second decision: which scheme On Blackwell + vLLM, for a dense-or-hybrid VL model you intend to serve at long context: | want | scheme | notes | |---|---|---| | **default, best speed/accuracy** | **mixed: NVFP4 W4A4 bulk MLPs + FP8 W8A8 attention/`lm_head`/last-8-layer MLPs** | the current answer. §2. | | max fidelity, don't care about prefill | NVFP4 **W4A16** (weight-only) | forces the **Marlin** kernel — ~half the prefill of native FP4 | | small model, VRAM is free | FP8 **W8A8** | safe and simple; 2× the weight bytes of 4-bit | | — | ~~"W4A8" = NVFP4 weights + FP8 activations~~ | **DOES NOT EXIST.** §3.1 | **Measured on Qwen3.8-27B (2026-08-15), W4A16 → mixed:** decode +18%, prefill **+78–98%**, MTP acceptance unchanged, perplexity +1.7%, weights −19%. Note the shape of that: **decode barely moves, prefill nearly doubles.** Decode at batch-1 is memory-bandwidth-bound and the weights are 4-bit under either scheme, so there is little to win; prefill is compute-bound, which is where native FP4 tensor cores replace the Marlin dequantize-to-BF16 path. If someone promises you a big *decode* win from a scheme change, be skeptical — and go measure §5 before believing it. **The accuracy cost is real and is paid on purpose.** Operator ruling 2026-08-15: the ~1.7% perplexity is an acceptable price for the speed. Settled — don't re-litigate. For correct attribution: it is the **activation**-quantization cost (A4/A8 vs BF16 activations), *not* an MTP cost. Turning MTP off does not recover it; only reverting the quant does. --- ## 2. The reference recipe (mixed-precision) Lifted from `unsloth/Qwen3.8-27B-NVFP4` and replicated in-house. **Prefer replicating a published recipe from a reputable quantizer over inventing one** — they have already paid for the sensitivity analysis. | group | scheme | targets | |---|---|---| | `group_0` | FP8 W8A8 — channel weights (static) + per-token dynamic activations | `self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`, `lm_head`, **the last 8 layers' MLPs** | | `group_1` | NVFP4 W4A4 — `tensor_group` gsize 16, fp8 scales, `imatrix_mse` weights, `dynamic:"local"` activations | **all remaining** MLP `{gate,up,down}_proj` | | kv cache | FP8 static tensor | | | ignore | vision tower, `linear_attn.{norm,in_proj_a,in_proj_b}`, `re:^mtp.*` | | Three things in there are load-bearing and easy to drop: - **Late layers stay FP8.** Holding the last ~8 layers' MLPs (and `lm_head`) at 8-bit is the accuracy-preservation trick — late layers are the sensitive ones. Uniform W4A4 is what collapses. - **`imatrix_mse` on the W4A4 weights**, not `memoryless_minmax`. Importance-weighted; needs calibration data. - **Group targets must be non-overlapping.** Do not let `group_1`'s `.*mlp\..*` also match the late layers and rely on group precedence to sort it out. Enumerate the early layers explicitly (`re:.*layers\.([0-9]|[1-4][0-9]|5[0-5])\.mlp\.…`) and **prove it** with a dry run (§4.1). **Toolchain:** `pip install llmcompressor` into stock `vllm/vllm-openai:latest` gives llmcompressor 0.13 + compressed-tensors 0.18 without disturbing torch/transformers. **Avoid nvidia-modelopt** — see §3.4. --- ## 3. The recurring landmines Ordered by how much time each has cost. Every one of these has bitten more than once. ### 3.1 "W4A8" is not a servable shape vLLM's compressed-tensors dispatcher (`compressed_tensors.py:704-713`) accepts NVFP4 weights with **exactly two** activation settings: | `input_activations` | result | |---|---| | `None` | W4A16 — and it **forces the Marlin kernel** (`kernels/linear/__init__.py:881-883`) | | NVFP4 | W4A4, native | Anything else — **FP8 included** — raises at load: ``` ValueError: For NVFP4 weights, input quantization must also be NVFP4 format, None for NVFP4A16 ``` `CompressedTensorsW4A8Fp8` exists but is **INT4** weights (`W4A8_SUPPORTED_TYPES_MAP = {4: int4}`) gated on `_check_scheme_supported(90, match_exact=True)` — Hopper-exact, so on Blackwell (sm_120) it is closed twice over. **FP8 enters per-layer-group, never as activations on NVFP4 weights.** *Cost: one queued task written against an impossible scheme.* ### 3.2 Wrong loader class → silent weight-load failure **Rediscovered three times.** Load the model through the class vLLM actually serves — the `…ForConditionalGeneration` / `…ForImageTextToText` **wrapper**, never `AutoModelForCausalLM`. `AutoModelForCausalLM` resolves a VL config to the text-only inner class and saves a **flat** config with `model.layers.*` keys. vLLM's weight mapper wants `model.language_model.*` (+ `model.visual.*`). The mismatch does not error — **every layer silently fails to load** and you get `!!!!` gibberish, or an engine that rejects the checkpoint outright. *Bit: heretic2 (gibberish), Dark-Scarlett (both vLLM and SGLang refused the checkpoint), and the 2026-08 rounds.* ### 3.3 The MTP head — three separate ways to lose it Speculative decoding is a large fraction of the seat's throughput. It fails **silently**: the model serves fine, just at 0% acceptance. 1. **The wrapper class does not instantiate `mtp.*`,** so the quant drops it. Post-quant you must graft the BF16 `model-mtp.safetensors` back and register its tensors in the output index. 2. **`re:^mtp.*` must be in `quantization_config.ignore`** — else vLLM loads the grafted BF16 head as though quantized, it comes up **uninitialised**, and acceptance is 0%. 3. **⭐ llm-compressor PRUNES `ignore` entries that matched no module at quant time.** Since the wrapper never loaded `mtp.*`, the entry matches nothing and is **silently deleted from the saved config — even though you put it in the recipe.** So it must be re-injected *after* the graft, and then **verified, not assumed.** *Cost: three rounds. The verify step caught it live on the third.* There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not propagate modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix (needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4. ### 3.8 ⭐⭐ Multi-turn degeneration from TWO real compounding causes — how they masked each other The most expensive diagnosis this project has had, because there were **two real causes at once** and each partial fix moved the needle enough to look like *the* answer. Recorded precisely because the first write-up of this section over-attributed it to the quant alone; that was wrong. **Cause 1 (real, upstream): the vLLM `qwen3_5_mtp` × Gated-DeltaNet bug.** Confirmed by two cross-frontier peers and the tracker (vllm#47087 symptom-twin, #43559 fix lineage, #51113 fix): the GDN recurrent state cannot roll back on a partial draft-accept, so speculative decoding corrupts it, worse with context. Architectural — vLLM/SGLang/llama.cpp mainline all shared it. **Genuinely fixed enough** by moving to vLLM **nightly** (`v0.27.2rc1.dev150+`, carries #51113): the operator reported it "significantly better" — this was a real bug, not just an amplifier. **Cause 2 (real, quant): full W4A4 is mildly subpar, per the known gradient.** `sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4` is **full** W4A4 — 4-bit *activations* on attention too, the bottom of the activation-precision ordering already in §1: **W4A4 (A4) < W4+FP8 (A8) < W4+bf16 (A16)**. Not "defective," just lowest-fidelity; on top of Cause 1 it degenerated ~15-20% of real multi-turn generations. The FP8-attention **mixed** build (`qwen38-27b-uncensored-nvfp4-mixed`, same base, same MTP, same nightly) sits a rung up that gradient and is coherent. AEON was purged 2026-08-17 (operator ruled it no-good; re-pullable from HF). **Why it cost days — and the process lessons that stand:** 1. **Two real causes compound and mask each other.** Each mitigation (MTP-off, APC-off, the nightly #51113 fix) partially helped, so each looked like the fix and then failed in real use. When a mitigation "helps but doesn't fix," suspect a *second* cause rather than a wrong one. 2. **Stochastic degeneration (~15-20%) is nearly invisible to a small synthetic probe** — a 7-turn run passes ~4 in 5. n=1 "clean" proves nothing; this class needs many runs or the operator's real high-volume use. Three non-fixes were "validated" by a single clean probe here. 3. **Isolate the WEIGHTS in parallel with the serving flags, not after.** Swapping to a different quant of the same base (AEON→mixed) is what finally separated Cause 2 from Cause 1; doing it earlier would have shortened the hunt. But note it would NOT have found Cause 1 — the vLLM bug was real and needed the nightly. 4. **Prefer FP8 attention (the §2 mixed recipe) over full W4A4** for a coherence- sensitive seat. AEON passed every static gate (abliteration 4/4, surface 6/6, a 36k needle, 52% acceptance) and was still the lower-fidelity of the two. Current primary gen: the mixed FP8-attention build on pinned vLLM nightly with MTP, until the DavidAU Qwen3.8 lands. A W4+bf16 (W4A16) build would be higher fidelity still (§1) at a prefill cost — an option if the mixed build ever proves marginal. ### 3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn §3.3 is about *losing* the head (0% acceptance, silent). This is the opposite and worse failure: the head loads, acceptance looks healthy, single-turn output is perfect — and then it **corrupts multi-turn conversations** once cumulative context passes **~2,000 tokens**. The reply collapses in length *and* bleeds earlier turns into the current answer (a "describe durian" reply that contained the Krebs-cycle and winter answers from three turns back). Single-turn probes and the acceptance gate (§5) **do not catch it** — it only appears as accumulated context grows. Isolated 2026-08-16 (operator-confirmed), each step measured on a fixed 7-turn probe: - **Not the serving gateway, not sampling, not repetition/template.** Identical input gateway-vs-direct behaves the same; presence_penalty 1.5/0.5/0.0 all collapse; higher temperature collapses harder; a conversation of *unrelated* topics collapses at the same ~2k tokens as a repetitive one → it is context- length-driven, not template lock-in. - **Model-independent across every Qwen3.8-27B quant** (AEON W4A4, unsloth FP8-attn, our in-house mixed) — so not a quant-brand or scheme artifact. - **DECISIVE: same model + same conversation, MTP OFF → coherent through 4k+ tokens, zero bleed.** Toggle it back on → collapse returns. MTP is the cause. **Qwen3.6-27B running the same `qwen3_5_mtp` method is CLEAN.** So the 3.6 MTP head/graft is fine and the 3.8 one is not — suspects: the bf16 graft being subtly wrong for the 3.8 head, or the vLLM `qwen3_5_mtp` impl diverging at `num_speculative_tokens=3`. Open upstream question (queried dvalin/bil-smithy 2026-08-17). **Rule: gate MTP on a MULTI-TURN coherence probe, not just single-shot acceptance.** Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens for length-collapse and cross-turn bleed. **THE MITIGATION (resolved 2026-08-17): disable prefix caching, keep MTP.** The corruption is gated on MTP × prefix-caching *together* (vllm#43559 / #47194) — with `--no-enable-prefix-caching` the GDN cache runs in a mode where the buggy partial-accept align-path is inert. Confirmed on our stack: AEON W4A4, MTP on + prefix-caching off → the 7-turn varied series stays coherent through 3.9k tokens, zero bleed, at **104.6 tok/s / 53.6% acceptance** — i.e. the FULL MTP speedup back (vs ~half with MTP off), losing only prefix-cache reuse. The gen seat runs this config as of 2026-08-17. Things that do **not** work, ruled out: `num_speculative_tokens=1` (corruption is depth-independent — reproduces at n=1 and n=2, deterministically probed upstream); switching engine (vLLM / SGLang / llama.cpp mainline all share the GDN-rollback bug — it is architectural). The proper upstream fix (vllm#51113) is in `main` / `v0.27.2rc0` only — not in a stable release, so we hold at APC-off until it lands. Two cross-frontier peers (dvalin/bil-smithy) confirmed the bug class and pointed at the open symptom-twin issue #47087. ### 3.4 Toolchain version deadlocks Both directions have burned us, so the resolution is: **use llm-compressor / compressed-tensors, not nvidia-modelopt.** - modelopt **0.45** ↔ transformers 5.12: `mtq.quantize` dies `TypeError: issubclass() arg 2 must be a class` (modelopt registers transformers' `FusedMoE`, a *function* in 5.x, as an nn class). - modelopt **0.43** doesn't fix it — it drags transformers back to 4.57, which cannot load `qwen3_5` at all. - modelopt's config API also trails the current model families by a version. ### 3.5 Vision tower and its configs - Keep the **vision tower in `ignore`** (BF16). Only the LLM backbone gets quantized. - The wrapper-class save **drops `preprocessor_config.json`** (and the video one). Without it the seat crash-loops `Can't load image processor`. Restore from the source — and if the upstream repo omits it, **reconstruct it from `processor_config.json`'s `image_processor` sub-dict**. ### 3.6 Memory and device placement (large models) - **`device_map=None`/`"cpu"`, never `"auto"`.** `auto` fills GPU0 and OOMs during un-fusing; constraining with `max_memory` then offloads to the *meta* device, which cannot be `.copy_()`d. CPU-resident keeps every tensor real; the sequential pipeline still onloads per-layer to GPU. - **Avoid mmap on `/tank`.** `safetensors.safe_open()` mmaps a whole shard; on ZFS a 50 GB shard ENOMEMs regardless of free RAM (MAP_SHARED never consults the commit limit). Read with plain `read()` + `load(bytes)`, one shard cached at a time. - **`vm.overcommit_memory=1`** on ana-ml2 (durable via `playbooks/ana-ml2-overcommit-memory.yaml`). --- ## 4. Pipeline shape ### 4.1 Prove the targets before spending GPU time Enumerate module names from the safetensors index and check your regexes against them: **zero overlap between groups, and the union covers every layer you intended.** This is free, takes seconds, and catches a mis-scoped regex that would otherwise surface as a mystery quality regression hours later. Reference: `services/gen-seat-mixed-quant/validate_targets.py`. ### 4.2 Quantize Calibration data matters for `imatrix_mse` + static activation observers. We use `/tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl` (512 chat samples, RP/GM-flavoured — appropriate for our seats). 256 samples @ 2048 tokens ≈ 20 min for a 27B on one Blackwell. ### 4.3 The mandatory post-steps Never optional, always in this order, and the last one **verifies rather than assumes**: 1. Graft `model-mtp.safetensors` + register its tensors in the output index. 2. Restore `preprocessor_config.json` / `processor_config.json` / `video_preprocessor_config.json`. 3. **Re-inject `re:^mtp.*` into `quantization_config.ignore` and confirm it is there** (§3.3). Reference implementation: `services/gen-seat-mixed-quant/post_quant.py`. ### 4.4 Test on a temp port, never on the live seat Serve the candidate on an alt port with the live seat's **exact** flags, run the gate (§5), and only then flip `.env`. Keep the previous build on disk; rollback is one `.env` line. --- ## 5. The acceptance gate — and how measurement lies to you Speed alone does not justify cutting over a shared seat. Gate on **all** of: decode tok/s, MTP acceptance, perplexity, a behavioural surface test, and — for an abliterated model — that the abliteration survived. **Three ways the numbers have lied to us. All three produced confident, wrong results.** 1. **Prefix caching fakes both speed metrics.** A fixed prompt returns byte-identical timings run after run; you are measuring cache, not compute. Worse for prefill: a *seeded* nonce regenerates the previous run's prompts verbatim and reads **~41k tok/s of cache-hit instead of ~5k of real prefill**. Use a fresh unseeded nonce per request; never seed a cache-buster. 2. **`prompt_logprobs` are garbage while speculative decoding is on** — ~uniform over the vocab (median rank ~10⁵; " Paris" after "The capital of France is" ranked 69698). **Perplexity must be measured on a seat served without `--speculative-config`,** on both sides of the comparison. 3. **A 0600 `.env` makes `docker compose` silently no-op.** Without `sudo` it fails `permission denied` reading `.env`, **leaves the old container running**, and reports success — producing a full page of "benchmark results" that were just the unchanged baseline. **Hard-verify the change landed against `docker inspect …Config.Cmd`.** **Re-measure the baseline before believing a target.** The 2026-08-15 handoff quoted ~68 tok/s; cache-busted, the incumbent was already doing 80.1 — essentially the *target* of the work queued against it. Had that not been re-measured, doing nothing would have looked like a 20% win. **Cheap shortcut worth taking first:** if a reputable published quant of the same architecture is already on-box (or is a small pull), **serve it as a probe and measure it** before committing hours to your own. It answers "is this gain even real?" in ten minutes *and* hands you the recipe. Harness: `services/gen-seat-mixed-quant/bench/` — `quickbench.py` (decode + acceptance), `prefill_bench.py`, `eval_quality.py` (PPL + abliteration), `surface_test.py` (chat, vision, tools, thinking split, long-context needle, streaming), `serve_probe.sh`. --- ## 6. Hardware and co-residency - **ana-ml2 = Blackwell (sm_120)**, 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths (`match_exact=True` on sm90) are **closed** here — do not plan around them. - **irv-ml1 = Ampere (sm_86)**, 3090 + A6000. **No native FP8/FP4** — 4-bit there is a VRAM saving only, not a speed win. Don't port a Blackwell scheme over and expect the throughput. - **GPU co-residency is a zero-sum budget, and a *smaller* model can break its neighbour.** `gpu-memory-utilization` is a fraction of the *whole card*, so when new weights are smaller the seat absorbs the slack as extra KV rather than releasing it. That is exactly how a −5.2 GB requant left the co-resident seat **0.18 GiB** short and crash-looping. **After any requant, re-check both seats' budgets** and hand the space back explicitly. --- ## 7. Superseded claims — do not follow these Old docs stay for their history, but these specific claims are **false now** and will cost you a day if followed: | claim | where | status | |---|---|---| | "Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance" | `docs/runbooks/heretic2-nvfp4-mtp-seat.md` §landmine 2 | **SUPERSEDED 2026-08-14.** The 0% was the missing `re:^mtp.*` ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.7–83.2% acceptance, live. Use compressed-tensors. | | "Abliteration desyncs the MTP head → uncensored models can't do MTP" | earlier auto-memory | **SUPERSEDED 2026-08-14.** A modest abliteration preserves MTP (83.7% at bf16). Test MTP on **bf16 first** to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source. | | "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). | --- ## 8. Measured negatives — don't re-chase - **`num_speculative_tokens` = 3 is optimal** on the Qwen3.8-27B seat. Swept: n=2 → 77.1, **n=3 → 80.1**, n=4 → 78.7, n=5 → 75.9 tok/s. Higher n trades acceptance for draft width and loses. Re-sweep only if the drafter architecture changes. - **Uniform W4A4** — see §7 row 3. - **Dense-VL as the anatomy judge** — A/B'd, MoE retained. Don't re-propose. --- ## 9. Worked examples Per-model artifacts. Read for *how a specific model went*, not for the general lessons — those are above, and where the two disagree, **this file wins**. | artifact | what it is | |---|---| | `services/gen-seat-mixed-quant/` | **current reference.** Mixed NVFP4+FP8 on Qwen3.8-27B-Uncensored: scripts, acceptance harness, raw measurements. | | `stacks/gen-seat/README.md` | the live `gen` seat (7 LiteLLM aliases) | | `stacks/meromero-charrp/README.md` | Gemma-4 seat — the **tool-call/reasoning-parser** trap (a parser default that returns null `content` for all prose) | | `services/heretic2-nvfp4-quant/` | modelopt-format MTP seat — historical; see §7 before following it | | `tools/mistral-small4-nvfp4/` | MoE + native-convert path; source of §3.6 | | `docs/pfi/recommended-model-settings.md` | serve-time sampler/flag defaults (not quant) | **A new model just dropped and needs requanting?** §1 → §2 → §4 → §5. Skim §3 first; it is the part that costs hours.