esh-pfi-infrastructure/docs/pfi/model-quantization-playbook.md

# Model quantization playbook — the lessons that keep costing us hours

**Read this before starting any new quant.** Not the per-model runbooks — those are worked
examples of a *specific* model at a *specific* point in time, and several carry claims that are
now false (see §7). This file owns the **transferable** part: what recurs regardless of which
model dropped this week.

Written 2026-08-15, after the fourth quant in five weeks re-discovered the third-known instance
of the same loader-class bug. Scope: NVFP4 / FP8 / mixed-precision on the Blackwell boxes
(ana-ml2), vLLM-served. Ampere (irv-ml1) has no native FP4/FP8 — see §6.

**Maintenance rule.** When a quant teaches you something *model-agnostic*, it lands here and the
per-model README links up. When it's model-specific (this checkpoint's odd tensor names, this
finetune's missing config), it stays in the per-model artifact. If you find yourself writing a
"Gotchas" section that repeats §3, you are re-litigating — add the delta here instead.

---

## 1. The 60-second decision: which scheme

On Blackwell + vLLM, for a dense-or-hybrid VL model you intend to serve at long context:

| want | scheme | notes |
|---|---|---|
| **default, best speed/accuracy** | **mixed: NVFP4 W4A4 bulk MLPs + FP8 W8A8 attention/`lm_head`/last-8-layer MLPs** | the current answer. §2. |
| max fidelity, don't care about prefill | NVFP4 **W4A16** (weight-only) | forces the **Marlin** kernel — ~half the prefill of native FP4 |
| small model, VRAM is free | FP8 **W8A8** | safe and simple; 2× the weight bytes of 4-bit |
| — | ~~"W4A8" = NVFP4 weights + FP8 activations~~ | **DOES NOT EXIST.** §3.1 |

**Measured on Qwen3.8-27B (2026-08-15), W4A16 → mixed:** decode +18%, prefill **+78–98%**,
MTP acceptance unchanged, perplexity +1.7%, weights −19%.

Note the shape of that: **decode barely moves, prefill nearly doubles.** Decode at batch-1 is
memory-bandwidth-bound and the weights are 4-bit under either scheme, so there is little to win;
prefill is compute-bound, which is where native FP4 tensor cores replace the Marlin
dequantize-to-BF16 path. If someone promises you a big *decode* win from a scheme change, be
skeptical — and go measure §5 before believing it.

**The accuracy cost is real and is paid on purpose.** Operator ruling 2026-08-15: the ~1.7%
perplexity is an acceptable price for the speed. Settled — don't re-litigate. For correct
attribution: it is the **activation**-quantization cost (A4/A8 vs BF16 activations), *not* an MTP
cost. Turning MTP off does not recover it; only reverting the quant does.

---

## 2. The reference recipe (mixed-precision)

Lifted from `unsloth/Qwen3.8-27B-NVFP4` and replicated in-house. **Prefer replicating a published
recipe from a reputable quantizer over inventing one** — they have already paid for the
sensitivity analysis.

| group | scheme | targets |
|---|---|---|
| `group_0` | FP8 W8A8 — channel weights (static) + per-token dynamic activations | `self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`, `lm_head`, **the last 8 layers' MLPs** |
| `group_1` | NVFP4 W4A4 — `tensor_group` gsize 16, fp8 scales, `imatrix_mse` weights, `dynamic:"local"` activations | **all remaining** MLP `{gate,up,down}_proj` |
| kv cache | FP8 static tensor | |
| ignore | vision tower, `linear_attn.{norm,in_proj_a,in_proj_b}`, `re:^mtp.*` | |

Three things in there are load-bearing and easy to drop:

- **Late layers stay FP8.** Holding the last ~8 layers' MLPs (and `lm_head`) at 8-bit is the
  accuracy-preservation trick — late layers are the sensitive ones. Uniform W4A4 is what collapses.
- **`imatrix_mse` on the W4A4 weights**, not `memoryless_minmax`. Importance-weighted; needs
  calibration data.
- **Group targets must be non-overlapping.** Do not let `group_1`'s `.*mlp\..*` also match the
  late layers and rely on group precedence to sort it out. Enumerate the early layers explicitly
  (`re:.*layers\.([0-9]|[1-4][0-9]|5[0-5])\.mlp\.…`) and **prove it** with a dry run (§4.1).

**Toolchain:** `pip install llmcompressor` into stock `vllm/vllm-openai:latest` gives
llmcompressor 0.13 + compressed-tensors 0.18 without disturbing torch/transformers.
**Avoid nvidia-modelopt** — see §3.4.

---

## 3. The recurring landmines

Ordered by how much time each has cost. Every one of these has bitten more than once.

### 3.1 "W4A8" is not a servable shape

vLLM's compressed-tensors dispatcher (`compressed_tensors.py:704-713`) accepts NVFP4 weights with
**exactly two** activation settings:

| `input_activations` | result |
|---|---|
| `None` | W4A16 — and it **forces the Marlin kernel** (`kernels/linear/__init__.py:881-883`) |
| NVFP4 | W4A4, native |

Anything else — **FP8 included** — raises at load:

```
ValueError: For NVFP4 weights, input quantization must also be NVFP4 format, None for NVFP4A16
```

`CompressedTensorsW4A8Fp8` exists but is **INT4** weights (`W4A8_SUPPORTED_TYPES_MAP = {4: int4}`)
gated on `_check_scheme_supported(90, match_exact=True)` — Hopper-exact, so on Blackwell (sm_120)
it is closed twice over. **FP8 enters per-layer-group, never as activations on NVFP4 weights.**

*Cost: one queued task written against an impossible scheme.*

### 3.2 Wrong loader class → silent weight-load failure

**Rediscovered three times.** Load the model through the class vLLM actually serves — the
`…ForConditionalGeneration` / `…ForImageTextToText` **wrapper**, never `AutoModelForCausalLM`.

`AutoModelForCausalLM` resolves a VL config to the text-only inner class and saves a **flat**
config with `model.layers.*` keys. vLLM's weight mapper wants `model.language_model.*` (+
`model.visual.*`). The mismatch does not error — **every layer silently fails to load** and you
get `!!!!` gibberish, or an engine that rejects the checkpoint outright.

*Bit: heretic2 (gibberish), Dark-Scarlett (both vLLM and SGLang refused the checkpoint), and the
2026-08 rounds.*

### 3.3 The MTP head — three separate ways to lose it

Speculative decoding is a large fraction of the seat's throughput. It fails **silently**: the
model serves fine, just at 0% acceptance.

1. **The wrapper class does not instantiate `mtp.*`,** so the quant drops it. Post-quant you must
   graft the BF16 `model-mtp.safetensors` back and register its tensors in the output index.
2. **`re:^mtp.*` must be in `quantization_config.ignore`** — else vLLM loads the grafted BF16 head
   as though quantized, it comes up **uninitialised**, and acceptance is 0%.
3. **⭐ llm-compressor PRUNES `ignore` entries that matched no module at quant time.** Since the
   wrapper never loaded `mtp.*`, the entry matches nothing and is **silently deleted from the
   saved config — even though you put it in the recipe.** So it must be re-injected *after* the
   graft, and then **verified, not assumed.**

*Cost: three rounds. The verify step caught it live on the third.*

There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not propagate
modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix
(needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4.

### 3.8 ⭐⭐ Multi-turn degeneration from TWO real compounding causes — how they masked each other

The most expensive diagnosis this project has had, because there were **two real
causes at once** and each partial fix moved the needle enough to look like *the*
answer. Recorded precisely because the first write-up of this section
over-attributed it to the quant alone; that was wrong.

**Cause 1 (real, upstream): the vLLM `qwen3_5_mtp` × Gated-DeltaNet bug.**
Confirmed by two cross-frontier peers and the tracker (vllm#47087 symptom-twin,
#43559 fix lineage, #51113 fix): the GDN recurrent state cannot roll back on a
partial draft-accept, so speculative decoding corrupts it, worse with context.
Architectural — vLLM/SGLang/llama.cpp mainline all shared it. **Genuinely fixed
enough** by moving to vLLM **nightly** (`v0.27.2rc1.dev150+`, carries #51113):
the operator reported it "significantly better" — this was a real bug, not just
an amplifier.

**Cause 2 (real, quant): full W4A4 is mildly subpar, per the known gradient.**
`sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4` is **full** W4A4 — 4-bit
*activations* on attention too, the bottom of the activation-precision ordering
already in §1: **W4A4 (A4) < W4+FP8 (A8) < W4+bf16 (A16)**. Not "defective," just
lowest-fidelity; on top of Cause 1 it degenerated ~15-20% of real multi-turn
generations. The FP8-attention **mixed** build (`qwen38-27b-uncensored-nvfp4-mixed`,
same base, same MTP, same nightly) sits a rung up that gradient and is coherent.
AEON was purged 2026-08-17 (operator ruled it no-good; re-pullable from HF).

**Why it cost days — and the process lessons that stand:**
1. **Two real causes compound and mask each other.** Each mitigation (MTP-off,
   APC-off, the nightly #51113 fix) partially helped, so each looked like the fix
   and then failed in real use. When a mitigation "helps but doesn't fix," suspect
   a *second* cause rather than a wrong one.
2. **Stochastic degeneration (~15-20%) is nearly invisible to a small synthetic
   probe** — a 7-turn run passes ~4 in 5. n=1 "clean" proves nothing; this class
   needs many runs or the operator's real high-volume use. Three non-fixes were
   "validated" by a single clean probe here.
3. **Isolate the WEIGHTS in parallel with the serving flags, not after.** Swapping
   to a different quant of the same base (AEON→mixed) is what finally separated
   Cause 2 from Cause 1; doing it earlier would have shortened the hunt. But note
   it would NOT have found Cause 1 — the vLLM bug was real and needed the nightly.
4. **Prefer FP8 attention (the §2 mixed recipe) over full W4A4** for a coherence-
   sensitive seat. AEON passed every static gate (abliteration 4/4, surface 6/6, a
   36k needle, 52% acceptance) and was still the lower-fidelity of the two.

Current primary gen: the mixed FP8-attention build on pinned vLLM nightly with
MTP, until the DavidAU Qwen3.8 lands. A W4+bf16 (W4A16) build would be higher
fidelity still (§1) at a prefill cost — an option if the mixed build ever proves
marginal.

### 3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn

§3.3 is about *losing* the head (0% acceptance, silent). This is the opposite and
worse failure: the head loads, acceptance looks healthy, single-turn output is
perfect — and then it **corrupts multi-turn conversations** once cumulative context
passes **~2,000 tokens**. The reply collapses in length *and* bleeds earlier turns
into the current answer (a "describe durian" reply that contained the Krebs-cycle
and winter answers from three turns back). Single-turn probes and the acceptance
gate (§5) **do not catch it** — it only appears as accumulated context grows.

Isolated 2026-08-16 (operator-confirmed), each step measured on a fixed 7-turn probe:

- **Not the serving gateway, not sampling, not repetition/template.** Identical
  input gateway-vs-direct behaves the same; presence_penalty 1.5/0.5/0.0 all
  collapse; higher temperature collapses harder; a conversation of *unrelated*
  topics collapses at the same ~2k tokens as a repetitive one → it is context-
  length-driven, not template lock-in.
- **Model-independent across every Qwen3.8-27B quant** (AEON W4A4, unsloth
  FP8-attn, our in-house mixed) — so not a quant-brand or scheme artifact.
- **DECISIVE: same model + same conversation, MTP OFF → coherent through 4k+
  tokens, zero bleed.** Toggle it back on → collapse returns. MTP is the cause.

**Qwen3.6-27B running the same `qwen3_5_mtp` method is CLEAN.** So the 3.6 MTP
head/graft is fine and the 3.8 one is not — suspects: the bf16 graft being subtly
wrong for the 3.8 head, or the vLLM `qwen3_5_mtp` impl diverging at `num_speculative_tokens=3`.
Open upstream question (queried dvalin/bil-smithy 2026-08-17).

**Rule: gate MTP on a MULTI-TURN coherence probe, not just single-shot acceptance.**
Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens
for length-collapse and cross-turn bleed.

**THE MITIGATION (resolved 2026-08-17): disable prefix caching, keep MTP.** The
corruption is gated on MTP × prefix-caching *together* (vllm#43559 / #47194) — with
`--no-enable-prefix-caching` the GDN cache runs in a mode where the buggy
partial-accept align-path is inert. Confirmed on our stack: AEON W4A4, MTP on +
prefix-caching off → the 7-turn varied series stays coherent through 3.9k tokens,
zero bleed, at **104.6 tok/s / 53.6% acceptance** — i.e. the FULL MTP speedup back
(vs ~half with MTP off), losing only prefix-cache reuse. The gen seat runs this
config as of 2026-08-17.

Things that do **not** work, ruled out: `num_speculative_tokens=1` (corruption is
depth-independent — reproduces at n=1 and n=2, deterministically probed upstream);
switching engine (vLLM / SGLang / llama.cpp mainline all share the GDN-rollback
bug — it is architectural). The proper upstream fix (vllm#51113) is in `main` /
`v0.27.2rc0` only — not in a stable release, so we hold at APC-off until it lands.
Two cross-frontier peers (dvalin/bil-smithy) confirmed the bug class and pointed
at the open symptom-twin issue #47087.

### 3.4 Toolchain version deadlocks

Both directions have burned us, so the resolution is: **use llm-compressor / compressed-tensors,
not nvidia-modelopt.**

- modelopt **0.45** ↔ transformers 5.12: `mtq.quantize` dies `TypeError: issubclass() arg 2 must
  be a class` (modelopt registers transformers' `FusedMoE`, a *function* in 5.x, as an nn class).
- modelopt **0.43** doesn't fix it — it drags transformers back to 4.57, which cannot load
  `qwen3_5` at all.
- modelopt's config API also trails the current model families by a version.

### 3.5 Vision tower and its configs

- Keep the **vision tower in `ignore`** (BF16). Only the LLM backbone gets quantized.
- The wrapper-class save **drops `preprocessor_config.json`** (and the video one). Without it the
  seat crash-loops `Can't load image processor`. Restore from the source — and if the upstream repo
  omits it, **reconstruct it from `processor_config.json`'s `image_processor` sub-dict**.

### 3.6 Memory and device placement (large models)

- **`device_map=None`/`"cpu"`, never `"auto"`.** `auto` fills GPU0 and OOMs during un-fusing;
  constraining with `max_memory` then offloads to the *meta* device, which cannot be `.copy_()`d.
  CPU-resident keeps every tensor real; the sequential pipeline still onloads per-layer to GPU.
- **Avoid mmap on `/tank`.** `safetensors.safe_open()` mmaps a whole shard; on ZFS a 50 GB shard
  ENOMEMs regardless of free RAM (MAP_SHARED never consults the commit limit). Read with plain
  `read()` + `load(bytes)`, one shard cached at a time.
- **`vm.overcommit_memory=1`** on ana-ml2 (durable via `playbooks/ana-ml2-overcommit-memory.yaml`).

---

## 4. Pipeline shape

### 4.1 Prove the targets before spending GPU time

Enumerate module names from the safetensors index and check your regexes against them: **zero
overlap between groups, and the union covers every layer you intended.** This is free, takes
seconds, and catches a mis-scoped regex that would otherwise surface as a mystery quality
regression hours later. Reference: `services/gen-seat-mixed-quant/validate_targets.py`.

### 4.2 Quantize

Calibration data matters for `imatrix_mse` + static activation observers. We use
`/tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl` (512 chat samples, RP/GM-flavoured
— appropriate for our seats). 256 samples @ 2048 tokens ≈ 20 min for a 27B on one Blackwell.

### 4.3 The mandatory post-steps

Never optional, always in this order, and the last one **verifies rather than assumes**:

1. Graft `model-mtp.safetensors` + register its tensors in the output index.
2. Restore `preprocessor_config.json` / `processor_config.json` / `video_preprocessor_config.json`.
3. **Re-inject `re:^mtp.*` into `quantization_config.ignore` and confirm it is there** (§3.3).

Reference implementation: `services/gen-seat-mixed-quant/post_quant.py`.

### 4.4 Test on a temp port, never on the live seat

Serve the candidate on an alt port with the live seat's **exact** flags, run the gate (§5), and
only then flip `.env`. Keep the previous build on disk; rollback is one `.env` line.

---

## 5. The acceptance gate — and how measurement lies to you

Speed alone does not justify cutting over a shared seat. Gate on **all** of: decode tok/s, MTP
acceptance, perplexity, a behavioural surface test, and — for an abliterated model — that the
abliteration survived.

**Three ways the numbers have lied to us. All three produced confident, wrong results.**

1. **Prefix caching fakes both speed metrics.** A fixed prompt returns byte-identical timings run
   after run; you are measuring cache, not compute. Worse for prefill: a *seeded* nonce
   regenerates the previous run's prompts verbatim and reads **~41k tok/s of cache-hit instead of
   ~5k of real prefill**. Use a fresh unseeded nonce per request; never seed a cache-buster.
2. **`prompt_logprobs` are garbage while speculative decoding is on** — ~uniform over the vocab
   (median rank ~10⁵; " Paris" after "The capital of France is" ranked 69698). **Perplexity must be
   measured on a seat served without `--speculative-config`,** on both sides of the comparison.
3. **A 0600 `.env` makes `docker compose` silently no-op.** Without `sudo` it fails
   `permission denied` reading `.env`, **leaves the old container running**, and reports success —
   producing a full page of "benchmark results" that were just the unchanged baseline.
   **Hard-verify the change landed against `docker inspect …Config.Cmd`.**

**Re-measure the baseline before believing a target.** The 2026-08-15 handoff quoted ~68 tok/s;
cache-busted, the incumbent was already doing 80.1 — essentially the *target* of the work queued
against it. Had that not been re-measured, doing nothing would have looked like a 20% win.

**Cheap shortcut worth taking first:** if a reputable published quant of the same architecture is
already on-box (or is a small pull), **serve it as a probe and measure it** before committing
hours to your own. It answers "is this gain even real?" in ten minutes *and* hands you the recipe.

Harness: `services/gen-seat-mixed-quant/bench/` — `quickbench.py` (decode + acceptance),
`prefill_bench.py`, `eval_quality.py` (PPL + abliteration), `surface_test.py` (chat, vision, tools,
thinking split, long-context needle, streaming), `serve_probe.sh`.

---

## 6. Hardware and co-residency

- **ana-ml2 = Blackwell (sm_120)**, 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths
  (`match_exact=True` on sm90) are **closed** here — do not plan around them.
- **irv-ml1 = Ampere (sm_86)**, 3090 + A6000. **No native FP8/FP4** — 4-bit there is a VRAM saving
  only, not a speed win. Don't port a Blackwell scheme over and expect the throughput.
- **GPU co-residency is a zero-sum budget, and a *smaller* model can break its neighbour.**
  `gpu-memory-utilization` is a fraction of the *whole card*, so when new weights are smaller the
  seat absorbs the slack as extra KV rather than releasing it. That is exactly how a −5.2 GB
  requant left the co-resident seat **0.18 GiB** short and crash-looping. **After any requant,
  re-check both seats' budgets** and hand the space back explicitly.

---

## 7. Superseded claims — do not follow these

Old docs stay for their history, but these specific claims are **false now** and will cost you a
day if followed:

| claim | where | status |
|---|---|---|
| "Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance" | `docs/runbooks/heretic2-nvfp4-mtp-seat.md` §landmine 2 | **SUPERSEDED 2026-08-14.** The 0% was the missing `re:^mtp.*` ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.7–83.2% acceptance, live. Use compressed-tensors. |
| "Abliteration desyncs the MTP head → uncensored models can't do MTP" | earlier auto-memory | **SUPERSEDED 2026-08-14.** A modest abliteration preserves MTP (83.7% at bf16). Test MTP on **bf16 first** to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source. |
| "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). |

---

## 8. Measured negatives — don't re-chase

- **`num_speculative_tokens` = 3 is optimal** on the Qwen3.8-27B seat. Swept: n=2 → 77.1,
  **n=3 → 80.1**, n=4 → 78.7, n=5 → 75.9 tok/s. Higher n trades acceptance for draft width and
  loses. Re-sweep only if the drafter architecture changes.
- **Uniform W4A4** — see §7 row 3.
- **Dense-VL as the anatomy judge** — A/B'd, MoE retained. Don't re-propose.

---

## 9. Worked examples

Per-model artifacts. Read for *how a specific model went*, not for the general lessons — those are
above, and where the two disagree, **this file wins**.

| artifact | what it is |
|---|---|
| `services/gen-seat-mixed-quant/` | **current reference.** Mixed NVFP4+FP8 on Qwen3.8-27B-Uncensored: scripts, acceptance harness, raw measurements. |
| `stacks/gen-seat/README.md` | the live `gen` seat (7 LiteLLM aliases) |
| `stacks/meromero-charrp/README.md` | Gemma-4 seat — the **tool-call/reasoning-parser** trap (a parser default that returns null `content` for all prose) |
| `services/heretic2-nvfp4-quant/` | modelopt-format MTP seat — historical; see §7 before following it |
| `tools/mistral-small4-nvfp4/` | MoE + native-convert path; source of §3.6 |
| `docs/pfi/recommended-model-settings.md` | serve-time sampler/flag defaults (not quant) |

**A new model just dropped and needs requanting?** §1 → §2 → §4 → §5. Skim §3 first; it is the
part that costs hours.