esh-pfi-infrastructure/docs/pfi/model-quantization-playbook.md

# Model quantization playbook — the lessons that keep costing us hours

**Read this before starting any new quant.** Not the per-model runbooks — those are worked
examples of a *specific* model at a *specific* point in time, and several carry claims that are
now false (see §7). This file owns the **transferable** part: what recurs regardless of which
model dropped this week.

Written 2026-08-15, after the fourth quant in five weeks re-discovered the third-known instance
of the same loader-class bug. Scope: NVFP4 / FP8 / mixed-precision on the Blackwell boxes
(ana-ml2), vLLM-served. Ampere (irv-ml1) has no native FP4/FP8 — see §6.

**Maintenance rule.** When a quant teaches you something *model-agnostic*, it lands here and the
per-model README links up. When it's model-specific (this checkpoint's odd tensor names, this
finetune's missing config), it stays in the per-model artifact. If you find yourself writing a
"Gotchas" section that repeats §3, you are re-litigating — add the delta here instead.

---

## 1. The 60-second decision: which scheme

On Blackwell + vLLM, for a dense-or-hybrid VL model you intend to serve at long context:

| want | scheme | notes |
|---|---|---|
| **default, best speed/accuracy** | **mixed: NVFP4 W4A4 bulk MLPs + FP8 W8A8 attention/`lm_head`/last-8-layer MLPs** | the current answer. §2. |
| max fidelity, don't care about prefill | NVFP4 **W4A16** (weight-only) | forces the **Marlin** kernel — ~half the prefill of native FP4 |
| small model, VRAM is free | FP8 **W8A8** | safe and simple; 2× the weight bytes of 4-bit |
| — | ~~"W4A8" = NVFP4 weights + FP8 activations~~ | **DOES NOT EXIST.** §3.1 |

**Measured on Qwen3.8-27B (2026-08-15), W4A16 → mixed:** decode +18%, prefill **+78–98%**,
MTP acceptance unchanged, perplexity +1.7%, weights −19%.

Note the shape of that: **decode barely moves, prefill nearly doubles.** Decode at batch-1 is
memory-bandwidth-bound and the weights are 4-bit under either scheme, so there is little to win;
prefill is compute-bound, which is where native FP4 tensor cores replace the Marlin
dequantize-to-BF16 path. If someone promises you a big *decode* win from a scheme change, be
skeptical — and go measure §5 before believing it.

**The accuracy cost is real and is paid on purpose.** Operator ruling 2026-08-15: the ~1.7%
perplexity is an acceptable price for the speed. Settled — don't re-litigate. For correct
attribution: it is the **activation**-quantization cost (A4/A8 vs BF16 activations), *not* an MTP
cost. Turning MTP off does not recover it; only reverting the quant does.

---

## 2. The reference recipe (mixed-precision)

Lifted from `unsloth/Qwen3.8-27B-NVFP4` and replicated in-house. **Prefer replicating a published
recipe from a reputable quantizer over inventing one** — they have already paid for the
sensitivity analysis.

| group | scheme | targets |
|---|---|---|
| `group_0` | FP8 W8A8 — channel weights (static) + per-token dynamic activations | `self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`, `lm_head`, **the last 8 layers' MLPs** |
| `group_1` | NVFP4 W4A4 — `tensor_group` gsize 16, fp8 scales, `imatrix_mse` weights, `dynamic:"local"` activations | **all remaining** MLP `{gate,up,down}_proj` |
| kv cache | FP8 static tensor | |
| ignore | vision tower, `linear_attn.{norm,in_proj_a,in_proj_b}`, `re:^mtp.*` | |

Three things in there are load-bearing and easy to drop:

- **Late layers stay FP8.** Holding the last ~8 layers' MLPs (and `lm_head`) at 8-bit is the
  accuracy-preservation trick — late layers are the sensitive ones. Uniform W4A4 is what collapses.
- **`imatrix_mse` on the W4A4 weights**, not `memoryless_minmax`. Importance-weighted; needs
  calibration data.
- **Group targets must be non-overlapping.** Do not let `group_1`'s `.*mlp\..*` also match the
  late layers and rely on group precedence to sort it out. Enumerate the early layers explicitly
  (`re:.*layers\.([0-9]|[1-4][0-9]|5[0-5])\.mlp\.…`) and **prove it** with a dry run (§4.1).

**Toolchain:** `pip install llmcompressor` into stock `vllm/vllm-openai:latest` gives
llmcompressor 0.13 + compressed-tensors 0.18 without disturbing torch/transformers.
**Avoid nvidia-modelopt** — see §3.4.

---

## 3. The recurring landmines

Ordered by how much time each has cost. Every one of these has bitten more than once.

### 3.1 "W4A8" is not a servable shape

vLLM's compressed-tensors dispatcher (`compressed_tensors.py:704-713`) accepts NVFP4 weights with
**exactly two** activation settings:

| `input_activations` | result |
|---|---|
| `None` | W4A16 — and it **forces the Marlin kernel** (`kernels/linear/__init__.py:881-883`) |
| NVFP4 | W4A4, native |

Anything else — **FP8 included** — raises at load:

```
ValueError: For NVFP4 weights, input quantization must also be NVFP4 format, None for NVFP4A16
```

`CompressedTensorsW4A8Fp8` exists but is **INT4** weights (`W4A8_SUPPORTED_TYPES_MAP = {4: int4}`)
gated on `_check_scheme_supported(90, match_exact=True)` — Hopper-exact, so on Blackwell (sm_120)
it is closed twice over. **FP8 enters per-layer-group, never as activations on NVFP4 weights.**

*Cost: one queued task written against an impossible scheme.*

### 3.2 Wrong loader class → silent weight-load failure

**Rediscovered three times.** Load the model through the class vLLM actually serves — the
`…ForConditionalGeneration` / `…ForImageTextToText` **wrapper**, never `AutoModelForCausalLM`.

`AutoModelForCausalLM` resolves a VL config to the text-only inner class and saves a **flat**
config with `model.layers.*` keys. vLLM's weight mapper wants `model.language_model.*` (+
`model.visual.*`). The mismatch does not error — **every layer silently fails to load** and you
get `!!!!` gibberish, or an engine that rejects the checkpoint outright.

*Bit: heretic2 (gibberish), Dark-Scarlett (both vLLM and SGLang refused the checkpoint), and the
2026-08 rounds.*

### 3.3 The MTP head — three separate ways to lose it

Speculative decoding is a large fraction of the seat's throughput. It fails **silently**: the
model serves fine, just at 0% acceptance.

1. **The wrapper class does not instantiate `mtp.*`,** so the quant drops it. Post-quant you must
   graft the BF16 `model-mtp.safetensors` back and register its tensors in the output index.
2. **`re:^mtp.*` must be in `quantization_config.ignore`** — else vLLM loads the grafted BF16 head
   as though quantized, it comes up **uninitialised**, and acceptance is 0%.
3. **⭐ llm-compressor PRUNES `ignore` entries that matched no module at quant time.** Since the
   wrapper never loaded `mtp.*`, the entry matches nothing and is **silently deleted from the
   saved config — even though you put it in the recipe.** So it must be re-injected *after* the
   graft, and then **verified, not assumed.**

*Cost: three rounds. The verify step caught it live on the third.*

There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not propagate
modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix
(needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4.

### 3.8 ⭐⭐ Multi-turn degeneration from TWO real compounding causes — how they masked each other

The most expensive diagnosis this project has had, because there were **two real
causes at once** and each partial fix moved the needle enough to look like *the*
answer. Recorded precisely because the first write-up of this section
over-attributed it to the quant alone; that was wrong.

**Cause 1 (real, upstream): the vLLM `qwen3_5_mtp` × Gated-DeltaNet bug.**
Confirmed by two cross-frontier peers and the tracker (vllm#47087 symptom-twin,
#43559 fix lineage, #51113 fix): the GDN recurrent state cannot roll back on a
partial draft-accept, so speculative decoding corrupts it, worse with context.
Architectural — vLLM/SGLang/llama.cpp mainline all shared it. **Genuinely fixed
enough** by moving to vLLM **nightly** (`v0.27.2rc1.dev150+`, carries #51113):
the operator reported it "significantly better" — this was a real bug, not just
an amplifier.

**Cause 2 (real, quant): full W4A4 is mildly subpar, per the known gradient.**
`sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4` is **full** W4A4 — 4-bit
*activations* on attention too, the bottom of the activation-precision ordering
already in §1: **W4A4 (A4) < W4+FP8 (A8) < W4+bf16 (A16)**. Not "defective," just
lowest-fidelity; on top of Cause 1 it degenerated ~15-20% of real multi-turn
generations. The FP8-attention **mixed** build (`qwen38-27b-uncensored-nvfp4-mixed`,
same base, same MTP, same nightly) sits a rung up that gradient and is coherent.
AEON was purged 2026-08-17 (operator ruled it no-good; re-pullable from HF).

**Why it cost days — and the process lessons that stand:**
1. **Two real causes compound and mask each other.** Each mitigation (MTP-off,
   APC-off, the nightly #51113 fix) partially helped, so each looked like the fix
   and then failed in real use. When a mitigation "helps but doesn't fix," suspect
   a *second* cause rather than a wrong one.
2. **Stochastic degeneration (~15-20%) is nearly invisible to a small synthetic
   probe** — a 7-turn run passes ~4 in 5. n=1 "clean" proves nothing; this class
   needs many runs or the operator's real high-volume use. Three non-fixes were
   "validated" by a single clean probe here.
3. **Isolate the WEIGHTS in parallel with the serving flags, not after.** Swapping
   to a different quant of the same base (AEON→mixed) is what finally separated
   Cause 2 from Cause 1; doing it earlier would have shortened the hunt. But note
   it would NOT have found Cause 1 — the vLLM bug was real and needed the nightly.
4. **Prefer FP8 attention (the §2 mixed recipe) over full W4A4** for a coherence-
   sensitive seat. AEON passed every static gate (abliteration 4/4, surface 6/6, a
   36k needle, 52% acceptance) and was still the lower-fidelity of the two.

Current primary gen: the mixed FP8-attention build on pinned vLLM nightly with
MTP, until the DavidAU Qwen3.8 lands. A W4+bf16 (W4A16) build would be higher
fidelity still (§1) at a prefill cost — an option if the mixed build ever proves
marginal.

### 3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn

§3.3 is about *losing* the head (0% acceptance, silent). This is the opposite and
worse failure: the head loads, acceptance looks healthy, single-turn output is
perfect — and then it **corrupts multi-turn conversations** once cumulative context
passes **~2,000 tokens**. The reply collapses in length *and* bleeds earlier turns
into the current answer (a "describe durian" reply that contained the Krebs-cycle
and winter answers from three turns back). Single-turn probes and the acceptance
gate (§5) **do not catch it** — it only appears as accumulated context grows.

Isolated 2026-08-16 (operator-confirmed), each step measured on a fixed 7-turn probe:

- **Not the serving gateway, not sampling, not repetition/template.** Identical
  input gateway-vs-direct behaves the same; presence_penalty 1.5/0.5/0.0 all
  collapse; higher temperature collapses harder; a conversation of *unrelated*
  topics collapses at the same ~2k tokens as a repetitive one → it is context-
  length-driven, not template lock-in.
- **Model-independent across every Qwen3.8-27B quant** (AEON W4A4, unsloth
  FP8-attn, our in-house mixed) — so not a quant-brand or scheme artifact.
- **DECISIVE: same model + same conversation, MTP OFF → coherent through 4k+
  tokens, zero bleed.** Toggle it back on → collapse returns. MTP is the cause.

**Qwen3.6-27B running the same `qwen3_5_mtp` method is CLEAN.** So the 3.6 MTP
head/graft is fine and the 3.8 one is not — suspects: the bf16 graft being subtly
wrong for the 3.8 head, or the vLLM `qwen3_5_mtp` impl diverging at `num_speculative_tokens=3`.
Open upstream question (queried dvalin/bil-smithy 2026-08-17).

**Rule: gate MTP on a MULTI-TURN coherence probe, not just single-shot acceptance.**
Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens
for length-collapse and cross-turn bleed.

**THE MITIGATION (resolved 2026-08-17): disable prefix caching, keep MTP.** The
corruption is gated on MTP × prefix-caching *together* (vllm#43559 / #47194) — with
`--no-enable-prefix-caching` the GDN cache runs in a mode where the buggy
partial-accept align-path is inert. Confirmed on our stack: AEON W4A4, MTP on +
prefix-caching off → the 7-turn varied series stays coherent through 3.9k tokens,
zero bleed, at **104.6 tok/s / 53.6% acceptance** — i.e. the FULL MTP speedup back
(vs ~half with MTP off), losing only prefix-cache reuse. The gen seat runs this
config as of 2026-08-17.

Things that do **not** work, ruled out: `num_speculative_tokens=1` (corruption is
depth-independent — reproduces at n=1 and n=2, deterministically probed upstream);
switching engine (vLLM / SGLang / llama.cpp mainline all share the GDN-rollback
bug — it is architectural). The proper upstream fix (vllm#51113) is in `main` /
`v0.27.2rc0` only — not in a stable release, so we hold at APC-off until it lands.
Two cross-frontier peers (dvalin/bil-smithy) confirmed the bug class and pointed
at the open symptom-twin issue #47087.

### 3.15 ⭐⭐ Fused 3-D MoE experts are INVISIBLE to a `targets=["Linear"]` recipe

**Symptom: none.** The quant completes, the artifact loads, and 88.5% of the
model is still BF16. Nothing warns you.

Modern MoE checkpoints store each layer's experts as **two fused 3-D
`nn.Parameter` tensors**, not as N `nn.Linear` modules. Gemma-4 26B-A4B:

    model.language_model.layers.N.experts.gate_up_proj   BF16 [128, 1408, 2816]
    model.language_model.layers.N.experts.down_proj      BF16 [128, 2816,  704]

Note the **absent `.weight` suffix** — that is the tell. `mlp.down_proj.weight`
is an `nn.Linear`; `experts.down_proj` is a bare parameter.

Measured on that checkpoint, recipe targeting `["Linear"]`:

    Linear modules   427
    WILL quantize    205   (experts: 0)     <- 22.84 B params untouched

**This is the same defect that killed QLoRA on this architecture** —
`bitsandbytes` 4-bit replacement also walks `nn.Linear` modules and also
silently skipped the experts. Two different tools, one blind spot, because the
blind spot is in the *checkpoint layout*, not the tool.

**The fix** (llm-compressor ≥ 0.12):

```python
from llmcompressor.modeling.moe.linearize import linearize_moe
model = SomeForConditionalGeneration.from_pretrained(...)
linearize_moe(model)          # BEFORE building the recipe
```

    Linear modules   11947
    WILL quantize    11725  (experts: 11520)   # 30 layers x 128 x 3 proj

`linearize_moe` unfuses the 3-D parameters into per-expert
`experts.N.{gate,up,down}_proj` Linears. **No registration is needed** if the
module satisfies `FusedExpertsProtocol` structurally — bare `down_proj` plus
`gate_up_proj`/`up_proj` Parameters. `load_quantizable_moe(model_cls)` is the
faster variant that linearizes during load rather than after.

**Always assert the expert count before spending GPU time** (§4.1). The
arithmetic is `layers × experts × projections`; if your target list does not
hit it exactly, the recipe is wrong and the failure is silent.

⚠ **Keep routers in `ignore`.** A 4-bit router picks *different experts* — that
error does not average out downstream, it changes which weights run at all.

### 3.4 Toolchain version deadlocks

Both directions have burned us, so the resolution is: **use llm-compressor / compressed-tensors,
not nvidia-modelopt.**

- modelopt **0.45** ↔ transformers 5.12: `mtq.quantize` dies `TypeError: issubclass() arg 2 must
  be a class` (modelopt registers transformers' `FusedMoE`, a *function* in 5.x, as an nn class).
- modelopt **0.43** doesn't fix it — it drags transformers back to 4.57, which cannot load
  `qwen3_5` at all.
- modelopt's config API also trails the current model families by a version.

### 3.5 Vision tower and its configs

- Keep the **vision tower in `ignore`** (BF16). Only the LLM backbone gets quantized.
- The wrapper-class save **drops `preprocessor_config.json`** (and the video one). Without it the
  seat crash-loops `Can't load image processor`. Restore from the source — and if the upstream repo
  omits it, **reconstruct it from `processor_config.json`'s `image_processor` sub-dict**.

### 3.6 Memory and device placement (large models)

- **`device_map=None`/`"cpu"`, never `"auto"`.** `auto` fills GPU0 and OOMs during un-fusing;
  constraining with `max_memory` then offloads to the *meta* device, which cannot be `.copy_()`d.
  CPU-resident keeps every tensor real; the sequential pipeline still onloads per-layer to GPU.
- **Avoid mmap on `/tank`.** `safetensors.safe_open()` mmaps a whole shard; on ZFS a 50 GB shard
  ENOMEMs regardless of free RAM (MAP_SHARED never consults the commit limit). Read with plain
  `read()` + `load(bytes)`, one shard cached at a time.
- **`vm.overcommit_memory=1`** on ana-ml2 (durable via `playbooks/ana-ml2-overcommit-memory.yaml`).

### 3.9 ⭐⭐ A sharded forward can be silently WRONG — never trust `device_map="auto"` for activations

Splitting **Qwen3.8-27B (Qwen3_5 hybrid)** across the two Blackwells with `device_map="auto"`
produces a model that loads clean, reports no error, and computes **garbage**: the residual stream
collapses to **exactly zero** a couple of layers past the GPU0→GPU1 boundary, and the logits decode
to rubbish (`'8'`, `'�'`, `'b'`). Every layer *below* the boundary stays healthy, deterministic, and
bit-identical to a single-GPU run — which is what makes it so dangerous. A capture that reads a
low layer looks perfectly plausible and is fine; one that reads a high layer is reading zeros, and
nothing in the pipeline says so. Measured 2026-08-20 (§9 Cold-Fusion).

**Rule: any workload that reads activations — refusal-direction capture, calibration, activation
statistics, PPL — must run on ONE device.** Sharding is for *storage*, and it is only safe when you
consume the model's final output through an engine that was built for it (vLLM does TP correctly;
`device_map="auto"` in transformers is not the same thing). If it does not fit on one card, shrink
the model, not the guarantee: **truncating the decoder to N layers is exact** for any activation
read at a layer < N (a causal stack's layer-N state cannot depend on layers above N), and it is
cheap — verified by reproducing the full model's layers 18/20/22/26 bit-for-bit.

**Gate it, don't remember it.** Assert single-device residency and zero offload before the forward:

```python
dmap = getattr(model, "hf_device_map", {}) or {}
gpus = {str(v) for v in dmap.values()} - {"cpu", "disk"}
offloaded = [k for k, v in dmap.items() if str(v) in ("cpu", "disk")]
if len(gpus) > 1 or offloaded:
    sys.exit("residency gate FAILED — sharded/offloaded forward reads garbage")
```

### 3.10 ⭐⭐ `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` corrupts retained tensors

On torch 2.12+cu130 / Blackwell, tensors that **outlive their allocation** come back corrupted with
this flag set: captured hidden states carried Inf / NaN / zeros that **moved between bit-identical
forwards** (same input, same weights → a different layer corrupted each time). Unset, the identical
forwards are exactly reproducible. Several runbooks recommend this flag for headroom on large
loads; for anything that *keeps* activations it buys corruption.

Two tells that distinguish this from a real numerical blowup, both worth knowing because they
generalise: a genuine blowup **propagates** to later layers and is **deterministic**. Corruption
does neither — downstream layers were finite and consistent, and the affected layer moved run to
run. **If a "NaN" fails to propagate, stop debugging the math and start debugging memory.**

Corollary: **do not read `output_hidden_states=True` off a returned object** on a large multi-device
load. Take what you need *during* the forward with a `register_forward_pre_hook` that clones to CPU
immediately — it closes the reuse window and never retains a `[B, seq, hidden]` tensor per layer, so
it is cheaper than the thing it replaces.

### 3.11 Determinism is a necessary check, not a sufficient one

Both defects above were found by the cheapest possible test — **run the same input twice and diff**
— which no amount of eyeballing plausible-looking numbers would have caught. Add it to any
activation-reading pipeline. But note the trap that followed: after fixing the allocator, the run
went perfectly "deterministic" *because the corrupted layers were now stably zero*. Pair the
determinism check with a **magnitude** check (residual norms should grow smoothly with depth; an
exact 0.0 mid-stack is impossible) and, where you can, a **coherence** check (generate 40 tokens and
read them).

### 3.12 ⭐⭐ You cannot free a 27B model in-process — give each model its own process

Any A/B that loads two large checkpoints in sequence (KL, logit diffing, teacher-vs-student)
will try to release the first before loading the second. **On this stack, it does not work.**
Measured 2026-08-20 on Qwen3.8-27B bf16, free VRAM after each attempt:

| teardown | free VRAM |
|---|---|
| `del model` + `gc.collect()` + `torch.cuda.empty_cache()` | 45,287 MiB |
| same, with the model confined to an inner frame that exits | 45,287 MiB |
| **the process exits** | **97,247 MiB** |

The ~51,300 MiB of weights stayed resident through both in-process teardowns. The first
run survived only because **PyTorch's allocator hit OOM on the second load, ran a collection
itself, and retried** — the second model landed by rescue, not by design. That is not a
release strategy: on an architecture where a silent CPU offload does not raise (§3.9), the
day the retry does not fire you get confident garbage instead of an error.

**Do this instead:** one process per model, hand results to disk between them
(first-token log-probs for a 250k vocab are ~715 MiB per model — nothing), and gate each
stage on free VRAM *before* the load. Reference implementation:
`services/coldfusion-abliteration/kl_divergence.py` (`--stage ref|cand|score`).

Two gate corollaries learned in the same session:

- **⭐ A residency gate that reads `hf_device_map` cannot fail.** The map is **empty**
  whenever transformers puts the whole model on one device, so the check reports
  "unsharded" both when everything is fine and when there is nothing to inspect. Read
  `{p.device for p in model.parameters()}` — ground truth in every case. (Generalises
  [[feedback_assert_effective_value_not_substring]]: presence of a passing check is not
  evidence of a check that can fail.)
- **⭐ Size VRAM from the checkpoint's own headers, never from a remembered figure.** A
  runbook carried "bf16 is 50 GB"; the real number was 50.10 **GiB** = 51,300 MiB of
  text-only weights. That 3.7 GB unit error is exactly the difference between "stop one
  co-tenant" and "stop both", and it cost an aborted window. Sum the safetensors header
  offsets (excluding tensors the loader class won't instantiate — vision, MTP); read only
  the 8-byte length prefix + JSON header, never `safe_open`, which mmaps the whole shard
  and ENOMEMs on ZFS (§ *Avoid mmap on `/tank`*).

### 3.13 ⭐⭐ The observer you ASKED for is not necessarily the observer you GOT

`quant_mixed_nvfp4.py` sets `observer="imatrix_mse"` on the NVFP4 W4A4 group. It has
**never once been used.** llm-compressor looks for importance data, finds none, and
silently degrades:

```
_get_validated_importance | WARNING - imatrix_mse: no importance data available.
                                      Falling back to uniform MSE.
```

Confirmed on the 2026-08-20 09:59 incumbent quant **and** the 22:45 Heretic-300
quant; `find /tank/aimodels -iname "*imatrix*" -o -iname "*importance*"` returns
nothing. Every NVFP4 build in the fleet has run uniform MSE while the recipe claimed
importance weighting.

**Why it went unseen for months:** the warning scrolls past inside a tqdm progress
bar during a ~20 minute quant. It is only visible if you read the log while it runs.

**The generalisable rule, which is bigger than imatrix.** A quantizer, optimiser or
observer that *silently falls back to a weaker default* is a whole class of invisible
quality loss — the config is accepted, nothing errors, the artifact benchmarks
plausibly, and you never learn you got the cheap path. So:

- **Grep the quant log for `WARNING`, `Falling back`, `not available`, `ignoring`
  before trusting an artifact.** Make it a step, not a habit.
- **Assert the effective setting, never the requested one** — the same rule as
  [[feedback_assert_effective_value_not_substring]], applied to quantizer internals
  rather than config files.
- If the fallback turns out to be unavoidable in your toolchain version, **change the
  recipe to say what it actually does.** A recipe line that silently lies is worse
  than one that admits a limitation.

⚠️ **Do not "fix" this by assuming an imatrix would help.** Verify first that your
llm-compressor version can consume an externally supplied importance matrix at all,
and in what format. Parked as `park/nvfp4-recipe-asks-for-imatrix-mse-but-silently-2`
(id 42) with the calibration corpus that would feed it.

✅ **Comparisons already made remain valid.** Because *every* build shares the
fallback, the incumbent-vs-candidate A/Bs (47.2% acceptance, PPL 6.910, and the
2026-08-20 Heretic-300 build) are apples-to-apples. This is unrealised upside, not a
correction to past numbers.

### 3.14 ⭐⭐ Calibration BAKES a truncation cap into the shipped tokenizer

**Symptom (on a newer transformers, at startup, on a vision model):**

```
ValueError: Mismatch in `image` token count between text and `input_ids`.
Got ids=[2047] and text=[16384]. Likely due to `truncation='max_length'`.
```

The engine never serves a request. The number in `ids=[…]` is your **calibration seqlen minus
one**, which is the tell.

**Cause — an in-place mutation you never wrote.** Calibration tokenizes like this:

```python
tok(b["text"], truncation=True, max_length=seqlen, add_special_tokens=False)
```

For a **fast** tokenizer that call does not just return ids — it **mutates the Rust backend's
truncation state in place**. A later `tok.save_pretrained(out)` then persists it:

```json
"truncation": {"direction": "Right", "max_length": 2048, "strategy": "LongestFirst", "stride": 0}
```

The source model has `"truncation": null`. **You shipped a tokenizer that clamps every prompt at
the calibration length, permanently.**

**Why it hid for months.** Older transformers does not enforce the text-vs-ids count check, so
the cap sits latent — the model serves, gates pass, vision works, nothing logs. It only detonates
when you bump the image, and then it presents as a *vision* bug at startup with no mention of
tokenizers. It also caps the effective image resolution long before it kills the seat: at a 2048
cap the largest servable image is ~1448×1448, because `(edge/patch)² / merge²` image tokens must
fit under it.

**The fix — never save the calibration tokenizer.** Re-read a pristine one from the source:

```python
from transformers import AutoTokenizer as _AutoTokenizer
_AutoTokenizer.from_pretrained(a.model, trust_remote_code=True).save_pretrained(a.out)
```

then **assert** it, because this is exactly the class of defect that returns silently:

```python
if json.load(open(f"{a.out}/tokenizer.json")).get("truncation"):
    raise SystemExit("FAILED CHECK: saved tokenizer carries a truncation cap")
```

Both live in `quant_mixed_nvfp4.py` as of 2026-08-22.

**Audit any build predating that.** One line per model:

```bash
python3 -c 'import json,sys;print(json.load(open(sys.argv[1]+"/tokenizer.json")).get("truncation"))' <model_dir>
```

Measured 2026-08-22 — every mixed-NVFP4 build from this pipeline was affected, and the two live
ones were corrected in place (backup `tokenizer.json.bak-truncation-20260822`; only the
`truncation` field changed, vocab and `added_tokens` byte-identical):

| build | truncation as found |
|---|---|
| `qwen38-27b-orcarouter-nvfp4-mixed` (live `gen`) | **2048** → fixed |
| `mog-sec-27b-nvfp4-mixed` (live `sec`) | **2048** → fixed |
| `qwen38-27b-heresy-nvfp4-mixed` (retired) | 2048, left as-is |
| `G4-MeroMero-v2-31B-NVFP4A16` (different pipeline) | `null` ✓ |
| `mog-sec-27b-bf16` (source) | `null` ✓ |

**Editing it is safe on a running seat** — vLLM reads the tokenizer at startup and holds its own
copy, so the fix lands on the next restart with no disruption.

**The general lesson, which is the transferable part:** this is the third defect in this playbook
where *the artifact carries config authored against an older transformers and a newer one starts
enforcing it* (see also the Gemma-4 heterogeneous `head_dim`). **Treat "we bumped the image" as a
config-compatibility event, not just a version change** — and prefer saving artifacts re-read
from the source over saving objects the pipeline has touched.

---

## 4. Pipeline shape

### 4.1 Prove the targets before spending GPU time

Enumerate module names from the safetensors index and check your regexes against them: **zero
overlap between groups, and the union covers every layer you intended.** This is free, takes
seconds, and catches a mis-scoped regex that would otherwise surface as a mystery quality
regression hours later. Reference: `services/gen-seat-mixed-quant/validate_targets.py`.

### 4.2 Quantize

Calibration data matters for `imatrix_mse` + static activation observers. We use
`/tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl` (512 chat samples, RP/GM-flavoured
— appropriate for our seats). 256 samples @ 2048 tokens ≈ 20 min for a 27B on one Blackwell.

### 4.3 The mandatory post-steps

Never optional, always in this order, and the last one **verifies rather than assumes**:

1. Graft `model-mtp.safetensors` + register its tensors in the output index.
2. Restore `preprocessor_config.json` / `processor_config.json` / `video_preprocessor_config.json`.
3. **Re-inject `re:^mtp.*` into `quantization_config.ignore` and confirm it is there** (§3.3).
4. **Confirm the saved `tokenizer.json` has `truncation: null`** (§3.14) — calibration mutates the
   fast tokenizer in place and `save_pretrained` bakes the cap in. Latent on an older
   transformers, fatal on a newer one.

Reference implementation: `services/gen-seat-mixed-quant/post_quant.py`.

### 4.4 Test on a temp port, never on the live seat

Serve the candidate on an alt port with the live seat's **exact** flags, run the gate (§5), and
only then flip `.env`. Keep the previous build on disk; rollback is one `.env` line.

---

## 5. The acceptance gate — and how measurement lies to you

Speed alone does not justify cutting over a shared seat. Gate on **all** of: decode tok/s, MTP
acceptance, perplexity, a behavioural surface test, and — for an abliterated model — that the
abliteration survived.

**Three ways the numbers have lied to us. All three produced confident, wrong results.**

1. **Prefix caching fakes both speed metrics.** A fixed prompt returns byte-identical timings run
   after run; you are measuring cache, not compute. Worse for prefill: a *seeded* nonce
   regenerates the previous run's prompts verbatim and reads **~41k tok/s of cache-hit instead of
   ~5k of real prefill**. Use a fresh unseeded nonce per request; never seed a cache-buster.
2. **`prompt_logprobs` are garbage while speculative decoding is on** — ~uniform over the vocab
   (median rank ~10⁵; " Paris" after "The capital of France is" ranked 69698). **Perplexity must be
   measured on a seat served without `--speculative-config`,** on both sides of the comparison.
3. **A 0600 `.env` makes `docker compose` silently no-op.** Without `sudo` it fails
   `permission denied` reading `.env`, **leaves the old container running**, and reports success —
   producing a full page of "benchmark results" that were just the unchanged baseline.
   **Hard-verify the change landed against `docker inspect …Config.Cmd`.**

**Re-measure the baseline before believing a target.** The 2026-08-15 handoff quoted ~68 tok/s;
cache-busted, the incumbent was already doing 80.1 — essentially the *target* of the work queued
against it. Had that not been re-measured, doing nothing would have looked like a 20% win.

**Cheap shortcut worth taking first:** if a reputable published quant of the same architecture is
already on-box (or is a small pull), **serve it as a probe and measure it** before committing
hours to your own. It answers "is this gain even real?" in ten minutes *and* hands you the recipe.

Harness: `services/gen-seat-mixed-quant/bench/` — `quickbench.py` (decode + acceptance),
`prefill_bench.py`, `eval_quality.py` (PPL + abliteration), `surface_test.py` (chat, vision, tools,
thinking split, long-context needle, streaming), `serve_probe.sh`.

---

### 5.1 ⭐⭐ Acceptance is not throughput — always run the DEPTH control

**Measured 2026-08-22**, same instrument (vLLM's own `spec_decode` counters, delta over a fixed
workload, temp 0), same target, same engine:

| config | accepted tok/forward | throughput |
|---|---|---|
| MTP k=3 | 2.753 | 114.9 tok/s |
| MTP k=7 | **3.041** ⬆ | **74.0 tok/s** ⬇ |

**Raising `num_speculative_tokens` improved acceptance and destroyed throughput.** Reporting
acceptance alone would have recommended a 36% regression.

**Why:** a single-module MTP head (`mtp_num_hidden_layers: 1`, one `mtp.layers.0`) has no depth
of its own — vLLM runs it **autoregressively**, so k draft tokens cost **k sequential forward
passes**. Past a shallow depth the drafting cost exceeds what the extra accepted tokens save.
Check `mtp_num_hidden_layers` before assuming depth is cheap.

**The rule: when comparing two speculative methods, match k, or you are measuring depth rather
than method.** A parallel-drafting drafter (DFlash2 and kin, which propose a whole block in one
pass) at k=7 versus an autoregressive MTP at k=3 is not a method comparison — the depth control
is what separates them. In our case the control showed most of the apparent acceptance win was
depth, while the *throughput* win was real and came from parallel drafting, not better drafts:
our MTP was **better at position 0** (79.6% vs 75.4%) and still lost overall.

**Corollary — report both, always.** Acceptance rate, mean accepted length, and end-to-end
tok/s. Any one of the three alone can point the wrong way.

---

## 6. Hardware and co-residency

- **ana-ml2 = Blackwell (sm_120)**, 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths
  (`match_exact=True` on sm90) are **closed** here — do not plan around them.
- **irv-ml1 = Ampere (sm_86)**, 3090 + A6000. **No native FP8/FP4** — 4-bit there is a VRAM saving
  only, not a speed win. Don't port a Blackwell scheme over and expect the throughput.
- **GPU co-residency is a zero-sum budget, and a *smaller* model can break its neighbour.**
  `gpu-memory-utilization` is a fraction of the *whole card*, so when new weights are smaller the
  seat absorbs the slack as extra KV rather than releasing it. That is exactly how a −5.2 GB
  requant left the co-resident seat **0.18 GiB** short and crash-looping. **After any requant,
  re-check both seats' budgets** and hand the space back explicitly.

---

## 7. Superseded claims — do not follow these

Old docs stay for their history, but these specific claims are **false now** and will cost you a
day if followed:

| claim | where | status |
|---|---|---|
| "Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance" | `docs/runbooks/heretic2-nvfp4-mtp-seat.md` §landmine 2 | **SUPERSEDED 2026-08-14.** The 0% was the missing `re:^mtp.*` ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.7–83.2% acceptance, live. Use compressed-tensors. |
| "Abliteration desyncs the MTP head → uncensored models can't do MTP" | earlier auto-memory | **SUPERSEDED 2026-08-14.** A modest abliteration preserves MTP (83.7% at bf16). Test MTP on **bf16 first** to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source. |
| "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). |
| "transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 without causal-conv1d; it is precision-driven cancellation and fp32 resolves it" | `services/coldfusion-abliteration/README.md`, `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` | **SUPERSEDED 2026-08-20.** Precision was never the variable. The NaN came from **multi-GPU sharding** and **`expandable_segments`** (§3.9, §3.10); fp32 only made it rarer, which is worse than failing. On one GPU with a plain allocator, **bf16 is exactly deterministic through all 64 layers and generates coherent prose** — at 50 GB and 4.3× the throughput of the 111 GB fp32 it replaced. |

---

## 8. Measured negatives — don't re-chase

- **`num_speculative_tokens` = 3 is optimal** on the Qwen3.8-27B seat. Swept: n=2 → 77.1,
  **n=3 → 80.1**, n=4 → 78.7, n=5 → 75.9 tok/s. Higher n trades acceptance for draft width and
  loses. Re-sweep only if the drafter architecture changes.
- **Uniform W4A4** — see §7 row 3.
- **Dense-VL as the anatomy judge** — A/B'd, MoE retained. Don't re-propose.

---

## 9. Worked examples

Per-model artifacts. Read for *how a specific model went*, not for the general lessons — those are
above, and where the two disagree, **this file wins**.

| artifact | what it is |
|---|---|
| `services/gen-seat-mixed-quant/` | **current reference.** Mixed NVFP4+FP8 on Qwen3.8-27B-Uncensored: scripts, acceptance harness, raw measurements. |
| `stacks/gen-seat/README.md` | the live `gen` seat (7 LiteLLM aliases) |
| `stacks/meromero-charrp/README.md` | Gemma-4 seat — the **tool-call/reasoning-parser** trap (a parser default that returns null `content` for all prose) |
| `services/heretic2-nvfp4-quant/` | modelopt-format MTP seat — historical; see §7 before following it |
| `tools/mistral-small4-nvfp4/` | MoE + native-convert path; source of §3.6 |
| `docs/pfi/recommended-model-settings.md` | serve-time sampler/flag defaults (not quant) |

**A new model just dropped and needs requanting?** §1 → §2 → §4 → §5. Skim §3 first; it is the
part that costs hours.