# Model quantization playbook — the lessons that keep costing us hours **Read this before starting any new quant.** Not the per-model runbooks — those are worked examples of a *specific* model at a *specific* point in time, and several carry claims that are now false (see §7). This file owns the **transferable** part: what recurs regardless of which model dropped this week. Written 2026-08-15, after the fourth quant in five weeks re-discovered the third-known instance of the same loader-class bug. Scope: NVFP4 / FP8 / mixed-precision on the Blackwell boxes (ana-ml2), vLLM-served. Ampere (irv-ml1) has no native FP4/FP8 — see §6. **Maintenance rule.** When a quant teaches you something *model-agnostic*, it lands here and the per-model README links up. When it's model-specific (this checkpoint's odd tensor names, this finetune's missing config), it stays in the per-model artifact. If you find yourself writing a "Gotchas" section that repeats §3, you are re-litigating — add the delta here instead. --- ## 1. The 60-second decision: which scheme On Blackwell + vLLM, for a dense-or-hybrid VL model you intend to serve at long context: | want | scheme | notes | |---|---|---| | **default, best speed/accuracy** | **mixed: NVFP4 W4A4 bulk MLPs + FP8 W8A8 attention/`lm_head`/last-8-layer MLPs** | the current answer. §2. | | max fidelity, don't care about prefill | NVFP4 **W4A16** (weight-only) | forces the **Marlin** kernel — ~half the prefill of native FP4 | | small model, VRAM is free | FP8 **W8A8** | safe and simple; 2× the weight bytes of 4-bit | | — | ~~"W4A8" = NVFP4 weights + FP8 activations~~ | **DOES NOT EXIST.** §3.1 | **Measured on Qwen3.8-27B (2026-08-15), W4A16 → mixed:** decode +18%, prefill **+78–98%**, MTP acceptance unchanged, perplexity +1.7%, weights −19%. Note the shape of that: **decode barely moves, prefill nearly doubles.** Decode at batch-1 is memory-bandwidth-bound and the weights are 4-bit under either scheme, so there is little to win; prefill is compute-bound, which is where native FP4 tensor cores replace the Marlin dequantize-to-BF16 path. If someone promises you a big *decode* win from a scheme change, be skeptical — and go measure §5 before believing it. **The accuracy cost is real and is paid on purpose.** Operator ruling 2026-08-15: the ~1.7% perplexity is an acceptable price for the speed. Settled — don't re-litigate. For correct attribution: it is the **activation**-quantization cost (A4/A8 vs BF16 activations), *not* an MTP cost. Turning MTP off does not recover it; only reverting the quant does. --- ## 2. The reference recipe (mixed-precision) Lifted from `unsloth/Qwen3.8-27B-NVFP4` and replicated in-house. **Prefer replicating a published recipe from a reputable quantizer over inventing one** — they have already paid for the sensitivity analysis. | group | scheme | targets | |---|---|---| | `group_0` | FP8 W8A8 — channel weights (static) + per-token dynamic activations | `self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`, `lm_head`, **the last 8 layers' MLPs** | | `group_1` | NVFP4 W4A4 — `tensor_group` gsize 16, fp8 scales, `imatrix_mse` weights, `dynamic:"local"` activations | **all remaining** MLP `{gate,up,down}_proj` | | kv cache | FP8 static tensor | | | ignore | vision tower, `linear_attn.{norm,in_proj_a,in_proj_b}`, `re:^mtp.*` | | Three things in there are load-bearing and easy to drop: - **Late layers stay FP8.** Holding the last ~8 layers' MLPs (and `lm_head`) at 8-bit is the accuracy-preservation trick — late layers are the sensitive ones. Uniform W4A4 is what collapses. - **`imatrix_mse` on the W4A4 weights**, not `memoryless_minmax`. Importance-weighted; needs calibration data. - **Group targets must be non-overlapping.** Do not let `group_1`'s `.*mlp\..*` also match the late layers and rely on group precedence to sort it out. Enumerate the early layers explicitly (`re:.*layers\.([0-9]|[1-4][0-9]|5[0-5])\.mlp\.…`) and **prove it** with a dry run (§4.1). **Toolchain:** `pip install llmcompressor` into stock `vllm/vllm-openai:latest` gives llmcompressor 0.13 + compressed-tensors 0.18 without disturbing torch/transformers. **Avoid nvidia-modelopt** — see §3.4. --- ## 3. The recurring landmines Ordered by how much time each has cost. Every one of these has bitten more than once. ### 3.1 "W4A8" is not a servable shape vLLM's compressed-tensors dispatcher (`compressed_tensors.py:704-713`) accepts NVFP4 weights with **exactly two** activation settings: | `input_activations` | result | |---|---| | `None` | W4A16 — and it **forces the Marlin kernel** (`kernels/linear/__init__.py:881-883`) | | NVFP4 | W4A4, native | Anything else — **FP8 included** — raises at load: ``` ValueError: For NVFP4 weights, input quantization must also be NVFP4 format, None for NVFP4A16 ``` `CompressedTensorsW4A8Fp8` exists but is **INT4** weights (`W4A8_SUPPORTED_TYPES_MAP = {4: int4}`) gated on `_check_scheme_supported(90, match_exact=True)` — Hopper-exact, so on Blackwell (sm_120) it is closed twice over. **FP8 enters per-layer-group, never as activations on NVFP4 weights.** *Cost: one queued task written against an impossible scheme.* ### 3.2 Wrong loader class → silent weight-load failure **Rediscovered three times.** Load the model through the class vLLM actually serves — the `…ForConditionalGeneration` / `…ForImageTextToText` **wrapper**, never `AutoModelForCausalLM`. `AutoModelForCausalLM` resolves a VL config to the text-only inner class and saves a **flat** config with `model.layers.*` keys. vLLM's weight mapper wants `model.language_model.*` (+ `model.visual.*`). The mismatch does not error — **every layer silently fails to load** and you get `!!!!` gibberish, or an engine that rejects the checkpoint outright. *Bit: heretic2 (gibberish), Dark-Scarlett (both vLLM and SGLang refused the checkpoint), and the 2026-08 rounds.* ### 3.3 The MTP head — three separate ways to lose it Speculative decoding is a large fraction of the seat's throughput. It fails **silently**: the model serves fine, just at 0% acceptance. 1. **The wrapper class does not instantiate `mtp.*`,** so the quant drops it. Post-quant you must graft the BF16 `model-mtp.safetensors` back and register its tensors in the output index. 2. **`re:^mtp.*` must be in `quantization_config.ignore`** — else vLLM loads the grafted BF16 head as though quantized, it comes up **uninitialised**, and acceptance is 0%. 3. **⭐ llm-compressor PRUNES `ignore` entries that matched no module at quant time.** Since the wrapper never loaded `mtp.*`, the entry matches nothing and is **silently deleted from the saved config — even though you put it in the recipe.** So it must be re-injected *after* the graft, and then **verified, not assumed.** *Cost: three rounds. The verify step caught it live on the third.* There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not propagate modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix (needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4. ### 3.8 ⭐⭐ Multi-turn degeneration from TWO real compounding causes — how they masked each other The most expensive diagnosis this project has had, because there were **two real causes at once** and each partial fix moved the needle enough to look like *the* answer. Recorded precisely because the first write-up of this section over-attributed it to the quant alone; that was wrong. **Cause 1 (real, upstream): the vLLM `qwen3_5_mtp` × Gated-DeltaNet bug.** Confirmed by two cross-frontier peers and the tracker (vllm#47087 symptom-twin, #43559 fix lineage, #51113 fix): the GDN recurrent state cannot roll back on a partial draft-accept, so speculative decoding corrupts it, worse with context. Architectural — vLLM/SGLang/llama.cpp mainline all shared it. **Genuinely fixed enough** by moving to vLLM **nightly** (`v0.27.2rc1.dev150+`, carries #51113): the operator reported it "significantly better" — this was a real bug, not just an amplifier. **Cause 2 (real, quant): full W4A4 is mildly subpar, per the known gradient.** `sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4` is **full** W4A4 — 4-bit *activations* on attention too, the bottom of the activation-precision ordering already in §1: **W4A4 (A4) < W4+FP8 (A8) < W4+bf16 (A16)**. Not "defective," just lowest-fidelity; on top of Cause 1 it degenerated ~15-20% of real multi-turn generations. The FP8-attention **mixed** build (`qwen38-27b-uncensored-nvfp4-mixed`, same base, same MTP, same nightly) sits a rung up that gradient and is coherent. AEON was purged 2026-08-17 (operator ruled it no-good; re-pullable from HF). **Why it cost days — and the process lessons that stand:** 1. **Two real causes compound and mask each other.** Each mitigation (MTP-off, APC-off, the nightly #51113 fix) partially helped, so each looked like the fix and then failed in real use. When a mitigation "helps but doesn't fix," suspect a *second* cause rather than a wrong one. 2. **Stochastic degeneration (~15-20%) is nearly invisible to a small synthetic probe** — a 7-turn run passes ~4 in 5. n=1 "clean" proves nothing; this class needs many runs or the operator's real high-volume use. Three non-fixes were "validated" by a single clean probe here. 3. **Isolate the WEIGHTS in parallel with the serving flags, not after.** Swapping to a different quant of the same base (AEON→mixed) is what finally separated Cause 2 from Cause 1; doing it earlier would have shortened the hunt. But note it would NOT have found Cause 1 — the vLLM bug was real and needed the nightly. 4. **Prefer FP8 attention (the §2 mixed recipe) over full W4A4** for a coherence- sensitive seat. AEON passed every static gate (abliteration 4/4, surface 6/6, a 36k needle, 52% acceptance) and was still the lower-fidelity of the two. Current primary gen: the mixed FP8-attention build on pinned vLLM nightly with MTP, until the DavidAU Qwen3.8 lands. A W4+bf16 (W4A16) build would be higher fidelity still (§1) at a prefill cost — an option if the mixed build ever proves marginal. ### 3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn §3.3 is about *losing* the head (0% acceptance, silent). This is the opposite and worse failure: the head loads, acceptance looks healthy, single-turn output is perfect — and then it **corrupts multi-turn conversations** once cumulative context passes **~2,000 tokens**. The reply collapses in length *and* bleeds earlier turns into the current answer (a "describe durian" reply that contained the Krebs-cycle and winter answers from three turns back). Single-turn probes and the acceptance gate (§5) **do not catch it** — it only appears as accumulated context grows. Isolated 2026-08-16 (operator-confirmed), each step measured on a fixed 7-turn probe: - **Not the serving gateway, not sampling, not repetition/template.** Identical input gateway-vs-direct behaves the same; presence_penalty 1.5/0.5/0.0 all collapse; higher temperature collapses harder; a conversation of *unrelated* topics collapses at the same ~2k tokens as a repetitive one → it is context- length-driven, not template lock-in. - **Model-independent across every Qwen3.8-27B quant** (AEON W4A4, unsloth FP8-attn, our in-house mixed) — so not a quant-brand or scheme artifact. - **DECISIVE: same model + same conversation, MTP OFF → coherent through 4k+ tokens, zero bleed.** Toggle it back on → collapse returns. MTP is the cause. **Qwen3.6-27B running the same `qwen3_5_mtp` method is CLEAN.** So the 3.6 MTP head/graft is fine and the 3.8 one is not — suspects: the bf16 graft being subtly wrong for the 3.8 head, or the vLLM `qwen3_5_mtp` impl diverging at `num_speculative_tokens=3`. Open upstream question (queried dvalin/bil-smithy 2026-08-17). **Rule: gate MTP on a MULTI-TURN coherence probe, not just single-shot acceptance.** Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens for length-collapse and cross-turn bleed. ⛔ **THE APC-OFF MITIGATION DID NOT HOLD — SUPERSEDED 2026-08-17, see §7.** What follows is kept for its history and **must not be applied**: *"disable prefix caching, keep MTP. The corruption is gated on MTP × prefix-caching together (vllm#43559 / #47194); with `--no-enable-prefix-caching` the buggy partial-accept align-path is inert. Confirmed on our stack: AEON W4A4, MTP on + prefix-caching off → the 7-turn varied series stays coherent through 3.9k tokens, zero bleed, at 104.6 tok/s / 53.6% acceptance."* **It passed that synthetic 7-turn probe and the operator still saw severe degeneration in real use.** Reverted the same day. The probe was structurally under-covering the real workload on both content distribution and depth — which is §3.7's own standing rule (*gate on a multi-turn coherence probe, not single-shot acceptance*) failing at one level up: the multi-turn probe was itself too small to gate on. **WHAT ACTUALLY RESOLVED IT.** The multi-day hunt root-caused to the **AEON W4A4 quant being defective** — ~15-20% of generations went degenerate — with MTP, prefix-caching and the gateway all merely *amplifying* it. That is why every partial mitigation "helped" without fixing anything (§3.8). The gen seat today runs the in-house Heretic **mixed NVFP4+FP8** build (FP8 attention, not W4A4) on vLLM nightly carrying #51113, with **MTP ON and prefix-caching ON**, and is coherent in real use. Verified against the live seat 2026-08-26: `vllm-gen` runs `--enable-prefix-caching` with `qwen3_5_mtp` / `num_speculative_tokens 3`. The compose file (`stacks/gen-seat/compose.yaml`) carries the full history inline and is the current authority; this section was stale against it for nine days. Things that do **not** work, ruled out: `num_speculative_tokens=1` (corruption is depth-independent — reproduces at n=1 and n=2, deterministically probed upstream); switching engine (vLLM / SGLang / llama.cpp mainline all share the GDN-rollback bug — it is architectural). The proper upstream fix (vllm#51113) is in `main` / `v0.27.2rc0` only — not in a stable release, so we hold at APC-off until it lands. Two cross-frontier peers (dvalin/bil-smithy) confirmed the bug class and pointed at the open symptom-twin issue #47087. ### 3.15 ⭐⭐ Fused 3-D MoE experts are INVISIBLE to a `targets=["Linear"]` recipe **Symptom: none.** The quant completes, the artifact loads, and 88.5% of the model is still BF16. Nothing warns you. Modern MoE checkpoints store each layer's experts as **two fused 3-D `nn.Parameter` tensors**, not as N `nn.Linear` modules. Gemma-4 26B-A4B: model.language_model.layers.N.experts.gate_up_proj BF16 [128, 1408, 2816] model.language_model.layers.N.experts.down_proj BF16 [128, 2816, 704] Note the **absent `.weight` suffix** — that is the tell. `mlp.down_proj.weight` is an `nn.Linear`; `experts.down_proj` is a bare parameter. Measured on that checkpoint, recipe targeting `["Linear"]`: Linear modules 427 WILL quantize 205 (experts: 0) <- 22.84 B params untouched **This is the same defect that killed QLoRA on this architecture** — `bitsandbytes` 4-bit replacement also walks `nn.Linear` modules and also silently skipped the experts. Two different tools, one blind spot, because the blind spot is in the *checkpoint layout*, not the tool. **The fix** (llm-compressor ≥ 0.12): ```python from llmcompressor.modeling.moe.linearize import linearize_moe model = SomeForConditionalGeneration.from_pretrained(...) linearize_moe(model) # BEFORE building the recipe ``` Linear modules 11947 WILL quantize 11725 (experts: 11520) # 30 layers x 128 x 3 proj `linearize_moe` unfuses the 3-D parameters into per-expert `experts.N.{gate,up,down}_proj` Linears. **No registration is needed** if the module satisfies `FusedExpertsProtocol` structurally — bare `down_proj` plus `gate_up_proj`/`up_proj` Parameters. `load_quantizable_moe(model_cls)` is the faster variant that linearizes during load rather than after. **Always assert the expert count before spending GPU time** (§4.1). The arithmetic is `layers × experts × projections`; if your target list does not hit it exactly, the recipe is wrong and the failure is silent. ⚠ **Keep routers in `ignore`.** A 4-bit router picks *different experts* — that error does not average out downstream, it changes which weights run at all. ### 3.4 Toolchain version deadlocks Both directions have burned us, so the resolution is: **use llm-compressor / compressed-tensors, not nvidia-modelopt.** - modelopt **0.45** ↔ transformers 5.12: `mtq.quantize` dies `TypeError: issubclass() arg 2 must be a class` (modelopt registers transformers' `FusedMoE`, a *function* in 5.x, as an nn class). - modelopt **0.43** doesn't fix it — it drags transformers back to 4.57, which cannot load `qwen3_5` at all. - modelopt's config API also trails the current model families by a version. ### 3.5 Vision tower and its configs - Keep the **vision tower in `ignore`** (BF16). Only the LLM backbone gets quantized. - The wrapper-class save **drops `preprocessor_config.json`** (and the video one). Without it the seat crash-loops `Can't load image processor`. Restore from the source — and if the upstream repo omits it, **reconstruct it from `processor_config.json`'s `image_processor` sub-dict**. ### 3.6 Memory and device placement (large models) - **`device_map=None`/`"cpu"`, never `"auto"`.** `auto` fills GPU0 and OOMs during un-fusing; constraining with `max_memory` then offloads to the *meta* device, which cannot be `.copy_()`d. CPU-resident keeps every tensor real; the sequential pipeline still onloads per-layer to GPU. - **Avoid mmap on `/tank`.** `safetensors.safe_open()` mmaps a whole shard; on ZFS a 50 GB shard ENOMEMs regardless of free RAM (MAP_SHARED never consults the commit limit). Read with plain `read()` + `load(bytes)`, one shard cached at a time. - **`vm.overcommit_memory=1`** on ana-ml2 (durable via `playbooks/ana-ml2-overcommit-memory.yaml`). ### 3.9 ⭐⭐ A sharded forward can be silently WRONG — never trust `device_map="auto"` for activations Splitting **Qwen3.8-27B (Qwen3_5 hybrid)** across the two Blackwells with `device_map="auto"` produces a model that loads clean, reports no error, and computes **garbage**: the residual stream collapses to **exactly zero** a couple of layers past the GPU0→GPU1 boundary, and the logits decode to rubbish (`'8'`, `'�'`, `'b'`). Every layer *below* the boundary stays healthy, deterministic, and bit-identical to a single-GPU run — which is what makes it so dangerous. A capture that reads a low layer looks perfectly plausible and is fine; one that reads a high layer is reading zeros, and nothing in the pipeline says so. Measured 2026-08-20 (§9 Cold-Fusion). **Rule: any workload that reads activations — refusal-direction capture, calibration, activation statistics, PPL — must run on ONE device.** Sharding is for *storage*, and it is only safe when you consume the model's final output through an engine that was built for it (vLLM does TP correctly; `device_map="auto"` in transformers is not the same thing). If it does not fit on one card, shrink the model, not the guarantee: **truncating the decoder to N layers is exact** for any activation read at a layer < N (a causal stack's layer-N state cannot depend on layers above N), and it is cheap — verified by reproducing the full model's layers 18/20/22/26 bit-for-bit. **Gate it, don't remember it.** Assert single-device residency and zero offload before the forward: ```python dmap = getattr(model, "hf_device_map", {}) or {} gpus = {str(v) for v in dmap.values()} - {"cpu", "disk"} offloaded = [k for k, v in dmap.items() if str(v) in ("cpu", "disk")] if len(gpus) > 1 or offloaded: sys.exit("residency gate FAILED — sharded/offloaded forward reads garbage") ``` ### 3.10 ⭐⭐ `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` corrupts retained tensors On torch 2.12+cu130 / Blackwell, tensors that **outlive their allocation** come back corrupted with this flag set: captured hidden states carried Inf / NaN / zeros that **moved between bit-identical forwards** (same input, same weights → a different layer corrupted each time). Unset, the identical forwards are exactly reproducible. Several runbooks recommend this flag for headroom on large loads; for anything that *keeps* activations it buys corruption. Two tells that distinguish this from a real numerical blowup, both worth knowing because they generalise: a genuine blowup **propagates** to later layers and is **deterministic**. Corruption does neither — downstream layers were finite and consistent, and the affected layer moved run to run. **If a "NaN" fails to propagate, stop debugging the math and start debugging memory.** Corollary: **do not read `output_hidden_states=True` off a returned object** on a large multi-device load. Take what you need *during* the forward with a `register_forward_pre_hook` that clones to CPU immediately — it closes the reuse window and never retains a `[B, seq, hidden]` tensor per layer, so it is cheaper than the thing it replaces. ### 3.11 Determinism is a necessary check, not a sufficient one Both defects above were found by the cheapest possible test — **run the same input twice and diff** — which no amount of eyeballing plausible-looking numbers would have caught. Add it to any activation-reading pipeline. But note the trap that followed: after fixing the allocator, the run went perfectly "deterministic" *because the corrupted layers were now stably zero*. Pair the determinism check with a **magnitude** check (residual norms should grow smoothly with depth; an exact 0.0 mid-stack is impossible) and, where you can, a **coherence** check (generate 40 tokens and read them). ### 3.12 ⭐⭐ You cannot free a 27B model in-process — give each model its own process Any A/B that loads two large checkpoints in sequence (KL, logit diffing, teacher-vs-student) will try to release the first before loading the second. **On this stack, it does not work.** Measured 2026-08-20 on Qwen3.8-27B bf16, free VRAM after each attempt: | teardown | free VRAM | |---|---| | `del model` + `gc.collect()` + `torch.cuda.empty_cache()` | 45,287 MiB | | same, with the model confined to an inner frame that exits | 45,287 MiB | | **the process exits** | **97,247 MiB** | The ~51,300 MiB of weights stayed resident through both in-process teardowns. The first run survived only because **PyTorch's allocator hit OOM on the second load, ran a collection itself, and retried** — the second model landed by rescue, not by design. That is not a release strategy: on an architecture where a silent CPU offload does not raise (§3.9), the day the retry does not fire you get confident garbage instead of an error. **Do this instead:** one process per model, hand results to disk between them (first-token log-probs for a 250k vocab are ~715 MiB per model — nothing), and gate each stage on free VRAM *before* the load. Reference implementation: `services/coldfusion-abliteration/kl_divergence.py` (`--stage ref|cand|score`). Two gate corollaries learned in the same session: - **⭐ A residency gate that reads `hf_device_map` cannot fail.** The map is **empty** whenever transformers puts the whole model on one device, so the check reports "unsharded" both when everything is fine and when there is nothing to inspect. Read `{p.device for p in model.parameters()}` — ground truth in every case. (Generalises [[feedback_assert_effective_value_not_substring]]: presence of a passing check is not evidence of a check that can fail.) - **⭐ Size VRAM from the checkpoint's own headers, never from a remembered figure.** A runbook carried "bf16 is 50 GB"; the real number was 50.10 **GiB** = 51,300 MiB of text-only weights. That 3.7 GB unit error is exactly the difference between "stop one co-tenant" and "stop both", and it cost an aborted window. Sum the safetensors header offsets (excluding tensors the loader class won't instantiate — vision, MTP); read only the 8-byte length prefix + JSON header, never `safe_open`, which mmaps the whole shard and ENOMEMs on ZFS (§ *Avoid mmap on `/tank`*). ### 3.13 ⭐⭐ The observer you ASKED for is not necessarily the observer you GOT `quant_mixed_nvfp4.py` sets `observer="imatrix_mse"` on the NVFP4 W4A4 group. It has **never once been used.** llm-compressor looks for importance data, finds none, and silently degrades: ``` _get_validated_importance | WARNING - imatrix_mse: no importance data available. Falling back to uniform MSE. ``` Confirmed on the 2026-08-20 09:59 incumbent quant **and** the 22:45 Heretic-300 quant; `find /tank/aimodels -iname "*imatrix*" -o -iname "*importance*"` returns nothing. Every NVFP4 build in the fleet has run uniform MSE while the recipe claimed importance weighting. **Why it went unseen for months:** the warning scrolls past inside a tqdm progress bar during a ~20 minute quant. It is only visible if you read the log while it runs. **The generalisable rule, which is bigger than imatrix.** A quantizer, optimiser or observer that *silently falls back to a weaker default* is a whole class of invisible quality loss — the config is accepted, nothing errors, the artifact benchmarks plausibly, and you never learn you got the cheap path. So: - **Grep the quant log for `WARNING`, `Falling back`, `not available`, `ignoring` before trusting an artifact.** Make it a step, not a habit. - **Assert the effective setting, never the requested one** — the same rule as [[feedback_assert_effective_value_not_substring]], applied to quantizer internals rather than config files. - If the fallback turns out to be unavoidable in your toolchain version, **change the recipe to say what it actually does.** A recipe line that silently lies is worse than one that admits a limitation. ⚠️ **Do not "fix" this by assuming an imatrix would help.** Verify first that your llm-compressor version can consume an externally supplied importance matrix at all, and in what format. Parked as `park/nvfp4-recipe-asks-for-imatrix-mse-but-silently-2` (id 42) with the calibration corpus that would feed it. ✅ **Comparisons already made remain valid.** Because *every* build shares the fallback, the incumbent-vs-candidate A/Bs (47.2% acceptance, PPL 6.910, and the 2026-08-20 Heretic-300 build) are apples-to-apples. This is unrealised upside, not a correction to past numbers. ### 3.16 Weight-only NVFP4A16 with a minmax observer is DATA-FREE — your calibration corpus is ignored, but its tokenizer side-effect is not Measured 2026-09-08 (Gemma-4 26B-A4B MoE, ERP run 6, llm-compressor 0.13): with `scheme="NVFP4A16"` (default `memoryless_minmax` weights, no activation quant) llm-compressor logs `Inferred DataFreePipeline for QuantizationModifier` and never touches the dataset — the whole 26B quant ran in ~90 s on one Blackwell. Two consequences: (1) do not budget calibration time or believe a corpus "shaped" the result — only `imatrix_mse`/activation observers consume data; (2) building the calibration set still calls the fast tokenizer with `truncation=True, max_length=N`, so §3.14's baked cap (`max_length: 8192` here) lands in the saved `tokenizer.json` **even though no calibration happened**. The §4.3 post-step caught it. Reference: `services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py` (linearize_moe + assert 11,520 expert Linears + post-steps; the published `prithivMLmods/gemma-4-26B-A4B-it-NVFP4A16` recipe replicated, 222→252 ignore entries with audio/norm/router regexes added). ### 3.14 ⭐⭐ Calibration BAKES a truncation cap into the shipped tokenizer **Symptom (on a newer transformers, at startup, on a vision model):** ``` ValueError: Mismatch in `image` token count between text and `input_ids`. Got ids=[2047] and text=[16384]. Likely due to `truncation='max_length'`. ``` The engine never serves a request. The number in `ids=[…]` is your **calibration seqlen minus one**, which is the tell. **Cause — an in-place mutation you never wrote.** Calibration tokenizes like this: ```python tok(b["text"], truncation=True, max_length=seqlen, add_special_tokens=False) ``` For a **fast** tokenizer that call does not just return ids — it **mutates the Rust backend's truncation state in place**. A later `tok.save_pretrained(out)` then persists it: ```json "truncation": {"direction": "Right", "max_length": 2048, "strategy": "LongestFirst", "stride": 0} ``` The source model has `"truncation": null`. **You shipped a tokenizer that clamps every prompt at the calibration length, permanently.** **Why it hid for months.** Older transformers does not enforce the text-vs-ids count check, so the cap sits latent — the model serves, gates pass, vision works, nothing logs. It only detonates when you bump the image, and then it presents as a *vision* bug at startup with no mention of tokenizers. It also caps the effective image resolution long before it kills the seat: at a 2048 cap the largest servable image is ~1448×1448, because `(edge/patch)² / merge²` image tokens must fit under it. **The fix — never save the calibration tokenizer.** Re-read a pristine one from the source: ```python from transformers import AutoTokenizer as _AutoTokenizer _AutoTokenizer.from_pretrained(a.model, trust_remote_code=True).save_pretrained(a.out) ``` then **assert** it, because this is exactly the class of defect that returns silently: ```python if json.load(open(f"{a.out}/tokenizer.json")).get("truncation"): raise SystemExit("FAILED CHECK: saved tokenizer carries a truncation cap") ``` Both live in `quant_mixed_nvfp4.py` as of 2026-08-22. **Audit any build predating that.** One line per model: ```bash python3 -c 'import json,sys;print(json.load(open(sys.argv[1]+"/tokenizer.json")).get("truncation"))' ``` Measured 2026-08-22 — every mixed-NVFP4 build from this pipeline was affected, and the two live ones were corrected in place (backup `tokenizer.json.bak-truncation-20260822`; only the `truncation` field changed, vocab and `added_tokens` byte-identical): | build | truncation as found | |---|---| | `qwen38-27b-orcarouter-nvfp4-mixed` (live `gen`) | **2048** → fixed | | `mog-sec-27b-nvfp4-mixed` (live `sec`) | **2048** → fixed | | `qwen38-27b-heresy-nvfp4-mixed` (retired) | 2048, left as-is | | `G4-MeroMero-v2-31B-NVFP4A16` (different pipeline) | `null` ✓ | | `mog-sec-27b-bf16` (source) | `null` ✓ | **Editing it is safe on a running seat** — vLLM reads the tokenizer at startup and holds its own copy, so the fix lands on the next restart with no disruption. **The general lesson, which is the transferable part:** this is the third defect in this playbook where *the artifact carries config authored against an older transformers and a newer one starts enforcing it* (see also the Gemma-4 heterogeneous `head_dim`). **Treat "we bumped the image" as a config-compatibility event, not just a version change** — and prefer saving artifacts re-read from the source over saving objects the pipeline has touched. --- ## 4. Pipeline shape ### 4.1 Prove the targets before spending GPU time Enumerate module names from the safetensors index and check your regexes against them: **zero overlap between groups, and the union covers every layer you intended.** This is free, takes seconds, and catches a mis-scoped regex that would otherwise surface as a mystery quality regression hours later. Reference: `services/gen-seat-mixed-quant/validate_targets.py`. ### 4.2 Quantize Calibration data matters for `imatrix_mse` + static activation observers. We use `/tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl` (512 chat samples, RP/GM-flavoured — appropriate for our seats). 256 samples @ 2048 tokens ≈ 20 min for a 27B on one Blackwell. ### 4.3 The mandatory post-steps Never optional, always in this order, and the last one **verifies rather than assumes**: 1. Graft `model-mtp.safetensors` + register its tensors in the output index. 2. Restore `preprocessor_config.json` / `processor_config.json` / `video_preprocessor_config.json`. 3. **Re-inject `re:^mtp.*` into `quantization_config.ignore` and confirm it is there** (§3.3). 4. **Confirm the saved `tokenizer.json` has `truncation: null`** (§3.14) — calibration mutates the fast tokenizer in place and `save_pretrained` bakes the cap in. Latent on an older transformers, fatal on a newer one. Reference implementation: `services/gen-seat-mixed-quant/post_quant.py`. ### 4.4 Test on a temp port, never on the live seat Serve the candidate on an alt port with the live seat's **exact** flags, run the gate (§5), and only then flip `.env`. Keep the previous build on disk; rollback is one `.env` line. --- ## 5. The acceptance gate — and how measurement lies to you Speed alone does not justify cutting over a shared seat. Gate on **all** of: decode tok/s, MTP acceptance, perplexity, a behavioural surface test, and — for an abliterated model — that the abliteration survived. **Three ways the numbers have lied to us. All three produced confident, wrong results.** 1. **Prefix caching fakes both speed metrics.** A fixed prompt returns byte-identical timings run after run; you are measuring cache, not compute. Worse for prefill: a *seeded* nonce regenerates the previous run's prompts verbatim and reads **~41k tok/s of cache-hit instead of ~5k of real prefill**. Use a fresh unseeded nonce per request; never seed a cache-buster. 2. **`prompt_logprobs` are garbage while speculative decoding is on** — ~uniform over the vocab (median rank ~10⁵; " Paris" after "The capital of France is" ranked 69698). **Perplexity must be measured on a seat served without `--speculative-config`,** on both sides of the comparison. 3. **A 0600 `.env` makes `docker compose` silently no-op.** Without `sudo` it fails `permission denied` reading `.env`, **leaves the old container running**, and reports success — producing a full page of "benchmark results" that were just the unchanged baseline. **Hard-verify the change landed against `docker inspect …Config.Cmd`.** **Re-measure the baseline before believing a target.** The 2026-08-15 handoff quoted ~68 tok/s; cache-busted, the incumbent was already doing 80.1 — essentially the *target* of the work queued against it. Had that not been re-measured, doing nothing would have looked like a 20% win. **Cheap shortcut worth taking first:** if a reputable published quant of the same architecture is already on-box (or is a small pull), **serve it as a probe and measure it** before committing hours to your own. It answers "is this gain even real?" in ten minutes *and* hands you the recipe. Harness: `services/gen-seat-mixed-quant/bench/` — `quickbench.py` (decode + acceptance), `prefill_bench.py`, `eval_quality.py` (PPL + abliteration), `surface_test.py` (chat, vision, tools, thinking split, long-context needle, streaming), `serve_probe.sh`. --- ### 5.1 ⭐⭐ Acceptance is not throughput — always run the DEPTH control **Measured 2026-08-22**, same instrument (vLLM's own `spec_decode` counters, delta over a fixed workload, temp 0), same target, same engine: | config | accepted tok/forward | throughput | |---|---|---| | MTP k=3 | 2.753 | 114.9 tok/s | | MTP k=7 | **3.041** ⬆ | **74.0 tok/s** ⬇ | **Raising `num_speculative_tokens` improved acceptance and destroyed throughput.** Reporting acceptance alone would have recommended a 36% regression. **Why:** a single-module MTP head (`mtp_num_hidden_layers: 1`, one `mtp.layers.0`) has no depth of its own — vLLM runs it **autoregressively**, so k draft tokens cost **k sequential forward passes**. Past a shallow depth the drafting cost exceeds what the extra accepted tokens save. Check `mtp_num_hidden_layers` before assuming depth is cheap. **The rule: when comparing two speculative methods, match k, or you are measuring depth rather than method.** A parallel-drafting drafter (DFlash2 and kin, which propose a whole block in one pass) at k=7 versus an autoregressive MTP at k=3 is not a method comparison — the depth control is what separates them. In our case the control showed most of the apparent acceptance win was depth, while the *throughput* win was real and came from parallel drafting, not better drafts: our MTP was **better at position 0** (79.6% vs 75.4%) and still lost overall. **Corollary — report both, always.** Acceptance rate, mean accepted length, and end-to-end tok/s. Any one of the three alone can point the wrong way. --- ## 6. Hardware and co-residency - **ana-ml2 = Blackwell (sm_120)**, 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths (`match_exact=True` on sm90) are **closed** here — do not plan around them. - **irv-ml1 = Ampere (sm_86)**, 3090 + A6000. **No native FP8/FP4** — 4-bit there is a VRAM saving only, not a speed win. Don't port a Blackwell scheme over and expect the throughput. - **GPU co-residency is a zero-sum budget, and a *smaller* model can break its neighbour.** `gpu-memory-utilization` is a fraction of the *whole card*, so when new weights are smaller the seat absorbs the slack as extra KV rather than releasing it. That is exactly how a −5.2 GB requant left the co-resident seat **0.18 GiB** short and crash-looping. **After any requant, re-check both seats' budgets** and hand the space back explicitly. --- ## 7. Superseded claims — do not follow these Old docs stay for their history, but these specific claims are **false now** and will cost you a day if followed: | claim | where | status | |---|---|---| | "Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance" | `docs/runbooks/heretic2-nvfp4-mtp-seat.md` §landmine 2 | **SUPERSEDED 2026-08-14.** The 0% was the missing `re:^mtp.*` ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.7–83.2% acceptance, live. Use compressed-tensors. | | "Abliteration desyncs the MTP head → uncensored models can't do MTP" | earlier auto-memory | **SUPERSEDED 2026-08-14.** A modest abliteration preserves MTP (83.7% at bf16). Test MTP on **bf16 first** to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source. | | "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). | | "The Qwen3.8 MTP corruption is fixed by disabling prefix caching while keeping MTP; the gen seat runs APC-off" | this playbook §3.7 (now marked), earlier auto-memory | **SUPERSEDED 2026-08-17, and the staleness was only caught 2026-08-26.** APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use; reverted the same day. The real cause was the **AEON W4A4 quant being defective** (~15-20% degenerate generations), with MTP / prefix-caching / gateway merely AMPLIFYING it (§3.8). The gen seat runs **MTP ON and prefix-caching ON** on the in-house mixed NVFP4+FP8 build — verified against the live container 2026-08-26. ⚠ The lesson inside the lesson: a *passing multi-turn probe* was not sufficient evidence either. | | "transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 without causal-conv1d; it is precision-driven cancellation and fp32 resolves it" | `services/coldfusion-abliteration/README.md`, `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` | **SUPERSEDED 2026-08-20.** Precision was never the variable. The NaN came from **multi-GPU sharding** and **`expandable_segments`** (§3.9, §3.10); fp32 only made it rarer, which is worse than failing. On one GPU with a plain allocator, **bf16 is exactly deterministic through all 64 layers and generates coherent prose** — at 50 GB and 4.3× the throughput of the 111 GB fp32 it replaced. | --- ## 8. Measured negatives — don't re-chase - **`num_speculative_tokens` = 3 is optimal** on the Qwen3.8-27B seat. Swept: n=2 → 77.1, **n=3 → 80.1**, n=4 → 78.7, n=5 → 75.9 tok/s. Higher n trades acceptance for draft width and loses. Re-sweep only if the drafter architecture changes. - **Uniform W4A4** — see §7 row 3. - **Dense-VL as the anatomy judge** — A/B'd, MoE retained. Don't re-propose. --- ## 9. Worked examples Per-model artifacts. Read for *how a specific model went*, not for the general lessons — those are above, and where the two disagree, **this file wins**. | artifact | what it is | |---|---| | `services/gen-seat-mixed-quant/` | **current reference.** Mixed NVFP4+FP8 on Qwen3.8-27B-Uncensored: scripts, acceptance harness, raw measurements. | | `stacks/gen-seat/README.md` | the live `gen` seat (7 LiteLLM aliases) | | `stacks/meromero-charrp/README.md` | Gemma-4 seat — the **tool-call/reasoning-parser** trap (a parser default that returns null `content` for all prose) | | `services/heretic2-nvfp4-quant/` | modelopt-format MTP seat — historical; see §7 before following it | | `tools/mistral-small4-nvfp4/` | MoE + native-convert path; source of §3.6 | | `docs/pfi/recommended-model-settings.md` | serve-time sampler/flag defaults (not quant) | **A new model just dropped and needs requanting?** §1 → §2 → §4 → §5. Skim §3 first; it is the part that costs hours.