# Model quantization playbook — the lessons that keep costing us hours **Read this before starting any new quant.** Not the per-model runbooks — those are worked examples of a *specific* model at a *specific* point in time, and several carry claims that are now false (see §7). This file owns the **transferable** part: what recurs regardless of which model dropped this week. Written 2026-08-15, after the fourth quant in five weeks re-discovered the third-known instance of the same loader-class bug. Scope: NVFP4 / FP8 / mixed-precision on the Blackwell boxes (ana-ml2), vLLM-served. Ampere (irv-ml1) has no native FP4/FP8 — see §6. **Maintenance rule.** When a quant teaches you something *model-agnostic*, it lands here and the per-model README links up. When it's model-specific (this checkpoint's odd tensor names, this finetune's missing config), it stays in the per-model artifact. If you find yourself writing a "Gotchas" section that repeats §3, you are re-litigating — add the delta here instead. --- ## 1. The 60-second decision: which scheme On Blackwell + vLLM, for a dense-or-hybrid VL model you intend to serve at long context: | want | scheme | notes | |---|---|---| | **default, best speed/accuracy** | **mixed: NVFP4 W4A4 bulk MLPs + FP8 W8A8 attention/`lm_head`/last-8-layer MLPs** | the current answer. §2. | | max fidelity, don't care about prefill | NVFP4 **W4A16** (weight-only) | forces the **Marlin** kernel — ~half the prefill of native FP4 | | small model, VRAM is free | FP8 **W8A8** | safe and simple; 2× the weight bytes of 4-bit | | — | ~~"W4A8" = NVFP4 weights + FP8 activations~~ | **DOES NOT EXIST.** §3.1 | **Measured on Qwen3.8-27B (2026-08-15), W4A16 → mixed:** decode +18%, prefill **+78–98%**, MTP acceptance unchanged, perplexity +1.7%, weights −19%. Note the shape of that: **decode barely moves, prefill nearly doubles.** Decode at batch-1 is memory-bandwidth-bound and the weights are 4-bit under either scheme, so there is little to win; prefill is compute-bound, which is where native FP4 tensor cores replace the Marlin dequantize-to-BF16 path. If someone promises you a big *decode* win from a scheme change, be skeptical — and go measure §5 before believing it. **The accuracy cost is real and is paid on purpose.** Operator ruling 2026-08-15: the ~1.7% perplexity is an acceptable price for the speed. Settled — don't re-litigate. For correct attribution: it is the **activation**-quantization cost (A4/A8 vs BF16 activations), *not* an MTP cost. Turning MTP off does not recover it; only reverting the quant does. --- ## 2. The reference recipe (mixed-precision) Lifted from `unsloth/Qwen3.8-27B-NVFP4` and replicated in-house. **Prefer replicating a published recipe from a reputable quantizer over inventing one** — they have already paid for the sensitivity analysis. | group | scheme | targets | |---|---|---| | `group_0` | FP8 W8A8 — channel weights (static) + per-token dynamic activations | `self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`, `lm_head`, **the last 8 layers' MLPs** | | `group_1` | NVFP4 W4A4 — `tensor_group` gsize 16, fp8 scales, `imatrix_mse` weights, `dynamic:"local"` activations | **all remaining** MLP `{gate,up,down}_proj` | | kv cache | FP8 static tensor | | | ignore | vision tower, `linear_attn.{norm,in_proj_a,in_proj_b}`, `re:^mtp.*` | | Three things in there are load-bearing and easy to drop: - **Late layers stay FP8.** Holding the last ~8 layers' MLPs (and `lm_head`) at 8-bit is the accuracy-preservation trick — late layers are the sensitive ones. Uniform W4A4 is what collapses. - **`imatrix_mse` on the W4A4 weights**, not `memoryless_minmax`. Importance-weighted; needs calibration data. - **Group targets must be non-overlapping.** Do not let `group_1`'s `.*mlp\..*` also match the late layers and rely on group precedence to sort it out. Enumerate the early layers explicitly (`re:.*layers\.([0-9]|[1-4][0-9]|5[0-5])\.mlp\.…`) and **prove it** with a dry run (§4.1). **Toolchain:** `pip install llmcompressor` into stock `vllm/vllm-openai:latest` gives llmcompressor 0.13 + compressed-tensors 0.18 without disturbing torch/transformers. **Avoid nvidia-modelopt** — see §3.4. --- ## 3. The recurring landmines Ordered by how much time each has cost. Every one of these has bitten more than once. ### 3.1 "W4A8" is not a servable shape vLLM's compressed-tensors dispatcher (`compressed_tensors.py:704-713`) accepts NVFP4 weights with **exactly two** activation settings: | `input_activations` | result | |---|---| | `None` | W4A16 — and it **forces the Marlin kernel** (`kernels/linear/__init__.py:881-883`) | | NVFP4 | W4A4, native | Anything else — **FP8 included** — raises at load: ``` ValueError: For NVFP4 weights, input quantization must also be NVFP4 format, None for NVFP4A16 ``` `CompressedTensorsW4A8Fp8` exists but is **INT4** weights (`W4A8_SUPPORTED_TYPES_MAP = {4: int4}`) gated on `_check_scheme_supported(90, match_exact=True)` — Hopper-exact, so on Blackwell (sm_120) it is closed twice over. **FP8 enters per-layer-group, never as activations on NVFP4 weights.** *Cost: one queued task written against an impossible scheme.* ### 3.2 Wrong loader class → silent weight-load failure **Rediscovered three times.** Load the model through the class vLLM actually serves — the `…ForConditionalGeneration` / `…ForImageTextToText` **wrapper**, never `AutoModelForCausalLM`. `AutoModelForCausalLM` resolves a VL config to the text-only inner class and saves a **flat** config with `model.layers.*` keys. vLLM's weight mapper wants `model.language_model.*` (+ `model.visual.*`). The mismatch does not error — **every layer silently fails to load** and you get `!!!!` gibberish, or an engine that rejects the checkpoint outright. *Bit: heretic2 (gibberish), Dark-Scarlett (both vLLM and SGLang refused the checkpoint), and the 2026-08 rounds.* ### 3.3 The MTP head — three separate ways to lose it Speculative decoding is a large fraction of the seat's throughput. It fails **silently**: the model serves fine, just at 0% acceptance. 1. **The wrapper class does not instantiate `mtp.*`,** so the quant drops it. Post-quant you must graft the BF16 `model-mtp.safetensors` back and register its tensors in the output index. 2. **`re:^mtp.*` must be in `quantization_config.ignore`** — else vLLM loads the grafted BF16 head as though quantized, it comes up **uninitialised**, and acceptance is 0%. 3. **⭐ llm-compressor PRUNES `ignore` entries that matched no module at quant time.** Since the wrapper never loaded `mtp.*`, the entry matches nothing and is **silently deleted from the saved config — even though you put it in the recipe.** So it must be re-injected *after* the graft, and then **verified, not assumed.** *Cost: three rounds. The verify step caught it live on the third.* There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not propagate modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix (needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4. ### 3.8 ⭐⭐ Multi-turn degeneration from TWO real compounding causes — how they masked each other The most expensive diagnosis this project has had, because there were **two real causes at once** and each partial fix moved the needle enough to look like *the* answer. Recorded precisely because the first write-up of this section over-attributed it to the quant alone; that was wrong. **Cause 1 (real, upstream): the vLLM `qwen3_5_mtp` × Gated-DeltaNet bug.** Confirmed by two cross-frontier peers and the tracker (vllm#47087 symptom-twin, #43559 fix lineage, #51113 fix): the GDN recurrent state cannot roll back on a partial draft-accept, so speculative decoding corrupts it, worse with context. Architectural — vLLM/SGLang/llama.cpp mainline all shared it. **Genuinely fixed enough** by moving to vLLM **nightly** (`v0.27.2rc1.dev150+`, carries #51113): the operator reported it "significantly better" — this was a real bug, not just an amplifier. **Cause 2 (real, quant): full W4A4 is mildly subpar, per the known gradient.** `sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4` is **full** W4A4 — 4-bit *activations* on attention too, the bottom of the activation-precision ordering already in §1: **W4A4 (A4) < W4+FP8 (A8) < W4+bf16 (A16)**. Not "defective," just lowest-fidelity; on top of Cause 1 it degenerated ~15-20% of real multi-turn generations. The FP8-attention **mixed** build (`qwen38-27b-uncensored-nvfp4-mixed`, same base, same MTP, same nightly) sits a rung up that gradient and is coherent. AEON was purged 2026-08-17 (operator ruled it no-good; re-pullable from HF). **Why it cost days — and the process lessons that stand:** 1. **Two real causes compound and mask each other.** Each mitigation (MTP-off, APC-off, the nightly #51113 fix) partially helped, so each looked like the fix and then failed in real use. When a mitigation "helps but doesn't fix," suspect a *second* cause rather than a wrong one. 2. **Stochastic degeneration (~15-20%) is nearly invisible to a small synthetic probe** — a 7-turn run passes ~4 in 5. n=1 "clean" proves nothing; this class needs many runs or the operator's real high-volume use. Three non-fixes were "validated" by a single clean probe here. 3. **Isolate the WEIGHTS in parallel with the serving flags, not after.** Swapping to a different quant of the same base (AEON→mixed) is what finally separated Cause 2 from Cause 1; doing it earlier would have shortened the hunt. But note it would NOT have found Cause 1 — the vLLM bug was real and needed the nightly. 4. **Prefer FP8 attention (the §2 mixed recipe) over full W4A4** for a coherence- sensitive seat. AEON passed every static gate (abliteration 4/4, surface 6/6, a 36k needle, 52% acceptance) and was still the lower-fidelity of the two. Current primary gen: the mixed FP8-attention build on pinned vLLM nightly with MTP, until the DavidAU Qwen3.8 lands. A W4+bf16 (W4A16) build would be higher fidelity still (§1) at a prefill cost — an option if the mixed build ever proves marginal. ### 3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn §3.3 is about *losing* the head (0% acceptance, silent). This is the opposite and worse failure: the head loads, acceptance looks healthy, single-turn output is perfect — and then it **corrupts multi-turn conversations** once cumulative context passes **~2,000 tokens**. The reply collapses in length *and* bleeds earlier turns into the current answer (a "describe durian" reply that contained the Krebs-cycle and winter answers from three turns back). Single-turn probes and the acceptance gate (§5) **do not catch it** — it only appears as accumulated context grows. Isolated 2026-08-16 (operator-confirmed), each step measured on a fixed 7-turn probe: - **Not the serving gateway, not sampling, not repetition/template.** Identical input gateway-vs-direct behaves the same; presence_penalty 1.5/0.5/0.0 all collapse; higher temperature collapses harder; a conversation of *unrelated* topics collapses at the same ~2k tokens as a repetitive one → it is context- length-driven, not template lock-in. - **Model-independent across every Qwen3.8-27B quant** (AEON W4A4, unsloth FP8-attn, our in-house mixed) — so not a quant-brand or scheme artifact. - **DECISIVE: same model + same conversation, MTP OFF → coherent through 4k+ tokens, zero bleed.** Toggle it back on → collapse returns. MTP is the cause. **Qwen3.6-27B running the same `qwen3_5_mtp` method is CLEAN.** So the 3.6 MTP head/graft is fine and the 3.8 one is not — suspects: the bf16 graft being subtly wrong for the 3.8 head, or the vLLM `qwen3_5_mtp` impl diverging at `num_speculative_tokens=3`. Open upstream question (queried dvalin/bil-smithy 2026-08-17). **Rule: gate MTP on a MULTI-TURN coherence probe, not just single-shot acceptance.** Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens for length-collapse and cross-turn bleed. ⛔ **THE APC-OFF MITIGATION DID NOT HOLD — SUPERSEDED 2026-08-17, see §7.** What follows is kept for its history and **must not be applied**: *"disable prefix caching, keep MTP. The corruption is gated on MTP × prefix-caching together (vllm#43559 / #47194); with `--no-enable-prefix-caching` the buggy partial-accept align-path is inert. Confirmed on our stack: AEON W4A4, MTP on + prefix-caching off → the 7-turn varied series stays coherent through 3.9k tokens, zero bleed, at 104.6 tok/s / 53.6% acceptance."* **It passed that synthetic 7-turn probe and the operator still saw severe degeneration in real use.** Reverted the same day. The probe was structurally under-covering the real workload on both content distribution and depth — which is §3.7's own standing rule (*gate on a multi-turn coherence probe, not single-shot acceptance*) failing at one level up: the multi-turn probe was itself too small to gate on. **WHAT ACTUALLY RESOLVED IT.** The multi-day hunt root-caused to the **AEON W4A4 quant being defective** — ~15-20% of generations went degenerate — with MTP, prefix-caching and the gateway all merely *amplifying* it. That is why every partial mitigation "helped" without fixing anything (§3.8). The gen seat today runs the in-house Heretic **mixed NVFP4+FP8** build (FP8 attention, not W4A4) on vLLM nightly carrying #51113, with **MTP ON and prefix-caching ON**, and is coherent in real use. Verified against the live seat 2026-08-26: `vllm-gen` runs `--enable-prefix-caching` with `qwen3_5_mtp` / `num_speculative_tokens 3`. The compose file (`stacks/gen-seat/compose.yaml`) carries the full history inline and is the current authority; this section was stale against it for nine days. Things that do **not** work, ruled out: `num_speculative_tokens=1` (corruption is depth-independent — reproduces at n=1 and n=2, deterministically probed upstream); switching engine (vLLM / SGLang / llama.cpp mainline all share the GDN-rollback bug — it is architectural). The proper upstream fix (vllm#51113) is in `main` / `v0.27.2rc0` only — not in a stable release, so we hold at APC-off until it lands. Two cross-frontier peers (dvalin/bil-smithy) confirmed the bug class and pointed at the open symptom-twin issue #47087. ### 3.15 ⭐⭐ Fused 3-D MoE experts are INVISIBLE to a `targets=["Linear"]` recipe **Symptom: none.** The quant completes, the artifact loads, and 88.5% of the model is still BF16. Nothing warns you. Modern MoE checkpoints store each layer's experts as **two fused 3-D `nn.Parameter` tensors**, not as N `nn.Linear` modules. Gemma-4 26B-A4B: model.language_model.layers.N.experts.gate_up_proj BF16 [128, 1408, 2816] model.language_model.layers.N.experts.down_proj BF16 [128, 2816, 704] Note the **absent `.weight` suffix** — that is the tell. `mlp.down_proj.weight` is an `nn.Linear`; `experts.down_proj` is a bare parameter. Measured on that checkpoint, recipe targeting `["Linear"]`: Linear modules 427 WILL quantize 205 (experts: 0) <- 22.84 B params untouched **This is the same defect that killed QLoRA on this architecture** — `bitsandbytes` 4-bit replacement also walks `nn.Linear` modules and also silently skipped the experts. Two different tools, one blind spot, because the blind spot is in the *checkpoint layout*, not the tool. **The fix** (llm-compressor ≥ 0.12): ```python from llmcompressor.modeling.moe.linearize import linearize_moe model = SomeForConditionalGeneration.from_pretrained(...) linearize_moe(model) # BEFORE building the recipe ``` Linear modules 11947 WILL quantize 11725 (experts: 11520) # 30 layers x 128 x 3 proj `linearize_moe` unfuses the 3-D parameters into per-expert `experts.N.{gate,up,down}_proj` Linears. **No registration is needed** if the module satisfies `FusedExpertsProtocol` structurally — bare `down_proj` plus `gate_up_proj`/`up_proj` Parameters. `load_quantizable_moe(model_cls)` is the faster variant that linearizes during load rather than after. **Always assert the expert count before spending GPU time** (§4.1). The arithmetic is `layers × experts × projections`; if your target list does not hit it exactly, the recipe is wrong and the failure is silent. ⚠ **Keep routers in `ignore`.** A 4-bit router picks *different experts* — that error does not average out downstream, it changes which weights run at all. **And here is what that actually looks like when it ships — measured 2026-09-10 on the MeroMero 26B-A4B, which reached a live seat before anyone noticed.** The A4B was quantized with the *dense* recipe (`services/meromero-quant/quant_a16_datafree.py`), whose IGNORE list has no `re:.*router.*` entry. All 30 routers went to NVFP4. Then: - the quant **completed cleanly**, 16 G, no warning; - the tensor table looked **plausible** — 11,755 quantized modules against 11,725 in the known-good build, a 0.26% difference nobody eyeballs; - vLLM **started, passed its healthcheck, and reported the correct KV pool**; - every request returned `finish_reason: "length"` with the **full completion_tokens count** — 120 of 120, 600 of 600 — and `content: null`. The model was generating, and every token decoded to the empty string; - the give-away was **`logprobs` coming back NaN**, which surfaced only because a diagnostic asked for them and vLLM refused to serialize the response: `Out of range float values are not JSON compliant: nan`. ⚠⚠ **So the router mistake has no symptom you would catch by watching a seat come up.** It has exactly one cheap tell, and §4.4 now carries it: ask for `logprobs` once. **Two guards, both cheap, both would have caught this before the seat:** 1. **Use the architecture-class-correct recipe and let its guard fire.** `services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py` refuses outright — *"⚠ REFUSING: a router/vision/audio Linear is in the quantize set"* — and asserts `layers × experts × 3` before any GPU time. Its `--dry-run` does the whole check with no GPU and no save. The dense recipe has neither guard and will happily eat a MoE. 2. **Diff `quantization_config.ignore` against a known-good quant of the SAME ARCHITECTURE CLASS.** The broken build had **222** ignore entries against the good build's **252**; the 30 missing were exactly `model.language_model.layers.N.router.proj`. That diff is a two-line script and it names the defect precisely. ⚠ **The control has to match the architecture class, not just the family.** The broken A4B *was* structurally diffed before it shipped — against a verified-good **dense** 31B quant of the same Gemma-4 family. A dense model has no routers, so the one thing that was wrong was the one thing that control could not see, and the comparison came back clean. A positive control is only worth what it can distinguish; "same family" is not "same architecture class". ### 3.4 Toolchain version deadlocks Both directions have burned us, so the resolution is: **use llm-compressor / compressed-tensors, not nvidia-modelopt.** - modelopt **0.45** ↔ transformers 5.12: `mtq.quantize` dies `TypeError: issubclass() arg 2 must be a class` (modelopt registers transformers' `FusedMoE`, a *function* in 5.x, as an nn class). - modelopt **0.43** doesn't fix it — it drags transformers back to 4.57, which cannot load `qwen3_5` at all. - modelopt's config API also trails the current model families by a version. ### 3.5 Vision tower and its configs - Keep the **vision tower in `ignore`** (BF16). Only the LLM backbone gets quantized. - The wrapper-class save **drops `preprocessor_config.json`** (and the video one). Without it the seat crash-loops `Can't load image processor`. Restore from the source — and if the upstream repo omits it, **reconstruct it from `processor_config.json`'s `image_processor` sub-dict**. ### 3.6 Memory and device placement (large models) - **`device_map=None`/`"cpu"`, never `"auto"`.** `auto` fills GPU0 and OOMs during un-fusing; constraining with `max_memory` then offloads to the *meta* device, which cannot be `.copy_()`d. CPU-resident keeps every tensor real; the sequential pipeline still onloads per-layer to GPU. - **Avoid mmap on `/tank`.** `safetensors.safe_open()` mmaps a whole shard; on ZFS a 50 GB shard ENOMEMs regardless of free RAM (MAP_SHARED never consults the commit limit). Read with plain `read()` + `load(bytes)`, one shard cached at a time. - **`vm.overcommit_memory=1`** on ana-ml2 (durable via `playbooks/ana-ml2-overcommit-memory.yaml`). ### 3.9 ⭐⭐ A sharded forward can be silently WRONG — never trust `device_map="auto"` for activations Splitting **Qwen3.8-27B (Qwen3_5 hybrid)** across the two Blackwells with `device_map="auto"` produces a model that loads clean, reports no error, and computes **garbage**: the residual stream collapses to **exactly zero** a couple of layers past the GPU0→GPU1 boundary, and the logits decode to rubbish (`'8'`, `'�'`, `'b'`). Every layer *below* the boundary stays healthy, deterministic, and bit-identical to a single-GPU run — which is what makes it so dangerous. A capture that reads a low layer looks perfectly plausible and is fine; one that reads a high layer is reading zeros, and nothing in the pipeline says so. Measured 2026-08-20 (§9 Cold-Fusion). **Rule: any workload that reads activations — refusal-direction capture, calibration, activation statistics, PPL — must run on ONE device.** Sharding is for *storage*, and it is only safe when you consume the model's final output through an engine that was built for it (vLLM does TP correctly; `device_map="auto"` in transformers is not the same thing). If it does not fit on one card, shrink the model, not the guarantee: **truncating the decoder to N layers is exact** for any activation read at a layer < N (a causal stack's layer-N state cannot depend on layers above N), and it is cheap — verified by reproducing the full model's layers 18/20/22/26 bit-for-bit. **Gate it, don't remember it.** Assert single-device residency and zero offload before the forward: ```python dmap = getattr(model, "hf_device_map", {}) or {} gpus = {str(v) for v in dmap.values()} - {"cpu", "disk"} offloaded = [k for k, v in dmap.items() if str(v) in ("cpu", "disk")] if len(gpus) > 1 or offloaded: sys.exit("residency gate FAILED — sharded/offloaded forward reads garbage") ``` ### 3.10 ⭐⭐ `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` corrupts retained tensors On torch 2.12+cu130 / Blackwell, tensors that **outlive their allocation** come back corrupted with this flag set: captured hidden states carried Inf / NaN / zeros that **moved between bit-identical forwards** (same input, same weights → a different layer corrupted each time). Unset, the identical forwards are exactly reproducible. Several runbooks recommend this flag for headroom on large loads; for anything that *keeps* activations it buys corruption. Two tells that distinguish this from a real numerical blowup, both worth knowing because they generalise: a genuine blowup **propagates** to later layers and is **deterministic**. Corruption does neither — downstream layers were finite and consistent, and the affected layer moved run to run. **If a "NaN" fails to propagate, stop debugging the math and start debugging memory.** Corollary: **do not read `output_hidden_states=True` off a returned object** on a large multi-device load. Take what you need *during* the forward with a `register_forward_pre_hook` that clones to CPU immediately — it closes the reuse window and never retains a `[B, seq, hidden]` tensor per layer, so it is cheaper than the thing it replaces. ### 3.11 Determinism is a necessary check, not a sufficient one Both defects above were found by the cheapest possible test — **run the same input twice and diff** — which no amount of eyeballing plausible-looking numbers would have caught. Add it to any activation-reading pipeline. But note the trap that followed: after fixing the allocator, the run went perfectly "deterministic" *because the corrupted layers were now stably zero*. Pair the determinism check with a **magnitude** check (residual norms should grow smoothly with depth; an exact 0.0 mid-stack is impossible) and, where you can, a **coherence** check (generate 40 tokens and read them). ### 3.12 ⭐⭐ You cannot free a 27B model in-process — give each model its own process Any A/B that loads two large checkpoints in sequence (KL, logit diffing, teacher-vs-student) will try to release the first before loading the second. **On this stack, it does not work.** Measured 2026-08-20 on Qwen3.8-27B bf16, free VRAM after each attempt: | teardown | free VRAM | |---|---| | `del model` + `gc.collect()` + `torch.cuda.empty_cache()` | 45,287 MiB | | same, with the model confined to an inner frame that exits | 45,287 MiB | | **the process exits** | **97,247 MiB** | The ~51,300 MiB of weights stayed resident through both in-process teardowns. The first run survived only because **PyTorch's allocator hit OOM on the second load, ran a collection itself, and retried** — the second model landed by rescue, not by design. That is not a release strategy: on an architecture where a silent CPU offload does not raise (§3.9), the day the retry does not fire you get confident garbage instead of an error. **Do this instead:** one process per model, hand results to disk between them (first-token log-probs for a 250k vocab are ~715 MiB per model — nothing), and gate each stage on free VRAM *before* the load. Reference implementation: `services/coldfusion-abliteration/kl_divergence.py` (`--stage ref|cand|score`). Two gate corollaries learned in the same session: - **⭐ A residency gate that reads `hf_device_map` cannot fail.** The map is **empty** whenever transformers puts the whole model on one device, so the check reports "unsharded" both when everything is fine and when there is nothing to inspect. Read `{p.device for p in model.parameters()}` — ground truth in every case. (Generalises [[feedback_assert_effective_value_not_substring]]: presence of a passing check is not evidence of a check that can fail.) - **⭐ Size VRAM from the checkpoint's own headers, never from a remembered figure.** A runbook carried "bf16 is 50 GB"; the real number was 50.10 **GiB** = 51,300 MiB of text-only weights. That 3.7 GB unit error is exactly the difference between "stop one co-tenant" and "stop both", and it cost an aborted window. Sum the safetensors header offsets (excluding tensors the loader class won't instantiate — vision, MTP); read only the 8-byte length prefix + JSON header, never `safe_open`, which mmaps the whole shard and ENOMEMs on ZFS (§ *Avoid mmap on `/tank`*). ### 3.13 ⭐⭐ The observer you ASKED for is not necessarily the observer you GOT `quant_mixed_nvfp4.py` sets `observer="imatrix_mse"` on the NVFP4 W4A4 group. It has **never once been used.** llm-compressor looks for importance data, finds none, and silently degrades: ``` _get_validated_importance | WARNING - imatrix_mse: no importance data available. Falling back to uniform MSE. ``` Confirmed on the 2026-08-20 09:59 incumbent quant **and** the 22:45 Heretic-300 quant; `find /tank/aimodels -iname "*imatrix*" -o -iname "*importance*"` returns nothing. Every NVFP4 build in the fleet has run uniform MSE while the recipe claimed importance weighting. **Why it went unseen for months:** the warning scrolls past inside a tqdm progress bar during a ~20 minute quant. It is only visible if you read the log while it runs. **The generalisable rule, which is bigger than imatrix.** A quantizer, optimiser or observer that *silently falls back to a weaker default* is a whole class of invisible quality loss — the config is accepted, nothing errors, the artifact benchmarks plausibly, and you never learn you got the cheap path. So: - **Grep the quant log for `WARNING`, `Falling back`, `not available`, `ignoring` before trusting an artifact.** Make it a step, not a habit. - **Assert the effective setting, never the requested one** — the same rule as [[feedback_assert_effective_value_not_substring]], applied to quantizer internals rather than config files. - If the fallback turns out to be unavoidable in your toolchain version, **change the recipe to say what it actually does.** A recipe line that silently lies is worse than one that admits a limitation. ⚠️ **Do not "fix" this by assuming an imatrix would help.** Verify first that your llm-compressor version can consume an externally supplied importance matrix at all, and in what format. Parked as `park/nvfp4-recipe-asks-for-imatrix-mse-but-silently-2` (id 42) with the calibration corpus that would feed it. ✅ **Comparisons already made remain valid.** Because *every* build shares the fallback, the incumbent-vs-candidate A/Bs (47.2% acceptance, PPL 6.910, and the 2026-08-20 Heretic-300 build) are apples-to-apples. This is unrealised upside, not a correction to past numbers. ### 3.16 Weight-only NVFP4A16 with a minmax observer is DATA-FREE — your calibration corpus is ignored, but its tokenizer side-effect is not Measured 2026-09-08 (Gemma-4 26B-A4B MoE, ERP run 6, llm-compressor 0.13): with `scheme="NVFP4A16"` (default `memoryless_minmax` weights, no activation quant) llm-compressor logs `Inferred DataFreePipeline for QuantizationModifier` and never touches the dataset — the whole 26B quant ran in ~90 s on one Blackwell. Two consequences: (1) do not budget calibration time or believe a corpus "shaped" the result — only `imatrix_mse`/activation observers consume data; (2) building the calibration set still calls the fast tokenizer with `truncation=True, max_length=N`, so §3.14's baked cap (`max_length: 8192` here) lands in the saved `tokenizer.json` **even though no calibration happened**. The §4.3 post-step caught it. Reference: `services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py` (linearize_moe + assert 11,520 expert Linears + post-steps; the published `prithivMLmods/gemma-4-26B-A4B-it-NVFP4A16` recipe replicated, 222→252 ignore entries with audio/norm/router regexes added). **So for any `*A16` scheme, do not pass a dataset at all** — not a shorter one, none. It removes the §3.14 tokenizer bake-in *and* llm-compressor's "initialize model processor ... required when a dataset is provided" demand, which is fatal on any upload that ships no `processor_config.json`. Both of those cost an attempt on MeroMero v2 (2026-09-10); dropping the corpus costs nothing, because a `DataFreePipeline` was never going to read it. Driver: `services/meromero-quant/quant_a16_datafree.py`. **Confirmed twice more the same day**: the A4B heretic quant, run *with* the corpus, shipped `max_length: 8192` in its `tokenizer.json`; the v2 dense, run without it, came out `truncation: null`. ### 3.17 ⭐⭐ The transformers you measured is not the transformers that ran — llm-compressor pins it **Measured 2026-09-10, MeroMero v2-31B, and it cost a full misdiagnosis.** The quant died in `AutoTokenizer.from_pretrained` with `AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute`. The obvious reading was that the source config carried a `per_layer_config` key the canonical one lacked, so that key was the defect. It was not. `pip install llmcompressor==0.13.0` **downgrades transformers underneath you** — 5.16.1 → 5.14.1 in the `vllm/vllm-openai` image. The config had been serialized *by* 5.16.1, which materializes `per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard but not the gemma4 resolver, so it refuses the global read. Under the image's own 5.16.1 the very same config loads fine, which is exactly what makes this class expensive: **the version you print at the top of the script is not the version the quant runs on.** - **Print the version AFTER the install**, in the same container, and put it in the log. `python3 -c 'import transformers; print(transformers.__version__)'` as a pipeline step, not a thing you check by hand once. - **Pin the image by digest for the length of a campaign.** `:latest` was re-pulled between attempts 3 and 4 of this run and moved the toolchain mid-diagnosis, so the same command produced a different error for reasons that had nothing to do with the change under test. - **Reproduce the actual failing call, not a paraphrase.** A bare `AutoConfig.from_pretrained` did not reproduce this at all — the trigger was reached through `AutoTokenizer`. Testing the config in isolation would have "cleared" it. - **Keep a known-good tree as the positive control.** zerofata's canonical v2, quantized cleanly three weeks earlier, is what separated "this config is malformed" from "this toolchain moved". Without it, four green variants and one red one are just noise. Related but distinct from §3.4, which is about version deadlocks you can *see*. This one is a silent downgrade inside a line you already trusted. Instruments: `services/meromero-quant/` (`tok_repro.py`, `perlayer_test.py`). ### 3.14 ⭐⭐ Calibration BAKES a truncation cap into the shipped tokenizer **Symptom (on a newer transformers, at startup, on a vision model):** ``` ValueError: Mismatch in `image` token count between text and `input_ids`. Got ids=[2047] and text=[16384]. Likely due to `truncation='max_length'`. ``` The engine never serves a request. The number in `ids=[…]` is your **calibration seqlen minus one**, which is the tell. **Cause — an in-place mutation you never wrote.** Calibration tokenizes like this: ```python tok(b["text"], truncation=True, max_length=seqlen, add_special_tokens=False) ``` For a **fast** tokenizer that call does not just return ids — it **mutates the Rust backend's truncation state in place**. A later `tok.save_pretrained(out)` then persists it: ```json "truncation": {"direction": "Right", "max_length": 2048, "strategy": "LongestFirst", "stride": 0} ``` The source model has `"truncation": null`. **You shipped a tokenizer that clamps every prompt at the calibration length, permanently.** **Why it hid for months.** Older transformers does not enforce the text-vs-ids count check, so the cap sits latent — the model serves, gates pass, vision works, nothing logs. It only detonates when you bump the image, and then it presents as a *vision* bug at startup with no mention of tokenizers. It also caps the effective image resolution long before it kills the seat: at a 2048 cap the largest servable image is ~1448×1448, because `(edge/patch)² / merge²` image tokens must fit under it. **The fix — never save the calibration tokenizer.** Re-read a pristine one from the source: ```python from transformers import AutoTokenizer as _AutoTokenizer _AutoTokenizer.from_pretrained(a.model, trust_remote_code=True).save_pretrained(a.out) ``` then **assert** it, because this is exactly the class of defect that returns silently: ```python if json.load(open(f"{a.out}/tokenizer.json")).get("truncation"): raise SystemExit("FAILED CHECK: saved tokenizer carries a truncation cap") ``` Both live in `quant_mixed_nvfp4.py` as of 2026-08-22. **Audit any build predating that.** One line per model: ```bash python3 -c 'import json,sys;print(json.load(open(sys.argv[1]+"/tokenizer.json")).get("truncation"))' ``` Measured 2026-08-22 — every mixed-NVFP4 build from this pipeline was affected, and the two live ones were corrected in place (backup `tokenizer.json.bak-truncation-20260822`; only the `truncation` field changed, vocab and `added_tokens` byte-identical): | build | truncation as found | |---|---| | `qwen38-27b-orcarouter-nvfp4-mixed` (live `gen`) | **2048** → fixed | | `mog-sec-27b-nvfp4-mixed` (live `sec`) | **2048** → fixed | | `qwen38-27b-heresy-nvfp4-mixed` (retired) | 2048, left as-is | | `G4-MeroMero-v2-31B-NVFP4A16` (different pipeline) | `null` ✓ | | `mog-sec-27b-bf16` (source) | `null` ✓ | **Editing it is safe on a running seat** — vLLM reads the tokenizer at startup and holds its own copy, so the fix lands on the next restart with no disruption. **The general lesson, which is the transferable part:** this is the third defect in this playbook where *the artifact carries config authored against an older transformers and a newer one starts enforcing it* (see also the Gemma-4 heterogeneous `head_dim`). **Treat "we bumped the image" as a config-compatibility event, not just a version change** — and prefer saving artifacts re-read from the source over saving objects the pipeline has touched. --- ## 4. Pipeline shape ### 4.1 Prove the targets before spending GPU time Enumerate module names from the safetensors index and check your regexes against them: **zero overlap between groups, and the union covers every layer you intended.** This is free, takes seconds, and catches a mis-scoped regex that would otherwise surface as a mystery quality regression hours later. Reference: `services/gen-seat-mixed-quant/validate_targets.py`. ### 4.2 Quantize Calibration data matters for `imatrix_mse` + static activation observers. We use `/tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl` (512 chat samples, RP/GM-flavoured — appropriate for our seats). 256 samples @ 2048 tokens ≈ 20 min for a 27B on one Blackwell. ### 4.3 The mandatory post-steps Never optional, always in this order, and the last one **verifies rather than assumes**: 1. Graft `model-mtp.safetensors` + register its tensors in the output index. 2. Restore `preprocessor_config.json` / `processor_config.json` / `video_preprocessor_config.json`. 3. **Re-inject `re:^mtp.*` into `quantization_config.ignore` and confirm it is there** (§3.3). 4. **Confirm the saved `tokenizer.json` has `truncation: null`** (§3.14) — calibration mutates the fast tokenizer in place and `save_pretrained` bakes the cap in. Latent on an older transformers, fatal on a newer one. Reference implementation: `services/gen-seat-mixed-quant/post_quant.py`. On Gemma-4 steps 1 and 3 are N/A — the family ships no MTP head at all — which leaves 2 and 4, and 4 is the one that fires. `services/meromero-quant/post_quant_gemma4.py` runs them idempotently with a `--check` mode; point it at a tree you already trust before you trust its verdict on a new one. ### 4.4 Test on a temp port, never on the live seat Serve the candidate on an alt port with the live seat's **exact** flags, run the gate (§5), and only then flip `.env`. Keep the previous build on disk; rollback is one `.env` line. **Ask for `logprobs` once, on the temp port, before the alias moves.** A seat can pass its healthcheck, report the right KV pool, and answer every request with the full token count while every token decodes to the empty string — that is what a router-quantized MoE does (§3.15). NaN logits are invisible to `/health`, invisible to the token counts, and invisible to a tensor-table diff against the wrong control; a single `logprobs: 1` request surfaces them immediately, because vLLM cannot even serialize the response (`Out of range float values are not JSON compliant: nan`). Add it to the smoke set: **served name, one prose completion, one image if the model is multimodal, one tool call, and one `logprobs` request.** ⚠ **A co-resident temp port is not always reachable, and the fallback is reversibility, not skipping the test.** Measured 2026-09-10: with 19 GiB free on a shared card, a 16 G A4B refused admission at `gpu-memory-utilization 0.20` (18.26 free vs 18.99 requested), and at 0.185 it got past admission and past the KV reservation only to OOM in **multimodal encoder-cache profiling** (`profiled with 3 video items of the maximum feature size`) — a cost easy to forget when budgeting a vision model. When the card genuinely cannot hold both, the substitute is: back up the host `.env` to a named file first, prove the new seat on its real port **while no gateway alias points at it**, and move the alias last. That ordering is what kept a NaN-serving seat away from every consumer; the seat itself was down ~16 minutes and nothing downstream saw a broken alias. --- ## 5. The acceptance gate — and how measurement lies to you Speed alone does not justify cutting over a shared seat. Gate on **all** of: decode tok/s, MTP acceptance, perplexity, a behavioural surface test, and — for an abliterated model — that the abliteration survived. **Three ways the numbers have lied to us. All three produced confident, wrong results.** 1. **Prefix caching fakes both speed metrics.** A fixed prompt returns byte-identical timings run after run; you are measuring cache, not compute. Worse for prefill: a *seeded* nonce regenerates the previous run's prompts verbatim and reads **~41k tok/s of cache-hit instead of ~5k of real prefill**. Use a fresh unseeded nonce per request; never seed a cache-buster. 2. **`prompt_logprobs` are garbage while speculative decoding is on** — ~uniform over the vocab (median rank ~10⁵; " Paris" after "The capital of France is" ranked 69698). **Perplexity must be measured on a seat served without `--speculative-config`,** on both sides of the comparison. 3. **A 0600 `.env` makes `docker compose` silently no-op.** Without `sudo` it fails `permission denied` reading `.env`, **leaves the old container running**, and reports success — producing a full page of "benchmark results" that were just the unchanged baseline. **Hard-verify the change landed against `docker inspect …Config.Cmd`.** **Re-measure the baseline before believing a target.** The 2026-08-15 handoff quoted ~68 tok/s; cache-busted, the incumbent was already doing 80.1 — essentially the *target* of the work queued against it. Had that not been re-measured, doing nothing would have looked like a 20% win. **Cheap shortcut worth taking first:** if a reputable published quant of the same architecture is already on-box (or is a small pull), **serve it as a probe and measure it** before committing hours to your own. It answers "is this gain even real?" in ten minutes *and* hands you the recipe. Harness: `services/gen-seat-mixed-quant/bench/` — `quickbench.py` (decode + acceptance), `prefill_bench.py`, `eval_quality.py` (PPL + abliteration), `surface_test.py` (chat, vision, tools, thinking split, long-context needle, streaming), `serve_probe.sh`. --- ### 5.1 ⭐⭐ Acceptance is not throughput — always run the DEPTH control **Measured 2026-08-22**, same instrument (vLLM's own `spec_decode` counters, delta over a fixed workload, temp 0), same target, same engine: | config | accepted tok/forward | throughput | |---|---|---| | MTP k=3 | 2.753 | 114.9 tok/s | | MTP k=7 | **3.041** ⬆ | **74.0 tok/s** ⬇ | **Raising `num_speculative_tokens` improved acceptance and destroyed throughput.** Reporting acceptance alone would have recommended a 36% regression. **Why:** a single-module MTP head (`mtp_num_hidden_layers: 1`, one `mtp.layers.0`) has no depth of its own — vLLM runs it **autoregressively**, so k draft tokens cost **k sequential forward passes**. Past a shallow depth the drafting cost exceeds what the extra accepted tokens save. Check `mtp_num_hidden_layers` before assuming depth is cheap. **The rule: when comparing two speculative methods, match k, or you are measuring depth rather than method.** A parallel-drafting drafter (DFlash2 and kin, which propose a whole block in one pass) at k=7 versus an autoregressive MTP at k=3 is not a method comparison — the depth control is what separates them. In our case the control showed most of the apparent acceptance win was depth, while the *throughput* win was real and came from parallel drafting, not better drafts: our MTP was **better at position 0** (79.6% vs 75.4%) and still lost overall. **Corollary — report both, always.** Acceptance rate, mean accepted length, and end-to-end tok/s. Any one of the three alone can point the wrong way. --- ## 6. Hardware and co-residency - **ana-ml2 = Blackwell (sm_120)**, 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths (`match_exact=True` on sm90) are **closed** here — do not plan around them. - **irv-ml1 = Ampere (sm_86)**, 3090 + A6000. **No native FP8/FP4** — 4-bit there is a VRAM saving only, not a speed win. Don't port a Blackwell scheme over and expect the throughput. - **GPU co-residency is a zero-sum budget, and a *smaller* model can break its neighbour.** `gpu-memory-utilization` is a fraction of the *whole card*, so when new weights are smaller the seat absorbs the slack as extra KV rather than releasing it. That is exactly how a −5.2 GB requant left the co-resident seat **0.18 GiB** short and crash-looping. **After any requant, re-check both seats' budgets** and hand the space back explicitly. --- ## 7. Superseded claims — do not follow these Old docs stay for their history, but these specific claims are **false now** and will cost you a day if followed: | claim | where | status | |---|---|---| | "Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance" | `docs/runbooks/heretic2-nvfp4-mtp-seat.md` §landmine 2 | **SUPERSEDED 2026-08-14.** The 0% was the missing `re:^mtp.*` ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.7–83.2% acceptance, live. Use compressed-tensors. | | "Abliteration desyncs the MTP head → uncensored models can't do MTP" | earlier auto-memory | **SUPERSEDED 2026-08-14.** A modest abliteration preserves MTP (83.7% at bf16). Test MTP on **bf16 first** to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source. | | "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). | | "The Qwen3.8 MTP corruption is fixed by disabling prefix caching while keeping MTP; the gen seat runs APC-off" | this playbook §3.7 (now marked), earlier auto-memory | **SUPERSEDED 2026-08-17, and the staleness was only caught 2026-08-26.** APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use; reverted the same day. The real cause was the **AEON W4A4 quant being defective** (~15-20% degenerate generations), with MTP / prefix-caching / gateway merely AMPLIFYING it (§3.8). The gen seat runs **MTP ON and prefix-caching ON** on the in-house mixed NVFP4+FP8 build — verified against the live container 2026-08-26. ⚠ The lesson inside the lesson: a *passing multi-turn probe* was not sufficient evidence either. | | "transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 without causal-conv1d; it is precision-driven cancellation and fp32 resolves it" | `services/coldfusion-abliteration/README.md`, `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` | **SUPERSEDED 2026-08-20.** Precision was never the variable. The NaN came from **multi-GPU sharding** and **`expandable_segments`** (§3.9, §3.10); fp32 only made it rarer, which is worse than failing. On one GPU with a plain allocator, **bf16 is exactly deterministic through all 64 layers and generates coherent prose** — at 50 GB and 4.3× the throughput of the 111 GB fp32 it replaced. | --- ## 8. Measured negatives — don't re-chase - **`num_speculative_tokens` = 3 is optimal** on the Qwen3.8-27B seat. Swept: n=2 → 77.1, **n=3 → 80.1**, n=4 → 78.7, n=5 → 75.9 tok/s. Higher n trades acceptance for draft width and loses. Re-sweep only if the drafter architecture changes. - **Uniform W4A4** — see §7 row 3. - **Dense-VL as the anatomy judge** — A/B'd, MoE retained. Don't re-propose. --- ## 9. Worked examples Per-model artifacts. Read for *how a specific model went*, not for the general lessons — those are above, and where the two disagree, **this file wins**. | artifact | what it is | |---|---| | `services/gen-seat-mixed-quant/` | **current reference.** Mixed NVFP4+FP8 on Qwen3.8-27B-Uncensored: scripts, acceptance harness, raw measurements. | | `stacks/gen-seat/README.md` | the live `gen` seat (7 LiteLLM aliases) | | `services/meromero-quant/` | NVFP4A16 on Gemma-4 (MeroMero A4B + v2-31B ablits): the five-attempt failure chain, the pinned-transformers trap (§3.17), and the GPU-free verification instruments. | | `stacks/meromero-charrp/README.md` | Gemma-4 seat — the **tool-call/reasoning-parser** trap (a parser default that returns null `content` for all prose) | | `services/heretic2-nvfp4-quant/` | modelopt-format MTP seat — historical; see §7 before following it | | `tools/mistral-small4-nvfp4/` | MoE + native-convert path; source of §3.6 | | `docs/pfi/recommended-model-settings.md` | serve-time sampler/flag defaults (not quant) | **A new model just dropped and needs requanting?** §1 → §2 → §4 → §5. Skim §3 first; it is the part that costs hours.