esh-pfi-infrastructure/docs/pfi/model-quantization-playbook.md

Model quantization playbook — the lessons that keep costing us hours

Read this before starting any new quant. Not the per-model runbooks — those are worked examples of a specific model at a specific point in time, and several carry claims that are now false (see §7). This file owns the transferable part: what recurs regardless of which model dropped this week.

Written 2026-08-15, after the fourth quant in five weeks re-discovered the third-known instance of the same loader-class bug. Scope: NVFP4 / FP8 / mixed-precision on the Blackwell boxes (ana-ml2), vLLM-served. Ampere (irv-ml1) has no native FP4/FP8 — see §6.

Maintenance rule. When a quant teaches you something model-agnostic, it lands here and the per-model README links up. When it's model-specific (this checkpoint's odd tensor names, this finetune's missing config), it stays in the per-model artifact. If you find yourself writing a "Gotchas" section that repeats §3, you are re-litigating — add the delta here instead.

---

1. The 60-second decision: which scheme

On Blackwell + vLLM, for a dense-or-hybrid VL model you intend to serve at long context:

want	scheme	notes
default, best speed/accuracy	mixed: NVFP4 W4A4 bulk MLPs + FP8 W8A8 attention/lm_head/last-8-layer MLPs	the current answer. §2.
max fidelity, don't care about prefill	NVFP4 W4A16 (weight-only)	forces the Marlin kernel — ~half the prefill of native FP4
small model, VRAM is free	FP8 W8A8	safe and simple; 2× the weight bytes of 4-bit
—	"W4A8" = NVFP4 weights + FP8 activations	DOES NOT EXIST. §3.1

Measured on Qwen3.8-27B (2026-08-15), W4A16 → mixed: decode +18%, prefill +78–98%, MTP acceptance unchanged, perplexity +1.7%, weights −19%.

Note the shape of that: decode barely moves, prefill nearly doubles. Decode at batch-1 is memory-bandwidth-bound and the weights are 4-bit under either scheme, so there is little to win; prefill is compute-bound, which is where native FP4 tensor cores replace the Marlin dequantize-to-BF16 path. If someone promises you a big decode win from a scheme change, be skeptical — and go measure §5 before believing it.

The accuracy cost is real and is paid on purpose. Operator ruling 2026-08-15: the ~1.7% perplexity is an acceptable price for the speed. Settled — don't re-litigate. For correct attribution: it is the activation-quantization cost (A4/A8 vs BF16 activations), not an MTP cost. Turning MTP off does not recover it; only reverting the quant does.

---

2. The reference recipe (mixed-precision)

Lifted from unsloth/Qwen3.8-27B-NVFP4 and replicated in-house. Prefer replicating a published recipe from a reputable quantizer over inventing one — they have already paid for the sensitivity analysis.

group	scheme	targets
group_0	FP8 W8A8 — channel weights (static) + per-token dynamic activations	self_attn.{q,k,v,o}_proj, linear_attn.{in_proj_qkv,in_proj_z,out_proj}, lm_head, the last 8 layers' MLPs
group_1	NVFP4 W4A4 — tensor_group gsize 16, fp8 scales, imatrix_mse weights, dynamic:"local" activations	all remaining MLP {gate,up,down}_proj
kv cache	FP8 static tensor
ignore	vision tower, linear_attn.{norm,in_proj_a,in_proj_b}, re:^mtp.*

Three things in there are load-bearing and easy to drop:

• Late layers stay FP8. Holding the last ~8 layers' MLPs (and lm_head) at 8-bit is the accuracy-preservation trick — late layers are the sensitive ones. Uniform W4A4 is what collapses.
• imatrix_mse on the W4A4 weights, not memoryless_minmax. Importance-weighted; needs calibration data.
• Group targets must be non-overlapping. Do not let group_1's .*mlp\..* also match the late layers and rely on group precedence to sort it out. Enumerate the early layers explicitly (re:.*layers\.([0-9]|[1-4][0-9]|5[0-5])\.mlp\.…) and prove it with a dry run (§4.1).

Toolchain: pip install llmcompressor into stock vllm/vllm-openai:latest gives llmcompressor 0.13 + compressed-tensors 0.18 without disturbing torch/transformers. Avoid nvidia-modelopt — see §3.4.

---

3. The recurring landmines

Ordered by how much time each has cost. Every one of these has bitten more than once.

3.1 "W4A8" is not a servable shape

vLLM's compressed-tensors dispatcher (compressed_tensors.py:704-713) accepts NVFP4 weights with exactly two activation settings:

input_activations	result
None	W4A16 — and it forces the Marlin kernel (kernels/linear/__init__.py:881-883)
NVFP4	W4A4, native

Anything else — FP8 included — raises at load:

ValueError: For NVFP4 weights, input quantization must also be NVFP4 format, None for NVFP4A16

CompressedTensorsW4A8Fp8 exists but is INT4 weights (W4A8_SUPPORTED_TYPES_MAP = {4: int4}) gated on _check_scheme_supported(90, match_exact=True) — Hopper-exact, so on Blackwell (sm_120) it is closed twice over. FP8 enters per-layer-group, never as activations on NVFP4 weights.

Cost: one queued task written against an impossible scheme.

3.2 Wrong loader class → silent weight-load failure

Rediscovered three times. Load the model through the class vLLM actually serves — the …ForConditionalGeneration / …ForImageTextToText wrapper, never AutoModelForCausalLM.

AutoModelForCausalLM resolves a VL config to the text-only inner class and saves a flat config with model.layers.* keys. vLLM's weight mapper wants model.language_model.* (+ model.visual.*). The mismatch does not error — every layer silently fails to load and you get !!!! gibberish, or an engine that rejects the checkpoint outright.

Bit: heretic2 (gibberish), Dark-Scarlett (both vLLM and SGLang refused the checkpoint), and the 2026-08 rounds.

3.3 The MTP head — three separate ways to lose it

Speculative decoding is a large fraction of the seat's throughput. It fails silently: the model serves fine, just at 0% acceptance.

1. The wrapper class does not instantiate mtp.*, so the quant drops it. Post-quant you must graft the BF16 model-mtp.safetensors back and register its tensors in the output index.
2. re:^mtp.* must be in quantization_config.ignore — else vLLM loads the grafted BF16 head as though quantized, it comes up uninitialised, and acceptance is 0%.
3. ⭐ llm-compressor PRUNES ignore entries that matched no module at quant time. Since the wrapper never loaded mtp.*, the entry matches nothing and is silently deleted from the saved config — even though you put it in the recipe. So it must be re-injected after the graft, and then verified, not assumed.

Cost: three rounds. The verify step caught it live on the third.

There is also a modelopt-format-specific version of this: vLLM 0.24 does not propagate modelopt exclude_modules to the spec-decode draft model, which no checkpoint config can fix (needs a sitecustomize runtime patch). Using compressed-tensors avoids it entirely — §3.4.

3.8 ⭐⭐ Multi-turn degeneration from TWO real compounding causes — how they masked each other

The most expensive diagnosis this project has had, because there were two real causes at once and each partial fix moved the needle enough to look like the answer. Recorded precisely because the first write-up of this section over-attributed it to the quant alone; that was wrong.

Cause 1 (real, upstream): the vLLM qwen3_5_mtp × Gated-DeltaNet bug. Confirmed by two cross-frontier peers and the tracker (vllm#47087 symptom-twin, #43559 fix lineage, #51113 fix): the GDN recurrent state cannot roll back on a partial draft-accept, so speculative decoding corrupts it, worse with context. Architectural — vLLM/SGLang/llama.cpp mainline all shared it. Genuinely fixed enough by moving to vLLM nightly (v0.27.2rc1.dev150+, carries #51113): the operator reported it "significantly better" — this was a real bug, not just an amplifier.

Cause 2 (real, quant): full W4A4 is mildly subpar, per the known gradient. sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4 is full W4A4 — 4-bit activations on attention too, the bottom of the activation-precision ordering already in §1: W4A4 (A4) < W4+FP8 (A8) < W4+bf16 (A16). Not "defective," just lowest-fidelity; on top of Cause 1 it degenerated ~15-20% of real multi-turn generations. The FP8-attention mixed build (qwen38-27b-uncensored-nvfp4-mixed, same base, same MTP, same nightly) sits a rung up that gradient and is coherent. AEON was purged 2026-08-17 (operator ruled it no-good; re-pullable from HF).

Why it cost days — and the process lessons that stand:

1. Two real causes compound and mask each other. Each mitigation (MTP-off, APC-off, the nightly #51113 fix) partially helped, so each looked like the fix and then failed in real use. When a mitigation "helps but doesn't fix," suspect a second cause rather than a wrong one.
2. Stochastic degeneration (~15-20%) is nearly invisible to a small synthetic probe — a 7-turn run passes ~4 in 5. n=1 "clean" proves nothing; this class needs many runs or the operator's real high-volume use. Three non-fixes were "validated" by a single clean probe here.
3. Isolate the WEIGHTS in parallel with the serving flags, not after. Swapping to a different quant of the same base (AEON→mixed) is what finally separated Cause 2 from Cause 1; doing it earlier would have shortened the hunt. But note it would NOT have found Cause 1 — the vLLM bug was real and needed the nightly.
4. Prefer FP8 attention (the §2 mixed recipe) over full W4A4 for a coherence- sensitive seat. AEON passed every static gate (abliteration 4/4, surface 6/6, a 36k needle, 52% acceptance) and was still the lower-fidelity of the two.

Current primary gen: the mixed FP8-attention build on pinned vLLM nightly with MTP, until the DavidAU Qwen3.8 lands. A W4+bf16 (W4A16) build would be higher fidelity still (§1) at a prefill cost — an option if the mixed build ever proves marginal.

3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn

§3.3 is about losing the head (0% acceptance, silent). This is the opposite and worse failure: the head loads, acceptance looks healthy, single-turn output is perfect — and then it corrupts multi-turn conversations once cumulative context passes ~2,000 tokens. The reply collapses in length and bleeds earlier turns into the current answer (a "describe durian" reply that contained the Krebs-cycle and winter answers from three turns back). Single-turn probes and the acceptance gate (§5) do not catch it — it only appears as accumulated context grows.

Isolated 2026-08-16 (operator-confirmed), each step measured on a fixed 7-turn probe:

• Not the serving gateway, not sampling, not repetition/template. Identical input gateway-vs-direct behaves the same; presence_penalty 1.5/0.5/0.0 all collapse; higher temperature collapses harder; a conversation of unrelated topics collapses at the same ~2k tokens as a repetitive one → it is context- length-driven, not template lock-in.
• Model-independent across every Qwen3.8-27B quant (AEON W4A4, unsloth FP8-attn, our in-house mixed) — so not a quant-brand or scheme artifact.
• DECISIVE: same model + same conversation, MTP OFF → coherent through 4k+ tokens, zero bleed. Toggle it back on → collapse returns. MTP is the cause.

Qwen3.6-27B running the same qwen3_5_mtp method is CLEAN. So the 3.6 MTP head/graft is fine and the 3.8 one is not — suspects: the bf16 graft being subtly wrong for the 3.8 head, or the vLLM qwen3_5_mtp impl diverging at num_speculative_tokens=3. Open upstream question (queried dvalin/bil-smithy 2026-08-17).

Rule: gate MTP on a MULTI-TURN coherence probe, not just single-shot acceptance. Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens for length-collapse and cross-turn bleed.

⛔ THE APC-OFF MITIGATION DID NOT HOLD — SUPERSEDED 2026-08-17, see §7. What follows is kept for its history and must not be applied: "disable prefix caching, keep MTP. The corruption is gated on MTP × prefix-caching together (vllm#43559 / #47194); with --no-enable-prefix-caching the buggy partial-accept align-path is inert. Confirmed on our stack: AEON W4A4, MTP on + prefix-caching off → the 7-turn varied series stays coherent through 3.9k tokens, zero bleed, at 104.6 tok/s / 53.6% acceptance."

It passed that synthetic 7-turn probe and the operator still saw severe degeneration in real use. Reverted the same day. The probe was structurally under-covering the real workload on both content distribution and depth — which is §3.7's own standing rule (gate on a multi-turn coherence probe, not single-shot acceptance) failing at one level up: the multi-turn probe was itself too small to gate on.

WHAT ACTUALLY RESOLVED IT. The multi-day hunt root-caused to the AEON W4A4 quant being defective — ~15-20% of generations went degenerate — with MTP, prefix-caching and the gateway all merely amplifying it. That is why every partial mitigation "helped" without fixing anything (§3.8). The gen seat today runs the in-house Heretic mixed NVFP4+FP8 build (FP8 attention, not W4A4) on vLLM nightly carrying #51113, with MTP ON and prefix-caching ON, and is coherent in real use.

Verified against the live seat 2026-08-26: vllm-gen runs --enable-prefix-caching with qwen3_5_mtp / num_speculative_tokens 3. The compose file (stacks/gen-seat/compose.yaml) carries the full history inline and is the current authority; this section was stale against it for nine days.

Things that do not work, ruled out: num_speculative_tokens=1 (corruption is depth-independent — reproduces at n=1 and n=2, deterministically probed upstream); switching engine (vLLM / SGLang / llama.cpp mainline all share the GDN-rollback bug — it is architectural). The proper upstream fix (vllm#51113) is in main / v0.27.2rc0 only — not in a stable release, so we hold at APC-off until it lands. Two cross-frontier peers (dvalin/bil-smithy) confirmed the bug class and pointed at the open symptom-twin issue #47087.

3.15 ⭐⭐ Fused 3-D MoE experts are INVISIBLE to a targets=["Linear"] recipe

Symptom: none. The quant completes, the artifact loads, and 88.5% of the model is still BF16. Nothing warns you.

Modern MoE checkpoints store each layer's experts as two fused 3-D nn.Parameter tensors, not as N nn.Linear modules. Gemma-4 26B-A4B:

model.language_model.layers.N.experts.gate_up_proj   BF16 [128, 1408, 2816]
model.language_model.layers.N.experts.down_proj      BF16 [128, 2816,  704]

Note the absent .weight suffix — that is the tell. mlp.down_proj.weight is an nn.Linear; experts.down_proj is a bare parameter.

Measured on that checkpoint, recipe targeting ["Linear"]:

Linear modules   427
WILL quantize    205   (experts: 0)     <- 22.84 B params untouched

This is the same defect that killed QLoRA on this architecture — bitsandbytes 4-bit replacement also walks nn.Linear modules and also silently skipped the experts. Two different tools, one blind spot, because the blind spot is in the checkpoint layout, not the tool.

The fix (llm-compressor ≥ 0.12):

from llmcompressor.modeling.moe.linearize import linearize_moe
model = SomeForConditionalGeneration.from_pretrained(...)
linearize_moe(model)          # BEFORE building the recipe

Linear modules   11947
WILL quantize    11725  (experts: 11520)   # 30 layers x 128 x 3 proj

linearize_moe unfuses the 3-D parameters into per-expert experts.N.{gate,up,down}_proj Linears. No registration is needed if the module satisfies FusedExpertsProtocol structurally — bare down_proj plus gate_up_proj/up_proj Parameters. load_quantizable_moe(model_cls) is the faster variant that linearizes during load rather than after.

Always assert the expert count before spending GPU time (§4.1). The arithmetic is layers × experts × projections; if your target list does not hit it exactly, the recipe is wrong and the failure is silent.

⚠ Keep routers in ignore. A 4-bit router picks different experts — that error does not average out downstream, it changes which weights run at all.

3.4 Toolchain version deadlocks

Both directions have burned us, so the resolution is: use llm-compressor / compressed-tensors, not nvidia-modelopt.

• modelopt 0.45 ↔ transformers 5.12: mtq.quantize dies TypeError: issubclass() arg 2 must be a class (modelopt registers transformers' FusedMoE, a function in 5.x, as an nn class).
• modelopt 0.43 doesn't fix it — it drags transformers back to 4.57, which cannot load qwen3_5 at all.
• modelopt's config API also trails the current model families by a version.

3.5 Vision tower and its configs

• Keep the vision tower in ignore (BF16). Only the LLM backbone gets quantized.
• The wrapper-class save drops preprocessor_config.json (and the video one). Without it the seat crash-loops Can't load image processor. Restore from the source — and if the upstream repo omits it, reconstruct it from processor_config.json's image_processor sub-dict.

3.6 Memory and device placement (large models)

• device_map=None/"cpu", never "auto". auto fills GPU0 and OOMs during un-fusing; constraining with max_memory then offloads to the meta device, which cannot be .copy_()d. CPU-resident keeps every tensor real; the sequential pipeline still onloads per-layer to GPU.
• Avoid mmap on /tank. safetensors.safe_open() mmaps a whole shard; on ZFS a 50 GB shard ENOMEMs regardless of free RAM (MAP_SHARED never consults the commit limit). Read with plain read() + load(bytes), one shard cached at a time.
• vm.overcommit_memory=1 on ana-ml2 (durable via playbooks/ana-ml2-overcommit-memory.yaml).

3.9 ⭐⭐ A sharded forward can be silently WRONG — never trust device_map="auto" for activations

Splitting Qwen3.8-27B (Qwen3_5 hybrid) across the two Blackwells with device_map="auto" produces a model that loads clean, reports no error, and computes garbage: the residual stream collapses to exactly zero a couple of layers past the GPU0→GPU1 boundary, and the logits decode to rubbish ('8', '�', 'b'). Every layer below the boundary stays healthy, deterministic, and bit-identical to a single-GPU run — which is what makes it so dangerous. A capture that reads a low layer looks perfectly plausible and is fine; one that reads a high layer is reading zeros, and nothing in the pipeline says so. Measured 2026-08-20 (§9 Cold-Fusion).

Rule: any workload that reads activations — refusal-direction capture, calibration, activation statistics, PPL — must run on ONE device. Sharding is for storage, and it is only safe when you consume the model's final output through an engine that was built for it (vLLM does TP correctly; device_map="auto" in transformers is not the same thing). If it does not fit on one card, shrink the model, not the guarantee: truncating the decoder to N layers is exact for any activation read at a layer < N (a causal stack's layer-N state cannot depend on layers above N), and it is cheap — verified by reproducing the full model's layers 18/20/22/26 bit-for-bit.

Gate it, don't remember it. Assert single-device residency and zero offload before the forward:

dmap = getattr(model, "hf_device_map", {}) or {}
gpus = {str(v) for v in dmap.values()} - {"cpu", "disk"}
offloaded = [k for k, v in dmap.items() if str(v) in ("cpu", "disk")]
if len(gpus) > 1 or offloaded:
    sys.exit("residency gate FAILED — sharded/offloaded forward reads garbage")

3.10 ⭐⭐ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True corrupts retained tensors

On torch 2.12+cu130 / Blackwell, tensors that outlive their allocation come back corrupted with this flag set: captured hidden states carried Inf / NaN / zeros that moved between bit-identical forwards (same input, same weights → a different layer corrupted each time). Unset, the identical forwards are exactly reproducible. Several runbooks recommend this flag for headroom on large loads; for anything that keeps activations it buys corruption.

Two tells that distinguish this from a real numerical blowup, both worth knowing because they generalise: a genuine blowup propagates to later layers and is deterministic. Corruption does neither — downstream layers were finite and consistent, and the affected layer moved run to run. If a "NaN" fails to propagate, stop debugging the math and start debugging memory.

Corollary: do not read output_hidden_states=True off a returned object on a large multi-device load. Take what you need during the forward with a register_forward_pre_hook that clones to CPU immediately — it closes the reuse window and never retains a [B, seq, hidden] tensor per layer, so it is cheaper than the thing it replaces.

3.11 Determinism is a necessary check, not a sufficient one

Both defects above were found by the cheapest possible test — run the same input twice and diff — which no amount of eyeballing plausible-looking numbers would have caught. Add it to any activation-reading pipeline. But note the trap that followed: after fixing the allocator, the run went perfectly "deterministic" because the corrupted layers were now stably zero. Pair the determinism check with a magnitude check (residual norms should grow smoothly with depth; an exact 0.0 mid-stack is impossible) and, where you can, a coherence check (generate 40 tokens and read them).

3.12 ⭐⭐ You cannot free a 27B model in-process — give each model its own process

Any A/B that loads two large checkpoints in sequence (KL, logit diffing, teacher-vs-student) will try to release the first before loading the second. On this stack, it does not work. Measured 2026-08-20 on Qwen3.8-27B bf16, free VRAM after each attempt:

teardown	free VRAM
del model + gc.collect() + torch.cuda.empty_cache()	45,287 MiB
same, with the model confined to an inner frame that exits	45,287 MiB
the process exits	97,247 MiB

The ~51,300 MiB of weights stayed resident through both in-process teardowns. The first run survived only because PyTorch's allocator hit OOM on the second load, ran a collection itself, and retried — the second model landed by rescue, not by design. That is not a release strategy: on an architecture where a silent CPU offload does not raise (§3.9), the day the retry does not fire you get confident garbage instead of an error.

Do this instead: one process per model, hand results to disk between them (first-token log-probs for a 250k vocab are ~715 MiB per model — nothing), and gate each stage on free VRAM before the load. Reference implementation: services/coldfusion-abliteration/kl_divergence.py (--stage ref|cand|score).

Two gate corollaries learned in the same session:

• ⭐ A residency gate that reads hf_device_map cannot fail. The map is empty whenever transformers puts the whole model on one device, so the check reports "unsharded" both when everything is fine and when there is nothing to inspect. Read {p.device for p in model.parameters()} — ground truth in every case. (Generalises feedback_assert_effective_value_not_substring: presence of a passing check is not evidence of a check that can fail.)
• ⭐ Size VRAM from the checkpoint's own headers, never from a remembered figure. A runbook carried "bf16 is 50 GB"; the real number was 50.10 GiB = 51,300 MiB of text-only weights. That 3.7 GB unit error is exactly the difference between "stop one co-tenant" and "stop both", and it cost an aborted window. Sum the safetensors header offsets (excluding tensors the loader class won't instantiate — vision, MTP); read only the 8-byte length prefix + JSON header, never safe_open, which mmaps the whole shard and ENOMEMs on ZFS (§ Avoid mmap on /tank).

3.13 ⭐⭐ The observer you ASKED for is not necessarily the observer you GOT

quant_mixed_nvfp4.py sets observer="imatrix_mse" on the NVFP4 W4A4 group. It has never once been used. llm-compressor looks for importance data, finds none, and silently degrades:

_get_validated_importance | WARNING - imatrix_mse: no importance data available.
                                      Falling back to uniform MSE.

Confirmed on the 2026-08-20 09:59 incumbent quant and the 22:45 Heretic-300 quant; find /tank/aimodels -iname "*imatrix*" -o -iname "*importance*" returns nothing. Every NVFP4 build in the fleet has run uniform MSE while the recipe claimed importance weighting.

Why it went unseen for months: the warning scrolls past inside a tqdm progress bar during a ~20 minute quant. It is only visible if you read the log while it runs.

The generalisable rule, which is bigger than imatrix. A quantizer, optimiser or observer that silently falls back to a weaker default is a whole class of invisible quality loss — the config is accepted, nothing errors, the artifact benchmarks plausibly, and you never learn you got the cheap path. So:

• Grep the quant log for WARNING, Falling back, not available, ignoring before trusting an artifact. Make it a step, not a habit.
• Assert the effective setting, never the requested one — the same rule as feedback_assert_effective_value_not_substring, applied to quantizer internals rather than config files.
• If the fallback turns out to be unavoidable in your toolchain version, change the recipe to say what it actually does. A recipe line that silently lies is worse than one that admits a limitation.

⚠️ Do not "fix" this by assuming an imatrix would help. Verify first that your llm-compressor version can consume an externally supplied importance matrix at all, and in what format. Parked as park/nvfp4-recipe-asks-for-imatrix-mse-but-silently-2 (id 42) with the calibration corpus that would feed it.

✅ Comparisons already made remain valid. Because every build shares the fallback, the incumbent-vs-candidate A/Bs (47.2% acceptance, PPL 6.910, and the 2026-08-20 Heretic-300 build) are apples-to-apples. This is unrealised upside, not a correction to past numbers.

3.14 ⭐⭐ Calibration BAKES a truncation cap into the shipped tokenizer

Symptom (on a newer transformers, at startup, on a vision model):

ValueError: Mismatch in `image` token count between text and `input_ids`.
Got ids=[2047] and text=[16384]. Likely due to `truncation='max_length'`.

The engine never serves a request. The number in ids=[…] is your calibration seqlen minus one, which is the tell.

Cause — an in-place mutation you never wrote. Calibration tokenizes like this:

tok(b["text"], truncation=True, max_length=seqlen, add_special_tokens=False)

For a fast tokenizer that call does not just return ids — it mutates the Rust backend's truncation state in place. A later tok.save_pretrained(out) then persists it:

"truncation": {"direction": "Right", "max_length": 2048, "strategy": "LongestFirst", "stride": 0}

The source model has "truncation": null. You shipped a tokenizer that clamps every prompt at the calibration length, permanently.

Why it hid for months. Older transformers does not enforce the text-vs-ids count check, so the cap sits latent — the model serves, gates pass, vision works, nothing logs. It only detonates when you bump the image, and then it presents as a vision bug at startup with no mention of tokenizers. It also caps the effective image resolution long before it kills the seat: at a 2048 cap the largest servable image is ~1448×1448, because (edge/patch)² / merge² image tokens must fit under it.

The fix — never save the calibration tokenizer. Re-read a pristine one from the source:

from transformers import AutoTokenizer as _AutoTokenizer
_AutoTokenizer.from_pretrained(a.model, trust_remote_code=True).save_pretrained(a.out)

then assert it, because this is exactly the class of defect that returns silently:

if json.load(open(f"{a.out}/tokenizer.json")).get("truncation"):
    raise SystemExit("FAILED CHECK: saved tokenizer carries a truncation cap")

Both live in quant_mixed_nvfp4.py as of 2026-08-22.

Audit any build predating that. One line per model:

python3 -c 'import json,sys;print(json.load(open(sys.argv[1]+"/tokenizer.json")).get("truncation"))' <model_dir>

Measured 2026-08-22 — every mixed-NVFP4 build from this pipeline was affected, and the two live ones were corrected in place (backup tokenizer.json.bak-truncation-20260822; only the truncation field changed, vocab and added_tokens byte-identical):

build	truncation as found
qwen38-27b-orcarouter-nvfp4-mixed (live gen)	2048 → fixed
mog-sec-27b-nvfp4-mixed (live sec)	2048 → fixed
qwen38-27b-heresy-nvfp4-mixed (retired)	2048, left as-is
G4-MeroMero-v2-31B-NVFP4A16 (different pipeline)	null ✓
mog-sec-27b-bf16 (source)	null ✓

Editing it is safe on a running seat — vLLM reads the tokenizer at startup and holds its own copy, so the fix lands on the next restart with no disruption.

The general lesson, which is the transferable part: this is the third defect in this playbook where the artifact carries config authored against an older transformers and a newer one starts enforcing it (see also the Gemma-4 heterogeneous head_dim). Treat "we bumped the image" as a config-compatibility event, not just a version change — and prefer saving artifacts re-read from the source over saving objects the pipeline has touched.

---

4. Pipeline shape

4.1 Prove the targets before spending GPU time

Enumerate module names from the safetensors index and check your regexes against them: zero overlap between groups, and the union covers every layer you intended. This is free, takes seconds, and catches a mis-scoped regex that would otherwise surface as a mystery quality regression hours later. Reference: services/gen-seat-mixed-quant/validate_targets.py.

4.2 Quantize

Calibration data matters for imatrix_mse + static activation observers. We use /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl (512 chat samples, RP/GM-flavoured — appropriate for our seats). 256 samples @ 2048 tokens ≈ 20 min for a 27B on one Blackwell.

4.3 The mandatory post-steps

Never optional, always in this order, and the last one verifies rather than assumes:

1. Graft model-mtp.safetensors + register its tensors in the output index.
2. Restore preprocessor_config.json / processor_config.json / video_preprocessor_config.json.
3. Re-inject re:^mtp.* into quantization_config.ignore and confirm it is there (§3.3).
4. Confirm the saved tokenizer.json has truncation: null (§3.14) — calibration mutates the fast tokenizer in place and save_pretrained bakes the cap in. Latent on an older transformers, fatal on a newer one.

Reference implementation: services/gen-seat-mixed-quant/post_quant.py.

4.4 Test on a temp port, never on the live seat

Serve the candidate on an alt port with the live seat's exact flags, run the gate (§5), and only then flip .env. Keep the previous build on disk; rollback is one .env line.

---

5. The acceptance gate — and how measurement lies to you

Speed alone does not justify cutting over a shared seat. Gate on all of: decode tok/s, MTP acceptance, perplexity, a behavioural surface test, and — for an abliterated model — that the abliteration survived.

Three ways the numbers have lied to us. All three produced confident, wrong results.

1. Prefix caching fakes both speed metrics. A fixed prompt returns byte-identical timings run after run; you are measuring cache, not compute. Worse for prefill: a seeded nonce regenerates the previous run's prompts verbatim and reads ~41k tok/s of cache-hit instead of ~5k of real prefill. Use a fresh unseeded nonce per request; never seed a cache-buster.
2. prompt_logprobs are garbage while speculative decoding is on — ~uniform over the vocab (median rank ~10⁵; " Paris" after "The capital of France is" ranked 69698). Perplexity must be measured on a seat served without --speculative-config, on both sides of the comparison.
3. A 0600 .env makes docker compose silently no-op. Without sudo it fails permission denied reading .env, leaves the old container running, and reports success — producing a full page of "benchmark results" that were just the unchanged baseline. Hard-verify the change landed against docker inspect …Config.Cmd.

Re-measure the baseline before believing a target. The 2026-08-15 handoff quoted ~68 tok/s; cache-busted, the incumbent was already doing 80.1 — essentially the target of the work queued against it. Had that not been re-measured, doing nothing would have looked like a 20% win.

Cheap shortcut worth taking first: if a reputable published quant of the same architecture is already on-box (or is a small pull), serve it as a probe and measure it before committing hours to your own. It answers "is this gain even real?" in ten minutes and hands you the recipe.

Harness: services/gen-seat-mixed-quant/bench/ — quickbench.py (decode + acceptance), prefill_bench.py, eval_quality.py (PPL + abliteration), surface_test.py (chat, vision, tools, thinking split, long-context needle, streaming), serve_probe.sh.

---

5.1 ⭐⭐ Acceptance is not throughput — always run the DEPTH control

Measured 2026-08-22, same instrument (vLLM's own spec_decode counters, delta over a fixed workload, temp 0), same target, same engine:

config	accepted tok/forward	throughput
MTP k=3	2.753	114.9 tok/s
MTP k=7	3.041 ⬆	74.0 tok/s ⬇

Raising num_speculative_tokens improved acceptance and destroyed throughput. Reporting acceptance alone would have recommended a 36% regression.

Why: a single-module MTP head (mtp_num_hidden_layers: 1, one mtp.layers.0) has no depth of its own — vLLM runs it autoregressively, so k draft tokens cost k sequential forward passes. Past a shallow depth the drafting cost exceeds what the extra accepted tokens save. Check mtp_num_hidden_layers before assuming depth is cheap.

The rule: when comparing two speculative methods, match k, or you are measuring depth rather than method. A parallel-drafting drafter (DFlash2 and kin, which propose a whole block in one pass) at k=7 versus an autoregressive MTP at k=3 is not a method comparison — the depth control is what separates them. In our case the control showed most of the apparent acceptance win was depth, while the throughput win was real and came from parallel drafting, not better drafts: our MTP was better at position 0 (79.6% vs 75.4%) and still lost overall.

Corollary — report both, always. Acceptance rate, mean accepted length, and end-to-end tok/s. Any one of the three alone can point the wrong way.

---

6. Hardware and co-residency

• ana-ml2 = Blackwell (sm_120), 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths (match_exact=True on sm90) are closed here — do not plan around them.
• irv-ml1 = Ampere (sm_86), 3090 + A6000. No native FP8/FP4 — 4-bit there is a VRAM saving only, not a speed win. Don't port a Blackwell scheme over and expect the throughput.
• GPU co-residency is a zero-sum budget, and a smaller model can break its neighbour. gpu-memory-utilization is a fraction of the whole card, so when new weights are smaller the seat absorbs the slack as extra KV rather than releasing it. That is exactly how a −5.2 GB requant left the co-resident seat 0.18 GiB short and crash-looping. After any requant, re-check both seats' budgets and hand the space back explicitly.

---

7. Superseded claims — do not follow these

Old docs stay for their history, but these specific claims are false now and will cost you a day if followed:

claim	where	status
"Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance"	docs/runbooks/heretic2-nvfp4-mtp-seat.md §landmine 2	SUPERSEDED 2026-08-14. The 0% was the missing re:^mtp.* ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.7–83.2% acceptance, live. Use compressed-tensors.
"Abliteration desyncs the MTP head → uncensored models can't do MTP"	earlier auto-memory	SUPERSEDED 2026-08-14. A modest abliteration preserves MTP (83.7% at bf16). Test MTP on bf16 first to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source.
"NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer"	reference_nvfp4_w4a4_granite_infeasible	NARROWED. True for uniform W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs with FP8 on attention and late layers is fine and is the current default (§2).
"The Qwen3.8 MTP corruption is fixed by disabling prefix caching while keeping MTP; the gen seat runs APC-off"	this playbook §3.7 (now marked), earlier auto-memory	SUPERSEDED 2026-08-17, and the staleness was only caught 2026-08-26. APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use; reverted the same day. The real cause was the AEON W4A4 quant being defective (~15-20% degenerate generations), with MTP / prefix-caching / gateway merely AMPLIFYING it (§3.8). The gen seat runs MTP ON and prefix-caching ON on the in-house mixed NVFP4+FP8 build — verified against the live container 2026-08-26. ⚠ The lesson inside the lesson: a passing multi-turn probe was not sufficient evidence either.
"transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 without causal-conv1d; it is precision-driven cancellation and fp32 resolves it"	services/coldfusion-abliteration/README.md, persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md	SUPERSEDED 2026-08-20. Precision was never the variable. The NaN came from multi-GPU sharding and expandable_segments (§3.9, §3.10); fp32 only made it rarer, which is worse than failing. On one GPU with a plain allocator, bf16 is exactly deterministic through all 64 layers and generates coherent prose — at 50 GB and 4.3× the throughput of the 111 GB fp32 it replaced.

---

8. Measured negatives — don't re-chase

• num_speculative_tokens = 3 is optimal on the Qwen3.8-27B seat. Swept: n=2 → 77.1, n=3 → 80.1, n=4 → 78.7, n=5 → 75.9 tok/s. Higher n trades acceptance for draft width and loses. Re-sweep only if the drafter architecture changes.
• Uniform W4A4 — see §7 row 3.
• Dense-VL as the anatomy judge — A/B'd, MoE retained. Don't re-propose.

---

9. Worked examples

Per-model artifacts. Read for how a specific model went, not for the general lessons — those are above, and where the two disagree, this file wins.

artifact	what it is
services/gen-seat-mixed-quant/	current reference. Mixed NVFP4+FP8 on Qwen3.8-27B-Uncensored: scripts, acceptance harness, raw measurements.
stacks/gen-seat/README.md	the live gen seat (7 LiteLLM aliases)
stacks/meromero-charrp/README.md	Gemma-4 seat — the tool-call/reasoning-parser trap (a parser default that returns null content for all prose)
services/heretic2-nvfp4-quant/	modelopt-format MTP seat — historical; see §7 before following it
tools/mistral-small4-nvfp4/	MoE + native-convert path; source of §3.6
docs/pfi/recommended-model-settings.md	serve-time sampler/flag defaults (not quant)

A new model just dropped and needs requanting? §1 → §2 → §4 → §5. Skim §3 first; it is the part that costs hours.