708 lines
41 KiB
Markdown
708 lines
41 KiB
Markdown
# Model quantization playbook — the lessons that keep costing us hours
|
||
|
||
**Read this before starting any new quant.** Not the per-model runbooks — those are worked
|
||
examples of a *specific* model at a *specific* point in time, and several carry claims that are
|
||
now false (see §7). This file owns the **transferable** part: what recurs regardless of which
|
||
model dropped this week.
|
||
|
||
Written 2026-08-15, after the fourth quant in five weeks re-discovered the third-known instance
|
||
of the same loader-class bug. Scope: NVFP4 / FP8 / mixed-precision on the Blackwell boxes
|
||
(ana-ml2), vLLM-served. Ampere (irv-ml1) has no native FP4/FP8 — see §6.
|
||
|
||
**Maintenance rule.** When a quant teaches you something *model-agnostic*, it lands here and the
|
||
per-model README links up. When it's model-specific (this checkpoint's odd tensor names, this
|
||
finetune's missing config), it stays in the per-model artifact. If you find yourself writing a
|
||
"Gotchas" section that repeats §3, you are re-litigating — add the delta here instead.
|
||
|
||
---
|
||
|
||
## 1. The 60-second decision: which scheme
|
||
|
||
On Blackwell + vLLM, for a dense-or-hybrid VL model you intend to serve at long context:
|
||
|
||
| want | scheme | notes |
|
||
|---|---|---|
|
||
| **default, best speed/accuracy** | **mixed: NVFP4 W4A4 bulk MLPs + FP8 W8A8 attention/`lm_head`/last-8-layer MLPs** | the current answer. §2. |
|
||
| max fidelity, don't care about prefill | NVFP4 **W4A16** (weight-only) | forces the **Marlin** kernel — ~half the prefill of native FP4 |
|
||
| small model, VRAM is free | FP8 **W8A8** | safe and simple; 2× the weight bytes of 4-bit |
|
||
| — | ~~"W4A8" = NVFP4 weights + FP8 activations~~ | **DOES NOT EXIST.** §3.1 |
|
||
|
||
**Measured on Qwen3.8-27B (2026-08-15), W4A16 → mixed:** decode +18%, prefill **+78–98%**,
|
||
MTP acceptance unchanged, perplexity +1.7%, weights −19%.
|
||
|
||
Note the shape of that: **decode barely moves, prefill nearly doubles.** Decode at batch-1 is
|
||
memory-bandwidth-bound and the weights are 4-bit under either scheme, so there is little to win;
|
||
prefill is compute-bound, which is where native FP4 tensor cores replace the Marlin
|
||
dequantize-to-BF16 path. If someone promises you a big *decode* win from a scheme change, be
|
||
skeptical — and go measure §5 before believing it.
|
||
|
||
**The accuracy cost is real and is paid on purpose.** Operator ruling 2026-08-15: the ~1.7%
|
||
perplexity is an acceptable price for the speed. Settled — don't re-litigate. For correct
|
||
attribution: it is the **activation**-quantization cost (A4/A8 vs BF16 activations), *not* an MTP
|
||
cost. Turning MTP off does not recover it; only reverting the quant does.
|
||
|
||
---
|
||
|
||
## 2. The reference recipe (mixed-precision)
|
||
|
||
Lifted from `unsloth/Qwen3.8-27B-NVFP4` and replicated in-house. **Prefer replicating a published
|
||
recipe from a reputable quantizer over inventing one** — they have already paid for the
|
||
sensitivity analysis.
|
||
|
||
| group | scheme | targets |
|
||
|---|---|---|
|
||
| `group_0` | FP8 W8A8 — channel weights (static) + per-token dynamic activations | `self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`, `lm_head`, **the last 8 layers' MLPs** |
|
||
| `group_1` | NVFP4 W4A4 — `tensor_group` gsize 16, fp8 scales, `imatrix_mse` weights, `dynamic:"local"` activations | **all remaining** MLP `{gate,up,down}_proj` |
|
||
| kv cache | FP8 static tensor | |
|
||
| ignore | vision tower, `linear_attn.{norm,in_proj_a,in_proj_b}`, `re:^mtp.*` | |
|
||
|
||
Three things in there are load-bearing and easy to drop:
|
||
|
||
- **Late layers stay FP8.** Holding the last ~8 layers' MLPs (and `lm_head`) at 8-bit is the
|
||
accuracy-preservation trick — late layers are the sensitive ones. Uniform W4A4 is what collapses.
|
||
- **`imatrix_mse` on the W4A4 weights**, not `memoryless_minmax`. Importance-weighted; needs
|
||
calibration data.
|
||
- **Group targets must be non-overlapping.** Do not let `group_1`'s `.*mlp\..*` also match the
|
||
late layers and rely on group precedence to sort it out. Enumerate the early layers explicitly
|
||
(`re:.*layers\.([0-9]|[1-4][0-9]|5[0-5])\.mlp\.…`) and **prove it** with a dry run (§4.1).
|
||
|
||
**Toolchain:** `pip install llmcompressor` into stock `vllm/vllm-openai:latest` gives
|
||
llmcompressor 0.13 + compressed-tensors 0.18 without disturbing torch/transformers.
|
||
**Avoid nvidia-modelopt** — see §3.4.
|
||
|
||
---
|
||
|
||
## 3. The recurring landmines
|
||
|
||
Ordered by how much time each has cost. Every one of these has bitten more than once.
|
||
|
||
### 3.1 "W4A8" is not a servable shape
|
||
|
||
vLLM's compressed-tensors dispatcher (`compressed_tensors.py:704-713`) accepts NVFP4 weights with
|
||
**exactly two** activation settings:
|
||
|
||
| `input_activations` | result |
|
||
|---|---|
|
||
| `None` | W4A16 — and it **forces the Marlin kernel** (`kernels/linear/__init__.py:881-883`) |
|
||
| NVFP4 | W4A4, native |
|
||
|
||
Anything else — **FP8 included** — raises at load:
|
||
|
||
```
|
||
ValueError: For NVFP4 weights, input quantization must also be NVFP4 format, None for NVFP4A16
|
||
```
|
||
|
||
`CompressedTensorsW4A8Fp8` exists but is **INT4** weights (`W4A8_SUPPORTED_TYPES_MAP = {4: int4}`)
|
||
gated on `_check_scheme_supported(90, match_exact=True)` — Hopper-exact, so on Blackwell (sm_120)
|
||
it is closed twice over. **FP8 enters per-layer-group, never as activations on NVFP4 weights.**
|
||
|
||
*Cost: one queued task written against an impossible scheme.*
|
||
|
||
### 3.2 Wrong loader class → silent weight-load failure
|
||
|
||
**Rediscovered three times.** Load the model through the class vLLM actually serves — the
|
||
`…ForConditionalGeneration` / `…ForImageTextToText` **wrapper**, never `AutoModelForCausalLM`.
|
||
|
||
`AutoModelForCausalLM` resolves a VL config to the text-only inner class and saves a **flat**
|
||
config with `model.layers.*` keys. vLLM's weight mapper wants `model.language_model.*` (+
|
||
`model.visual.*`). The mismatch does not error — **every layer silently fails to load** and you
|
||
get `!!!!` gibberish, or an engine that rejects the checkpoint outright.
|
||
|
||
*Bit: heretic2 (gibberish), Dark-Scarlett (both vLLM and SGLang refused the checkpoint), and the
|
||
2026-08 rounds.*
|
||
|
||
### 3.3 The MTP head — three separate ways to lose it
|
||
|
||
Speculative decoding is a large fraction of the seat's throughput. It fails **silently**: the
|
||
model serves fine, just at 0% acceptance.
|
||
|
||
1. **The wrapper class does not instantiate `mtp.*`,** so the quant drops it. Post-quant you must
|
||
graft the BF16 `model-mtp.safetensors` back and register its tensors in the output index.
|
||
2. **`re:^mtp.*` must be in `quantization_config.ignore`** — else vLLM loads the grafted BF16 head
|
||
as though quantized, it comes up **uninitialised**, and acceptance is 0%.
|
||
3. **⭐ llm-compressor PRUNES `ignore` entries that matched no module at quant time.** Since the
|
||
wrapper never loaded `mtp.*`, the entry matches nothing and is **silently deleted from the
|
||
saved config — even though you put it in the recipe.** So it must be re-injected *after* the
|
||
graft, and then **verified, not assumed.**
|
||
|
||
*Cost: three rounds. The verify step caught it live on the third.*
|
||
|
||
There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not propagate
|
||
modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix
|
||
(needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4.
|
||
|
||
### 3.8 ⭐⭐ Multi-turn degeneration from TWO real compounding causes — how they masked each other
|
||
|
||
The most expensive diagnosis this project has had, because there were **two real
|
||
causes at once** and each partial fix moved the needle enough to look like *the*
|
||
answer. Recorded precisely because the first write-up of this section
|
||
over-attributed it to the quant alone; that was wrong.
|
||
|
||
**Cause 1 (real, upstream): the vLLM `qwen3_5_mtp` × Gated-DeltaNet bug.**
|
||
Confirmed by two cross-frontier peers and the tracker (vllm#47087 symptom-twin,
|
||
#43559 fix lineage, #51113 fix): the GDN recurrent state cannot roll back on a
|
||
partial draft-accept, so speculative decoding corrupts it, worse with context.
|
||
Architectural — vLLM/SGLang/llama.cpp mainline all shared it. **Genuinely fixed
|
||
enough** by moving to vLLM **nightly** (`v0.27.2rc1.dev150+`, carries #51113):
|
||
the operator reported it "significantly better" — this was a real bug, not just
|
||
an amplifier.
|
||
|
||
**Cause 2 (real, quant): full W4A4 is mildly subpar, per the known gradient.**
|
||
`sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4` is **full** W4A4 — 4-bit
|
||
*activations* on attention too, the bottom of the activation-precision ordering
|
||
already in §1: **W4A4 (A4) < W4+FP8 (A8) < W4+bf16 (A16)**. Not "defective," just
|
||
lowest-fidelity; on top of Cause 1 it degenerated ~15-20% of real multi-turn
|
||
generations. The FP8-attention **mixed** build (`qwen38-27b-uncensored-nvfp4-mixed`,
|
||
same base, same MTP, same nightly) sits a rung up that gradient and is coherent.
|
||
AEON was purged 2026-08-17 (operator ruled it no-good; re-pullable from HF).
|
||
|
||
**Why it cost days — and the process lessons that stand:**
|
||
1. **Two real causes compound and mask each other.** Each mitigation (MTP-off,
|
||
APC-off, the nightly #51113 fix) partially helped, so each looked like the fix
|
||
and then failed in real use. When a mitigation "helps but doesn't fix," suspect
|
||
a *second* cause rather than a wrong one.
|
||
2. **Stochastic degeneration (~15-20%) is nearly invisible to a small synthetic
|
||
probe** — a 7-turn run passes ~4 in 5. n=1 "clean" proves nothing; this class
|
||
needs many runs or the operator's real high-volume use. Three non-fixes were
|
||
"validated" by a single clean probe here.
|
||
3. **Isolate the WEIGHTS in parallel with the serving flags, not after.** Swapping
|
||
to a different quant of the same base (AEON→mixed) is what finally separated
|
||
Cause 2 from Cause 1; doing it earlier would have shortened the hunt. But note
|
||
it would NOT have found Cause 1 — the vLLM bug was real and needed the nightly.
|
||
4. **Prefer FP8 attention (the §2 mixed recipe) over full W4A4** for a coherence-
|
||
sensitive seat. AEON passed every static gate (abliteration 4/4, surface 6/6, a
|
||
36k needle, 52% acceptance) and was still the lower-fidelity of the two.
|
||
|
||
Current primary gen: the mixed FP8-attention build on pinned vLLM nightly with
|
||
MTP, until the DavidAU Qwen3.8 lands. A W4+bf16 (W4A16) build would be higher
|
||
fidelity still (§1) at a prefill cost — an option if the mixed build ever proves
|
||
marginal.
|
||
|
||
### 3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn
|
||
|
||
§3.3 is about *losing* the head (0% acceptance, silent). This is the opposite and
|
||
worse failure: the head loads, acceptance looks healthy, single-turn output is
|
||
perfect — and then it **corrupts multi-turn conversations** once cumulative context
|
||
passes **~2,000 tokens**. The reply collapses in length *and* bleeds earlier turns
|
||
into the current answer (a "describe durian" reply that contained the Krebs-cycle
|
||
and winter answers from three turns back). Single-turn probes and the acceptance
|
||
gate (§5) **do not catch it** — it only appears as accumulated context grows.
|
||
|
||
Isolated 2026-08-16 (operator-confirmed), each step measured on a fixed 7-turn probe:
|
||
|
||
- **Not the serving gateway, not sampling, not repetition/template.** Identical
|
||
input gateway-vs-direct behaves the same; presence_penalty 1.5/0.5/0.0 all
|
||
collapse; higher temperature collapses harder; a conversation of *unrelated*
|
||
topics collapses at the same ~2k tokens as a repetitive one → it is context-
|
||
length-driven, not template lock-in.
|
||
- **Model-independent across every Qwen3.8-27B quant** (AEON W4A4, unsloth
|
||
FP8-attn, our in-house mixed) — so not a quant-brand or scheme artifact.
|
||
- **DECISIVE: same model + same conversation, MTP OFF → coherent through 4k+
|
||
tokens, zero bleed.** Toggle it back on → collapse returns. MTP is the cause.
|
||
|
||
**Qwen3.6-27B running the same `qwen3_5_mtp` method is CLEAN.** So the 3.6 MTP
|
||
head/graft is fine and the 3.8 one is not — suspects: the bf16 graft being subtly
|
||
wrong for the 3.8 head, or the vLLM `qwen3_5_mtp` impl diverging at `num_speculative_tokens=3`.
|
||
Open upstream question (queried dvalin/bil-smithy 2026-08-17).
|
||
|
||
**Rule: gate MTP on a MULTI-TURN coherence probe, not just single-shot acceptance.**
|
||
Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens
|
||
for length-collapse and cross-turn bleed.
|
||
|
||
⛔ **THE APC-OFF MITIGATION DID NOT HOLD — SUPERSEDED 2026-08-17, see §7.** What
|
||
follows is kept for its history and **must not be applied**: *"disable prefix caching,
|
||
keep MTP. The corruption is gated on MTP × prefix-caching together (vllm#43559 /
|
||
#47194); with `--no-enable-prefix-caching` the buggy partial-accept align-path is
|
||
inert. Confirmed on our stack: AEON W4A4, MTP on + prefix-caching off → the 7-turn
|
||
varied series stays coherent through 3.9k tokens, zero bleed, at 104.6 tok/s / 53.6%
|
||
acceptance."*
|
||
|
||
**It passed that synthetic 7-turn probe and the operator still saw severe degeneration
|
||
in real use.** Reverted the same day. The probe was structurally under-covering the
|
||
real workload on both content distribution and depth — which is §3.7's own standing
|
||
rule (*gate on a multi-turn coherence probe, not single-shot acceptance*) failing at
|
||
one level up: the multi-turn probe was itself too small to gate on.
|
||
|
||
**WHAT ACTUALLY RESOLVED IT.** The multi-day hunt root-caused to the **AEON W4A4 quant
|
||
being defective** — ~15-20% of generations went degenerate — with MTP, prefix-caching
|
||
and the gateway all merely *amplifying* it. That is why every partial mitigation
|
||
"helped" without fixing anything (§3.8). The gen seat today runs the in-house
|
||
Heretic **mixed NVFP4+FP8** build (FP8 attention, not W4A4) on vLLM nightly carrying
|
||
#51113, with **MTP ON and prefix-caching ON**, and is coherent in real use.
|
||
|
||
Verified against the live seat 2026-08-26: `vllm-gen` runs `--enable-prefix-caching`
|
||
with `qwen3_5_mtp` / `num_speculative_tokens 3`. The compose file
|
||
(`stacks/gen-seat/compose.yaml`) carries the full history inline and is the current
|
||
authority; this section was stale against it for nine days.
|
||
|
||
Things that do **not** work, ruled out: `num_speculative_tokens=1` (corruption is
|
||
depth-independent — reproduces at n=1 and n=2, deterministically probed upstream);
|
||
switching engine (vLLM / SGLang / llama.cpp mainline all share the GDN-rollback
|
||
bug — it is architectural). The proper upstream fix (vllm#51113) is in `main` /
|
||
`v0.27.2rc0` only — not in a stable release, so we hold at APC-off until it lands.
|
||
Two cross-frontier peers (dvalin/bil-smithy) confirmed the bug class and pointed
|
||
at the open symptom-twin issue #47087.
|
||
|
||
### 3.15 ⭐⭐ Fused 3-D MoE experts are INVISIBLE to a `targets=["Linear"]` recipe
|
||
|
||
**Symptom: none.** The quant completes, the artifact loads, and 88.5% of the
|
||
model is still BF16. Nothing warns you.
|
||
|
||
Modern MoE checkpoints store each layer's experts as **two fused 3-D
|
||
`nn.Parameter` tensors**, not as N `nn.Linear` modules. Gemma-4 26B-A4B:
|
||
|
||
model.language_model.layers.N.experts.gate_up_proj BF16 [128, 1408, 2816]
|
||
model.language_model.layers.N.experts.down_proj BF16 [128, 2816, 704]
|
||
|
||
Note the **absent `.weight` suffix** — that is the tell. `mlp.down_proj.weight`
|
||
is an `nn.Linear`; `experts.down_proj` is a bare parameter.
|
||
|
||
Measured on that checkpoint, recipe targeting `["Linear"]`:
|
||
|
||
Linear modules 427
|
||
WILL quantize 205 (experts: 0) <- 22.84 B params untouched
|
||
|
||
**This is the same defect that killed QLoRA on this architecture** —
|
||
`bitsandbytes` 4-bit replacement also walks `nn.Linear` modules and also
|
||
silently skipped the experts. Two different tools, one blind spot, because the
|
||
blind spot is in the *checkpoint layout*, not the tool.
|
||
|
||
**The fix** (llm-compressor ≥ 0.12):
|
||
|
||
```python
|
||
from llmcompressor.modeling.moe.linearize import linearize_moe
|
||
model = SomeForConditionalGeneration.from_pretrained(...)
|
||
linearize_moe(model) # BEFORE building the recipe
|
||
```
|
||
|
||
Linear modules 11947
|
||
WILL quantize 11725 (experts: 11520) # 30 layers x 128 x 3 proj
|
||
|
||
`linearize_moe` unfuses the 3-D parameters into per-expert
|
||
`experts.N.{gate,up,down}_proj` Linears. **No registration is needed** if the
|
||
module satisfies `FusedExpertsProtocol` structurally — bare `down_proj` plus
|
||
`gate_up_proj`/`up_proj` Parameters. `load_quantizable_moe(model_cls)` is the
|
||
faster variant that linearizes during load rather than after.
|
||
|
||
**Always assert the expert count before spending GPU time** (§4.1). The
|
||
arithmetic is `layers × experts × projections`; if your target list does not
|
||
hit it exactly, the recipe is wrong and the failure is silent.
|
||
|
||
⚠ **Keep routers in `ignore`.** A 4-bit router picks *different experts* — that
|
||
error does not average out downstream, it changes which weights run at all.
|
||
|
||
### 3.4 Toolchain version deadlocks
|
||
|
||
Both directions have burned us, so the resolution is: **use llm-compressor / compressed-tensors,
|
||
not nvidia-modelopt.**
|
||
|
||
- modelopt **0.45** ↔ transformers 5.12: `mtq.quantize` dies `TypeError: issubclass() arg 2 must
|
||
be a class` (modelopt registers transformers' `FusedMoE`, a *function* in 5.x, as an nn class).
|
||
- modelopt **0.43** doesn't fix it — it drags transformers back to 4.57, which cannot load
|
||
`qwen3_5` at all.
|
||
- modelopt's config API also trails the current model families by a version.
|
||
|
||
### 3.5 Vision tower and its configs
|
||
|
||
- Keep the **vision tower in `ignore`** (BF16). Only the LLM backbone gets quantized.
|
||
- The wrapper-class save **drops `preprocessor_config.json`** (and the video one). Without it the
|
||
seat crash-loops `Can't load image processor`. Restore from the source — and if the upstream repo
|
||
omits it, **reconstruct it from `processor_config.json`'s `image_processor` sub-dict**.
|
||
|
||
### 3.6 Memory and device placement (large models)
|
||
|
||
- **`device_map=None`/`"cpu"`, never `"auto"`.** `auto` fills GPU0 and OOMs during un-fusing;
|
||
constraining with `max_memory` then offloads to the *meta* device, which cannot be `.copy_()`d.
|
||
CPU-resident keeps every tensor real; the sequential pipeline still onloads per-layer to GPU.
|
||
- **Avoid mmap on `/tank`.** `safetensors.safe_open()` mmaps a whole shard; on ZFS a 50 GB shard
|
||
ENOMEMs regardless of free RAM (MAP_SHARED never consults the commit limit). Read with plain
|
||
`read()` + `load(bytes)`, one shard cached at a time.
|
||
- **`vm.overcommit_memory=1`** on ana-ml2 (durable via `playbooks/ana-ml2-overcommit-memory.yaml`).
|
||
|
||
### 3.9 ⭐⭐ A sharded forward can be silently WRONG — never trust `device_map="auto"` for activations
|
||
|
||
Splitting **Qwen3.8-27B (Qwen3_5 hybrid)** across the two Blackwells with `device_map="auto"`
|
||
produces a model that loads clean, reports no error, and computes **garbage**: the residual stream
|
||
collapses to **exactly zero** a couple of layers past the GPU0→GPU1 boundary, and the logits decode
|
||
to rubbish (`'8'`, `'�'`, `'b'`). Every layer *below* the boundary stays healthy, deterministic, and
|
||
bit-identical to a single-GPU run — which is what makes it so dangerous. A capture that reads a
|
||
low layer looks perfectly plausible and is fine; one that reads a high layer is reading zeros, and
|
||
nothing in the pipeline says so. Measured 2026-08-20 (§9 Cold-Fusion).
|
||
|
||
**Rule: any workload that reads activations — refusal-direction capture, calibration, activation
|
||
statistics, PPL — must run on ONE device.** Sharding is for *storage*, and it is only safe when you
|
||
consume the model's final output through an engine that was built for it (vLLM does TP correctly;
|
||
`device_map="auto"` in transformers is not the same thing). If it does not fit on one card, shrink
|
||
the model, not the guarantee: **truncating the decoder to N layers is exact** for any activation
|
||
read at a layer < N (a causal stack's layer-N state cannot depend on layers above N), and it is
|
||
cheap — verified by reproducing the full model's layers 18/20/22/26 bit-for-bit.
|
||
|
||
**Gate it, don't remember it.** Assert single-device residency and zero offload before the forward:
|
||
|
||
```python
|
||
dmap = getattr(model, "hf_device_map", {}) or {}
|
||
gpus = {str(v) for v in dmap.values()} - {"cpu", "disk"}
|
||
offloaded = [k for k, v in dmap.items() if str(v) in ("cpu", "disk")]
|
||
if len(gpus) > 1 or offloaded:
|
||
sys.exit("residency gate FAILED — sharded/offloaded forward reads garbage")
|
||
```
|
||
|
||
### 3.10 ⭐⭐ `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` corrupts retained tensors
|
||
|
||
On torch 2.12+cu130 / Blackwell, tensors that **outlive their allocation** come back corrupted with
|
||
this flag set: captured hidden states carried Inf / NaN / zeros that **moved between bit-identical
|
||
forwards** (same input, same weights → a different layer corrupted each time). Unset, the identical
|
||
forwards are exactly reproducible. Several runbooks recommend this flag for headroom on large
|
||
loads; for anything that *keeps* activations it buys corruption.
|
||
|
||
Two tells that distinguish this from a real numerical blowup, both worth knowing because they
|
||
generalise: a genuine blowup **propagates** to later layers and is **deterministic**. Corruption
|
||
does neither — downstream layers were finite and consistent, and the affected layer moved run to
|
||
run. **If a "NaN" fails to propagate, stop debugging the math and start debugging memory.**
|
||
|
||
Corollary: **do not read `output_hidden_states=True` off a returned object** on a large multi-device
|
||
load. Take what you need *during* the forward with a `register_forward_pre_hook` that clones to CPU
|
||
immediately — it closes the reuse window and never retains a `[B, seq, hidden]` tensor per layer, so
|
||
it is cheaper than the thing it replaces.
|
||
|
||
### 3.11 Determinism is a necessary check, not a sufficient one
|
||
|
||
Both defects above were found by the cheapest possible test — **run the same input twice and diff**
|
||
— which no amount of eyeballing plausible-looking numbers would have caught. Add it to any
|
||
activation-reading pipeline. But note the trap that followed: after fixing the allocator, the run
|
||
went perfectly "deterministic" *because the corrupted layers were now stably zero*. Pair the
|
||
determinism check with a **magnitude** check (residual norms should grow smoothly with depth; an
|
||
exact 0.0 mid-stack is impossible) and, where you can, a **coherence** check (generate 40 tokens and
|
||
read them).
|
||
|
||
### 3.12 ⭐⭐ You cannot free a 27B model in-process — give each model its own process
|
||
|
||
Any A/B that loads two large checkpoints in sequence (KL, logit diffing, teacher-vs-student)
|
||
will try to release the first before loading the second. **On this stack, it does not work.**
|
||
Measured 2026-08-20 on Qwen3.8-27B bf16, free VRAM after each attempt:
|
||
|
||
| teardown | free VRAM |
|
||
|---|---|
|
||
| `del model` + `gc.collect()` + `torch.cuda.empty_cache()` | 45,287 MiB |
|
||
| same, with the model confined to an inner frame that exits | 45,287 MiB |
|
||
| **the process exits** | **97,247 MiB** |
|
||
|
||
The ~51,300 MiB of weights stayed resident through both in-process teardowns. The first
|
||
run survived only because **PyTorch's allocator hit OOM on the second load, ran a collection
|
||
itself, and retried** — the second model landed by rescue, not by design. That is not a
|
||
release strategy: on an architecture where a silent CPU offload does not raise (§3.9), the
|
||
day the retry does not fire you get confident garbage instead of an error.
|
||
|
||
**Do this instead:** one process per model, hand results to disk between them
|
||
(first-token log-probs for a 250k vocab are ~715 MiB per model — nothing), and gate each
|
||
stage on free VRAM *before* the load. Reference implementation:
|
||
`services/coldfusion-abliteration/kl_divergence.py` (`--stage ref|cand|score`).
|
||
|
||
Two gate corollaries learned in the same session:
|
||
|
||
- **⭐ A residency gate that reads `hf_device_map` cannot fail.** The map is **empty**
|
||
whenever transformers puts the whole model on one device, so the check reports
|
||
"unsharded" both when everything is fine and when there is nothing to inspect. Read
|
||
`{p.device for p in model.parameters()}` — ground truth in every case. (Generalises
|
||
[[feedback_assert_effective_value_not_substring]]: presence of a passing check is not
|
||
evidence of a check that can fail.)
|
||
- **⭐ Size VRAM from the checkpoint's own headers, never from a remembered figure.** A
|
||
runbook carried "bf16 is 50 GB"; the real number was 50.10 **GiB** = 51,300 MiB of
|
||
text-only weights. That 3.7 GB unit error is exactly the difference between "stop one
|
||
co-tenant" and "stop both", and it cost an aborted window. Sum the safetensors header
|
||
offsets (excluding tensors the loader class won't instantiate — vision, MTP); read only
|
||
the 8-byte length prefix + JSON header, never `safe_open`, which mmaps the whole shard
|
||
and ENOMEMs on ZFS (§ *Avoid mmap on `/tank`*).
|
||
|
||
### 3.13 ⭐⭐ The observer you ASKED for is not necessarily the observer you GOT
|
||
|
||
`quant_mixed_nvfp4.py` sets `observer="imatrix_mse"` on the NVFP4 W4A4 group. It has
|
||
**never once been used.** llm-compressor looks for importance data, finds none, and
|
||
silently degrades:
|
||
|
||
```
|
||
_get_validated_importance | WARNING - imatrix_mse: no importance data available.
|
||
Falling back to uniform MSE.
|
||
```
|
||
|
||
Confirmed on the 2026-08-20 09:59 incumbent quant **and** the 22:45 Heretic-300
|
||
quant; `find /tank/aimodels -iname "*imatrix*" -o -iname "*importance*"` returns
|
||
nothing. Every NVFP4 build in the fleet has run uniform MSE while the recipe claimed
|
||
importance weighting.
|
||
|
||
**Why it went unseen for months:** the warning scrolls past inside a tqdm progress
|
||
bar during a ~20 minute quant. It is only visible if you read the log while it runs.
|
||
|
||
**The generalisable rule, which is bigger than imatrix.** A quantizer, optimiser or
|
||
observer that *silently falls back to a weaker default* is a whole class of invisible
|
||
quality loss — the config is accepted, nothing errors, the artifact benchmarks
|
||
plausibly, and you never learn you got the cheap path. So:
|
||
|
||
- **Grep the quant log for `WARNING`, `Falling back`, `not available`, `ignoring`
|
||
before trusting an artifact.** Make it a step, not a habit.
|
||
- **Assert the effective setting, never the requested one** — the same rule as
|
||
[[feedback_assert_effective_value_not_substring]], applied to quantizer internals
|
||
rather than config files.
|
||
- If the fallback turns out to be unavoidable in your toolchain version, **change the
|
||
recipe to say what it actually does.** A recipe line that silently lies is worse
|
||
than one that admits a limitation.
|
||
|
||
⚠️ **Do not "fix" this by assuming an imatrix would help.** Verify first that your
|
||
llm-compressor version can consume an externally supplied importance matrix at all,
|
||
and in what format. Parked as `park/nvfp4-recipe-asks-for-imatrix-mse-but-silently-2`
|
||
(id 42) with the calibration corpus that would feed it.
|
||
|
||
✅ **Comparisons already made remain valid.** Because *every* build shares the
|
||
fallback, the incumbent-vs-candidate A/Bs (47.2% acceptance, PPL 6.910, and the
|
||
2026-08-20 Heretic-300 build) are apples-to-apples. This is unrealised upside, not a
|
||
correction to past numbers.
|
||
|
||
### 3.16 Weight-only NVFP4A16 with a minmax observer is DATA-FREE — your calibration corpus is ignored, but its tokenizer side-effect is not
|
||
|
||
Measured 2026-09-08 (Gemma-4 26B-A4B MoE, ERP run 6, llm-compressor 0.13): with
|
||
`scheme="NVFP4A16"` (default `memoryless_minmax` weights, no activation quant) llm-compressor
|
||
logs `Inferred DataFreePipeline for QuantizationModifier` and never touches the dataset — the
|
||
whole 26B quant ran in ~90 s on one Blackwell. Two consequences: (1) do not budget calibration
|
||
time or believe a corpus "shaped" the result — only `imatrix_mse`/activation observers consume
|
||
data; (2) building the calibration set still calls the fast tokenizer with
|
||
`truncation=True, max_length=N`, so §3.14's baked cap (`max_length: 8192` here) lands in the
|
||
saved `tokenizer.json` **even though no calibration happened**. The §4.3 post-step caught it.
|
||
Reference: `services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py` (linearize_moe + assert
|
||
11,520 expert Linears + post-steps; the published `prithivMLmods/gemma-4-26B-A4B-it-NVFP4A16`
|
||
recipe replicated, 222→252 ignore entries with audio/norm/router regexes added).
|
||
|
||
### 3.14 ⭐⭐ Calibration BAKES a truncation cap into the shipped tokenizer
|
||
|
||
**Symptom (on a newer transformers, at startup, on a vision model):**
|
||
|
||
```
|
||
ValueError: Mismatch in `image` token count between text and `input_ids`.
|
||
Got ids=[2047] and text=[16384]. Likely due to `truncation='max_length'`.
|
||
```
|
||
|
||
The engine never serves a request. The number in `ids=[…]` is your **calibration seqlen minus
|
||
one**, which is the tell.
|
||
|
||
**Cause — an in-place mutation you never wrote.** Calibration tokenizes like this:
|
||
|
||
```python
|
||
tok(b["text"], truncation=True, max_length=seqlen, add_special_tokens=False)
|
||
```
|
||
|
||
For a **fast** tokenizer that call does not just return ids — it **mutates the Rust backend's
|
||
truncation state in place**. A later `tok.save_pretrained(out)` then persists it:
|
||
|
||
```json
|
||
"truncation": {"direction": "Right", "max_length": 2048, "strategy": "LongestFirst", "stride": 0}
|
||
```
|
||
|
||
The source model has `"truncation": null`. **You shipped a tokenizer that clamps every prompt at
|
||
the calibration length, permanently.**
|
||
|
||
**Why it hid for months.** Older transformers does not enforce the text-vs-ids count check, so
|
||
the cap sits latent — the model serves, gates pass, vision works, nothing logs. It only detonates
|
||
when you bump the image, and then it presents as a *vision* bug at startup with no mention of
|
||
tokenizers. It also caps the effective image resolution long before it kills the seat: at a 2048
|
||
cap the largest servable image is ~1448×1448, because `(edge/patch)² / merge²` image tokens must
|
||
fit under it.
|
||
|
||
**The fix — never save the calibration tokenizer.** Re-read a pristine one from the source:
|
||
|
||
```python
|
||
from transformers import AutoTokenizer as _AutoTokenizer
|
||
_AutoTokenizer.from_pretrained(a.model, trust_remote_code=True).save_pretrained(a.out)
|
||
```
|
||
|
||
then **assert** it, because this is exactly the class of defect that returns silently:
|
||
|
||
```python
|
||
if json.load(open(f"{a.out}/tokenizer.json")).get("truncation"):
|
||
raise SystemExit("FAILED CHECK: saved tokenizer carries a truncation cap")
|
||
```
|
||
|
||
Both live in `quant_mixed_nvfp4.py` as of 2026-08-22.
|
||
|
||
**Audit any build predating that.** One line per model:
|
||
|
||
```bash
|
||
python3 -c 'import json,sys;print(json.load(open(sys.argv[1]+"/tokenizer.json")).get("truncation"))' <model_dir>
|
||
```
|
||
|
||
Measured 2026-08-22 — every mixed-NVFP4 build from this pipeline was affected, and the two live
|
||
ones were corrected in place (backup `tokenizer.json.bak-truncation-20260822`; only the
|
||
`truncation` field changed, vocab and `added_tokens` byte-identical):
|
||
|
||
| build | truncation as found |
|
||
|---|---|
|
||
| `qwen38-27b-orcarouter-nvfp4-mixed` (live `gen`) | **2048** → fixed |
|
||
| `mog-sec-27b-nvfp4-mixed` (live `sec`) | **2048** → fixed |
|
||
| `qwen38-27b-heresy-nvfp4-mixed` (retired) | 2048, left as-is |
|
||
| `G4-MeroMero-v2-31B-NVFP4A16` (different pipeline) | `null` ✓ |
|
||
| `mog-sec-27b-bf16` (source) | `null` ✓ |
|
||
|
||
**Editing it is safe on a running seat** — vLLM reads the tokenizer at startup and holds its own
|
||
copy, so the fix lands on the next restart with no disruption.
|
||
|
||
**The general lesson, which is the transferable part:** this is the third defect in this playbook
|
||
where *the artifact carries config authored against an older transformers and a newer one starts
|
||
enforcing it* (see also the Gemma-4 heterogeneous `head_dim`). **Treat "we bumped the image" as a
|
||
config-compatibility event, not just a version change** — and prefer saving artifacts re-read
|
||
from the source over saving objects the pipeline has touched.
|
||
|
||
---
|
||
|
||
## 4. Pipeline shape
|
||
|
||
### 4.1 Prove the targets before spending GPU time
|
||
|
||
Enumerate module names from the safetensors index and check your regexes against them: **zero
|
||
overlap between groups, and the union covers every layer you intended.** This is free, takes
|
||
seconds, and catches a mis-scoped regex that would otherwise surface as a mystery quality
|
||
regression hours later. Reference: `services/gen-seat-mixed-quant/validate_targets.py`.
|
||
|
||
### 4.2 Quantize
|
||
|
||
Calibration data matters for `imatrix_mse` + static activation observers. We use
|
||
`/tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl` (512 chat samples, RP/GM-flavoured
|
||
— appropriate for our seats). 256 samples @ 2048 tokens ≈ 20 min for a 27B on one Blackwell.
|
||
|
||
### 4.3 The mandatory post-steps
|
||
|
||
Never optional, always in this order, and the last one **verifies rather than assumes**:
|
||
|
||
1. Graft `model-mtp.safetensors` + register its tensors in the output index.
|
||
2. Restore `preprocessor_config.json` / `processor_config.json` / `video_preprocessor_config.json`.
|
||
3. **Re-inject `re:^mtp.*` into `quantization_config.ignore` and confirm it is there** (§3.3).
|
||
4. **Confirm the saved `tokenizer.json` has `truncation: null`** (§3.14) — calibration mutates the
|
||
fast tokenizer in place and `save_pretrained` bakes the cap in. Latent on an older
|
||
transformers, fatal on a newer one.
|
||
|
||
Reference implementation: `services/gen-seat-mixed-quant/post_quant.py`.
|
||
|
||
### 4.4 Test on a temp port, never on the live seat
|
||
|
||
Serve the candidate on an alt port with the live seat's **exact** flags, run the gate (§5), and
|
||
only then flip `.env`. Keep the previous build on disk; rollback is one `.env` line.
|
||
|
||
---
|
||
|
||
## 5. The acceptance gate — and how measurement lies to you
|
||
|
||
Speed alone does not justify cutting over a shared seat. Gate on **all** of: decode tok/s, MTP
|
||
acceptance, perplexity, a behavioural surface test, and — for an abliterated model — that the
|
||
abliteration survived.
|
||
|
||
**Three ways the numbers have lied to us. All three produced confident, wrong results.**
|
||
|
||
1. **Prefix caching fakes both speed metrics.** A fixed prompt returns byte-identical timings run
|
||
after run; you are measuring cache, not compute. Worse for prefill: a *seeded* nonce
|
||
regenerates the previous run's prompts verbatim and reads **~41k tok/s of cache-hit instead of
|
||
~5k of real prefill**. Use a fresh unseeded nonce per request; never seed a cache-buster.
|
||
2. **`prompt_logprobs` are garbage while speculative decoding is on** — ~uniform over the vocab
|
||
(median rank ~10⁵; " Paris" after "The capital of France is" ranked 69698). **Perplexity must be
|
||
measured on a seat served without `--speculative-config`,** on both sides of the comparison.
|
||
3. **A 0600 `.env` makes `docker compose` silently no-op.** Without `sudo` it fails
|
||
`permission denied` reading `.env`, **leaves the old container running**, and reports success —
|
||
producing a full page of "benchmark results" that were just the unchanged baseline.
|
||
**Hard-verify the change landed against `docker inspect …Config.Cmd`.**
|
||
|
||
**Re-measure the baseline before believing a target.** The 2026-08-15 handoff quoted ~68 tok/s;
|
||
cache-busted, the incumbent was already doing 80.1 — essentially the *target* of the work queued
|
||
against it. Had that not been re-measured, doing nothing would have looked like a 20% win.
|
||
|
||
**Cheap shortcut worth taking first:** if a reputable published quant of the same architecture is
|
||
already on-box (or is a small pull), **serve it as a probe and measure it** before committing
|
||
hours to your own. It answers "is this gain even real?" in ten minutes *and* hands you the recipe.
|
||
|
||
Harness: `services/gen-seat-mixed-quant/bench/` — `quickbench.py` (decode + acceptance),
|
||
`prefill_bench.py`, `eval_quality.py` (PPL + abliteration), `surface_test.py` (chat, vision, tools,
|
||
thinking split, long-context needle, streaming), `serve_probe.sh`.
|
||
|
||
---
|
||
|
||
### 5.1 ⭐⭐ Acceptance is not throughput — always run the DEPTH control
|
||
|
||
**Measured 2026-08-22**, same instrument (vLLM's own `spec_decode` counters, delta over a fixed
|
||
workload, temp 0), same target, same engine:
|
||
|
||
| config | accepted tok/forward | throughput |
|
||
|---|---|---|
|
||
| MTP k=3 | 2.753 | 114.9 tok/s |
|
||
| MTP k=7 | **3.041** ⬆ | **74.0 tok/s** ⬇ |
|
||
|
||
**Raising `num_speculative_tokens` improved acceptance and destroyed throughput.** Reporting
|
||
acceptance alone would have recommended a 36% regression.
|
||
|
||
**Why:** a single-module MTP head (`mtp_num_hidden_layers: 1`, one `mtp.layers.0`) has no depth
|
||
of its own — vLLM runs it **autoregressively**, so k draft tokens cost **k sequential forward
|
||
passes**. Past a shallow depth the drafting cost exceeds what the extra accepted tokens save.
|
||
Check `mtp_num_hidden_layers` before assuming depth is cheap.
|
||
|
||
**The rule: when comparing two speculative methods, match k, or you are measuring depth rather
|
||
than method.** A parallel-drafting drafter (DFlash2 and kin, which propose a whole block in one
|
||
pass) at k=7 versus an autoregressive MTP at k=3 is not a method comparison — the depth control
|
||
is what separates them. In our case the control showed most of the apparent acceptance win was
|
||
depth, while the *throughput* win was real and came from parallel drafting, not better drafts:
|
||
our MTP was **better at position 0** (79.6% vs 75.4%) and still lost overall.
|
||
|
||
**Corollary — report both, always.** Acceptance rate, mean accepted length, and end-to-end
|
||
tok/s. Any one of the three alone can point the wrong way.
|
||
|
||
---
|
||
|
||
## 6. Hardware and co-residency
|
||
|
||
- **ana-ml2 = Blackwell (sm_120)**, 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths
|
||
(`match_exact=True` on sm90) are **closed** here — do not plan around them.
|
||
- **irv-ml1 = Ampere (sm_86)**, 3090 + A6000. **No native FP8/FP4** — 4-bit there is a VRAM saving
|
||
only, not a speed win. Don't port a Blackwell scheme over and expect the throughput.
|
||
- **GPU co-residency is a zero-sum budget, and a *smaller* model can break its neighbour.**
|
||
`gpu-memory-utilization` is a fraction of the *whole card*, so when new weights are smaller the
|
||
seat absorbs the slack as extra KV rather than releasing it. That is exactly how a −5.2 GB
|
||
requant left the co-resident seat **0.18 GiB** short and crash-looping. **After any requant,
|
||
re-check both seats' budgets** and hand the space back explicitly.
|
||
|
||
---
|
||
|
||
## 7. Superseded claims — do not follow these
|
||
|
||
Old docs stay for their history, but these specific claims are **false now** and will cost you a
|
||
day if followed:
|
||
|
||
| claim | where | status |
|
||
|---|---|---|
|
||
| "Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance" | `docs/runbooks/heretic2-nvfp4-mtp-seat.md` §landmine 2 | **SUPERSEDED 2026-08-14.** The 0% was the missing `re:^mtp.*` ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.7–83.2% acceptance, live. Use compressed-tensors. |
|
||
| "Abliteration desyncs the MTP head → uncensored models can't do MTP" | earlier auto-memory | **SUPERSEDED 2026-08-14.** A modest abliteration preserves MTP (83.7% at bf16). Test MTP on **bf16 first** to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source. |
|
||
| "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). |
|
||
| "The Qwen3.8 MTP corruption is fixed by disabling prefix caching while keeping MTP; the gen seat runs APC-off" | this playbook §3.7 (now marked), earlier auto-memory | **SUPERSEDED 2026-08-17, and the staleness was only caught 2026-08-26.** APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use; reverted the same day. The real cause was the **AEON W4A4 quant being defective** (~15-20% degenerate generations), with MTP / prefix-caching / gateway merely AMPLIFYING it (§3.8). The gen seat runs **MTP ON and prefix-caching ON** on the in-house mixed NVFP4+FP8 build — verified against the live container 2026-08-26. ⚠ The lesson inside the lesson: a *passing multi-turn probe* was not sufficient evidence either. |
|
||
| "transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 without causal-conv1d; it is precision-driven cancellation and fp32 resolves it" | `services/coldfusion-abliteration/README.md`, `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` | **SUPERSEDED 2026-08-20.** Precision was never the variable. The NaN came from **multi-GPU sharding** and **`expandable_segments`** (§3.9, §3.10); fp32 only made it rarer, which is worse than failing. On one GPU with a plain allocator, **bf16 is exactly deterministic through all 64 layers and generates coherent prose** — at 50 GB and 4.3× the throughput of the 111 GB fp32 it replaced. |
|
||
|
||
---
|
||
|
||
## 8. Measured negatives — don't re-chase
|
||
|
||
- **`num_speculative_tokens` = 3 is optimal** on the Qwen3.8-27B seat. Swept: n=2 → 77.1,
|
||
**n=3 → 80.1**, n=4 → 78.7, n=5 → 75.9 tok/s. Higher n trades acceptance for draft width and
|
||
loses. Re-sweep only if the drafter architecture changes.
|
||
- **Uniform W4A4** — see §7 row 3.
|
||
- **Dense-VL as the anatomy judge** — A/B'd, MoE retained. Don't re-propose.
|
||
|
||
---
|
||
|
||
## 9. Worked examples
|
||
|
||
Per-model artifacts. Read for *how a specific model went*, not for the general lessons — those are
|
||
above, and where the two disagree, **this file wins**.
|
||
|
||
| artifact | what it is |
|
||
|---|---|
|
||
| `services/gen-seat-mixed-quant/` | **current reference.** Mixed NVFP4+FP8 on Qwen3.8-27B-Uncensored: scripts, acceptance harness, raw measurements. |
|
||
| `stacks/gen-seat/README.md` | the live `gen` seat (7 LiteLLM aliases) |
|
||
| `stacks/meromero-charrp/README.md` | Gemma-4 seat — the **tool-call/reasoning-parser** trap (a parser default that returns null `content` for all prose) |
|
||
| `services/heretic2-nvfp4-quant/` | modelopt-format MTP seat — historical; see §7 before following it |
|
||
| `tools/mistral-small4-nvfp4/` | MoE + native-convert path; source of §3.6 |
|
||
| `docs/pfi/recommended-model-settings.md` | serve-time sampler/flag defaults (not quant) |
|
||
|
||
**A new model just dropped and needs requanting?** §1 → §2 → §4 → §5. Skim §3 first; it is the
|
||
part that costs hours.
|