docs(quant): consolidate quantization lessons into a durable playbook
Quants are hard-fought and we keep re-paying for the same lessons. A survey found quant knowledge scattered across 18 files in four trees, with three documents having independently discovered and recorded overlapping "landmines" sections — and one of them now actively misleading. Adds docs/pfi/model-quantization-playbook.md as the single home for the TRANSFERABLE lessons, with per-model artifacts demoted to worked examples that link up to it. Contents: - scheme decision table, incl. that a literal "W4A8" NVFP4 checkpoint is unservable on vLLM (two legal activation settings, FP8 is not one) - the reference mixed-precision recipe and the three parts of it that are load-bearing and easy to drop - the recurring landmines, ordered by cost: the loader-class trap (rediscovered THREE times), the three separate ways to lose the MTP head, toolchain deadlocks, vision configs, memory/device placement - pipeline shape: prove targets before spending GPU time; mandatory post-steps that verify rather than assume - the acceptance gate, and the three ways measurement has lied to us — prefix caching faking both speed metrics, prompt_logprobs going uniform under speculative decoding, and a 0600 .env making compose silently no-op - hardware/co-residency, including that a SMALLER model can starve its neighbour because gpu-memory-utilization is a fraction of the whole card - a superseded-claims table, and measured negatives not to re-chase The superseded table earns its place immediately: the heretic2 runbook tells readers to use modelopt because "compressed-tensors can't load the BF16 MTP head, 0% acceptance". That symptom was real but the cause was not the format -- it was the missing re:^mtp.* ignore entry. compressed-tensors gives 47.7-83.2% acceptance in production. A fresh session following that doc would be sent down the modelopt path that current memory calls dependency hell, so the runbook now carries a stale-warning header pointing here. Wires discovery: an orientation.md "Where to look for what" row, pointers from the gen-seat / heretic2 / mistral artifacts, and a CLAUDE.md maintenance rule so the playbook gets fed instead of going stale -- model-agnostic lessons land in the playbook, model-specific ones stay put, and a wrong claim earns a dated superseded row rather than a silent edit. Motivated by Qwen3.8 having just released: the next model swap will need a requant, and this is what that session should read first.
This commit is contained in:
@@ -212,6 +212,9 @@ These caught us once; don't let them catch you twice.
|
||||
| What's currently open / in-flight? | `STATUS.md` |
|
||||
| What do I need to know that isn't in current code? | `MEMORY.md` + the `.md` files it links |
|
||||
| Why did we do X? | Check memory files + `STATUS.md` session milestones at the bottom |
|
||||
| **I need to quantize / requant a model** | **`docs/pfi/model-quantization-playbook.md` — READ IT FIRST.** Consolidated hard-won lessons (scheme choice, the recurring landmines, the acceptance gate, superseded claims). Per-model runbooks are worked examples, not the general guide. |
|
||||
| What sampler/serve settings for model X? | `docs/pfi/recommended-model-settings.md` |
|
||||
| Which model is on which GPU seat? | `servers/ana-ml2/README.md` + `stacks/<seat>/README.md` |
|
||||
|
||||
## Inventory + automation scripts
|
||||
|
||||
|
||||
@@ -0,0 +1,282 @@
|
||||
# Model quantization playbook — the lessons that keep costing us hours
|
||||
|
||||
**Read this before starting any new quant.** Not the per-model runbooks — those are worked
|
||||
examples of a *specific* model at a *specific* point in time, and several carry claims that are
|
||||
now false (see §7). This file owns the **transferable** part: what recurs regardless of which
|
||||
model dropped this week.
|
||||
|
||||
Written 2026-08-15, after the fourth quant in five weeks re-discovered the third-known instance
|
||||
of the same loader-class bug. Scope: NVFP4 / FP8 / mixed-precision on the Blackwell boxes
|
||||
(ana-ml2), vLLM-served. Ampere (irv-ml1) has no native FP4/FP8 — see §6.
|
||||
|
||||
**Maintenance rule.** When a quant teaches you something *model-agnostic*, it lands here and the
|
||||
per-model README links up. When it's model-specific (this checkpoint's odd tensor names, this
|
||||
finetune's missing config), it stays in the per-model artifact. If you find yourself writing a
|
||||
"Gotchas" section that repeats §3, you are re-litigating — add the delta here instead.
|
||||
|
||||
---
|
||||
|
||||
## 1. The 60-second decision: which scheme
|
||||
|
||||
On Blackwell + vLLM, for a dense-or-hybrid VL model you intend to serve at long context:
|
||||
|
||||
| want | scheme | notes |
|
||||
|---|---|---|
|
||||
| **default, best speed/accuracy** | **mixed: NVFP4 W4A4 bulk MLPs + FP8 W8A8 attention/`lm_head`/last-8-layer MLPs** | the current answer. §2. |
|
||||
| max fidelity, don't care about prefill | NVFP4 **W4A16** (weight-only) | forces the **Marlin** kernel — ~half the prefill of native FP4 |
|
||||
| small model, VRAM is free | FP8 **W8A8** | safe and simple; 2× the weight bytes of 4-bit |
|
||||
| — | ~~"W4A8" = NVFP4 weights + FP8 activations~~ | **DOES NOT EXIST.** §3.1 |
|
||||
|
||||
**Measured on Qwen3.8-27B (2026-08-15), W4A16 → mixed:** decode +18%, prefill **+78–98%**,
|
||||
MTP acceptance unchanged, perplexity +1.7%, weights −19%.
|
||||
|
||||
Note the shape of that: **decode barely moves, prefill nearly doubles.** Decode at batch-1 is
|
||||
memory-bandwidth-bound and the weights are 4-bit under either scheme, so there is little to win;
|
||||
prefill is compute-bound, which is where native FP4 tensor cores replace the Marlin
|
||||
dequantize-to-BF16 path. If someone promises you a big *decode* win from a scheme change, be
|
||||
skeptical — and go measure §5 before believing it.
|
||||
|
||||
**The accuracy cost is real and is paid on purpose.** Operator ruling 2026-08-15: the ~1.7%
|
||||
perplexity is an acceptable price for the speed. Settled — don't re-litigate. For correct
|
||||
attribution: it is the **activation**-quantization cost (A4/A8 vs BF16 activations), *not* an MTP
|
||||
cost. Turning MTP off does not recover it; only reverting the quant does.
|
||||
|
||||
---
|
||||
|
||||
## 2. The reference recipe (mixed-precision)
|
||||
|
||||
Lifted from `unsloth/Qwen3.8-27B-NVFP4` and replicated in-house. **Prefer replicating a published
|
||||
recipe from a reputable quantizer over inventing one** — they have already paid for the
|
||||
sensitivity analysis.
|
||||
|
||||
| group | scheme | targets |
|
||||
|---|---|---|
|
||||
| `group_0` | FP8 W8A8 — channel weights (static) + per-token dynamic activations | `self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`, `lm_head`, **the last 8 layers' MLPs** |
|
||||
| `group_1` | NVFP4 W4A4 — `tensor_group` gsize 16, fp8 scales, `imatrix_mse` weights, `dynamic:"local"` activations | **all remaining** MLP `{gate,up,down}_proj` |
|
||||
| kv cache | FP8 static tensor | |
|
||||
| ignore | vision tower, `linear_attn.{norm,in_proj_a,in_proj_b}`, `re:^mtp.*` | |
|
||||
|
||||
Three things in there are load-bearing and easy to drop:
|
||||
|
||||
- **Late layers stay FP8.** Holding the last ~8 layers' MLPs (and `lm_head`) at 8-bit is the
|
||||
accuracy-preservation trick — late layers are the sensitive ones. Uniform W4A4 is what collapses.
|
||||
- **`imatrix_mse` on the W4A4 weights**, not `memoryless_minmax`. Importance-weighted; needs
|
||||
calibration data.
|
||||
- **Group targets must be non-overlapping.** Do not let `group_1`'s `.*mlp\..*` also match the
|
||||
late layers and rely on group precedence to sort it out. Enumerate the early layers explicitly
|
||||
(`re:.*layers\.([0-9]|[1-4][0-9]|5[0-5])\.mlp\.…`) and **prove it** with a dry run (§4.1).
|
||||
|
||||
**Toolchain:** `pip install llmcompressor` into stock `vllm/vllm-openai:latest` gives
|
||||
llmcompressor 0.13 + compressed-tensors 0.18 without disturbing torch/transformers.
|
||||
**Avoid nvidia-modelopt** — see §3.4.
|
||||
|
||||
---
|
||||
|
||||
## 3. The recurring landmines
|
||||
|
||||
Ordered by how much time each has cost. Every one of these has bitten more than once.
|
||||
|
||||
### 3.1 "W4A8" is not a servable shape
|
||||
|
||||
vLLM's compressed-tensors dispatcher (`compressed_tensors.py:704-713`) accepts NVFP4 weights with
|
||||
**exactly two** activation settings:
|
||||
|
||||
| `input_activations` | result |
|
||||
|---|---|
|
||||
| `None` | W4A16 — and it **forces the Marlin kernel** (`kernels/linear/__init__.py:881-883`) |
|
||||
| NVFP4 | W4A4, native |
|
||||
|
||||
Anything else — **FP8 included** — raises at load:
|
||||
|
||||
```
|
||||
ValueError: For NVFP4 weights, input quantization must also be NVFP4 format, None for NVFP4A16
|
||||
```
|
||||
|
||||
`CompressedTensorsW4A8Fp8` exists but is **INT4** weights (`W4A8_SUPPORTED_TYPES_MAP = {4: int4}`)
|
||||
gated on `_check_scheme_supported(90, match_exact=True)` — Hopper-exact, so on Blackwell (sm_120)
|
||||
it is closed twice over. **FP8 enters per-layer-group, never as activations on NVFP4 weights.**
|
||||
|
||||
*Cost: one queued task written against an impossible scheme.*
|
||||
|
||||
### 3.2 Wrong loader class → silent weight-load failure
|
||||
|
||||
**Rediscovered three times.** Load the model through the class vLLM actually serves — the
|
||||
`…ForConditionalGeneration` / `…ForImageTextToText` **wrapper**, never `AutoModelForCausalLM`.
|
||||
|
||||
`AutoModelForCausalLM` resolves a VL config to the text-only inner class and saves a **flat**
|
||||
config with `model.layers.*` keys. vLLM's weight mapper wants `model.language_model.*` (+
|
||||
`model.visual.*`). The mismatch does not error — **every layer silently fails to load** and you
|
||||
get `!!!!` gibberish, or an engine that rejects the checkpoint outright.
|
||||
|
||||
*Bit: heretic2 (gibberish), Dark-Scarlett (both vLLM and SGLang refused the checkpoint), and the
|
||||
2026-08 rounds.*
|
||||
|
||||
### 3.3 The MTP head — three separate ways to lose it
|
||||
|
||||
Speculative decoding is a large fraction of the seat's throughput. It fails **silently**: the
|
||||
model serves fine, just at 0% acceptance.
|
||||
|
||||
1. **The wrapper class does not instantiate `mtp.*`,** so the quant drops it. Post-quant you must
|
||||
graft the BF16 `model-mtp.safetensors` back and register its tensors in the output index.
|
||||
2. **`re:^mtp.*` must be in `quantization_config.ignore`** — else vLLM loads the grafted BF16 head
|
||||
as though quantized, it comes up **uninitialised**, and acceptance is 0%.
|
||||
3. **⭐ llm-compressor PRUNES `ignore` entries that matched no module at quant time.** Since the
|
||||
wrapper never loaded `mtp.*`, the entry matches nothing and is **silently deleted from the
|
||||
saved config — even though you put it in the recipe.** So it must be re-injected *after* the
|
||||
graft, and then **verified, not assumed.**
|
||||
|
||||
*Cost: three rounds. The verify step caught it live on the third.*
|
||||
|
||||
There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not propagate
|
||||
modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix
|
||||
(needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4.
|
||||
|
||||
### 3.4 Toolchain version deadlocks
|
||||
|
||||
Both directions have burned us, so the resolution is: **use llm-compressor / compressed-tensors,
|
||||
not nvidia-modelopt.**
|
||||
|
||||
- modelopt **0.45** ↔ transformers 5.12: `mtq.quantize` dies `TypeError: issubclass() arg 2 must
|
||||
be a class` (modelopt registers transformers' `FusedMoE`, a *function* in 5.x, as an nn class).
|
||||
- modelopt **0.43** doesn't fix it — it drags transformers back to 4.57, which cannot load
|
||||
`qwen3_5` at all.
|
||||
- modelopt's config API also trails the current model families by a version.
|
||||
|
||||
### 3.5 Vision tower and its configs
|
||||
|
||||
- Keep the **vision tower in `ignore`** (BF16). Only the LLM backbone gets quantized.
|
||||
- The wrapper-class save **drops `preprocessor_config.json`** (and the video one). Without it the
|
||||
seat crash-loops `Can't load image processor`. Restore from the source — and if the upstream repo
|
||||
omits it, **reconstruct it from `processor_config.json`'s `image_processor` sub-dict**.
|
||||
|
||||
### 3.6 Memory and device placement (large models)
|
||||
|
||||
- **`device_map=None`/`"cpu"`, never `"auto"`.** `auto` fills GPU0 and OOMs during un-fusing;
|
||||
constraining with `max_memory` then offloads to the *meta* device, which cannot be `.copy_()`d.
|
||||
CPU-resident keeps every tensor real; the sequential pipeline still onloads per-layer to GPU.
|
||||
- **Avoid mmap on `/tank`.** `safetensors.safe_open()` mmaps a whole shard; on ZFS a 50 GB shard
|
||||
ENOMEMs regardless of free RAM (MAP_SHARED never consults the commit limit). Read with plain
|
||||
`read()` + `load(bytes)`, one shard cached at a time.
|
||||
- **`vm.overcommit_memory=1`** on ana-ml2 (durable via `playbooks/ana-ml2-overcommit-memory.yaml`).
|
||||
|
||||
---
|
||||
|
||||
## 4. Pipeline shape
|
||||
|
||||
### 4.1 Prove the targets before spending GPU time
|
||||
|
||||
Enumerate module names from the safetensors index and check your regexes against them: **zero
|
||||
overlap between groups, and the union covers every layer you intended.** This is free, takes
|
||||
seconds, and catches a mis-scoped regex that would otherwise surface as a mystery quality
|
||||
regression hours later. Reference: `services/gen-seat-mixed-quant/validate_targets.py`.
|
||||
|
||||
### 4.2 Quantize
|
||||
|
||||
Calibration data matters for `imatrix_mse` + static activation observers. We use
|
||||
`/tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl` (512 chat samples, RP/GM-flavoured
|
||||
— appropriate for our seats). 256 samples @ 2048 tokens ≈ 20 min for a 27B on one Blackwell.
|
||||
|
||||
### 4.3 The mandatory post-steps
|
||||
|
||||
Never optional, always in this order, and the last one **verifies rather than assumes**:
|
||||
|
||||
1. Graft `model-mtp.safetensors` + register its tensors in the output index.
|
||||
2. Restore `preprocessor_config.json` / `processor_config.json` / `video_preprocessor_config.json`.
|
||||
3. **Re-inject `re:^mtp.*` into `quantization_config.ignore` and confirm it is there** (§3.3).
|
||||
|
||||
Reference implementation: `services/gen-seat-mixed-quant/post_quant.py`.
|
||||
|
||||
### 4.4 Test on a temp port, never on the live seat
|
||||
|
||||
Serve the candidate on an alt port with the live seat's **exact** flags, run the gate (§5), and
|
||||
only then flip `.env`. Keep the previous build on disk; rollback is one `.env` line.
|
||||
|
||||
---
|
||||
|
||||
## 5. The acceptance gate — and how measurement lies to you
|
||||
|
||||
Speed alone does not justify cutting over a shared seat. Gate on **all** of: decode tok/s, MTP
|
||||
acceptance, perplexity, a behavioural surface test, and — for an abliterated model — that the
|
||||
abliteration survived.
|
||||
|
||||
**Three ways the numbers have lied to us. All three produced confident, wrong results.**
|
||||
|
||||
1. **Prefix caching fakes both speed metrics.** A fixed prompt returns byte-identical timings run
|
||||
after run; you are measuring cache, not compute. Worse for prefill: a *seeded* nonce
|
||||
regenerates the previous run's prompts verbatim and reads **~41k tok/s of cache-hit instead of
|
||||
~5k of real prefill**. Use a fresh unseeded nonce per request; never seed a cache-buster.
|
||||
2. **`prompt_logprobs` are garbage while speculative decoding is on** — ~uniform over the vocab
|
||||
(median rank ~10⁵; " Paris" after "The capital of France is" ranked 69698). **Perplexity must be
|
||||
measured on a seat served without `--speculative-config`,** on both sides of the comparison.
|
||||
3. **A 0600 `.env` makes `docker compose` silently no-op.** Without `sudo` it fails
|
||||
`permission denied` reading `.env`, **leaves the old container running**, and reports success —
|
||||
producing a full page of "benchmark results" that were just the unchanged baseline.
|
||||
**Hard-verify the change landed against `docker inspect …Config.Cmd`.**
|
||||
|
||||
**Re-measure the baseline before believing a target.** The 2026-08-15 handoff quoted ~68 tok/s;
|
||||
cache-busted, the incumbent was already doing 80.1 — essentially the *target* of the work queued
|
||||
against it. Had that not been re-measured, doing nothing would have looked like a 20% win.
|
||||
|
||||
**Cheap shortcut worth taking first:** if a reputable published quant of the same architecture is
|
||||
already on-box (or is a small pull), **serve it as a probe and measure it** before committing
|
||||
hours to your own. It answers "is this gain even real?" in ten minutes *and* hands you the recipe.
|
||||
|
||||
Harness: `services/gen-seat-mixed-quant/bench/` — `quickbench.py` (decode + acceptance),
|
||||
`prefill_bench.py`, `eval_quality.py` (PPL + abliteration), `surface_test.py` (chat, vision, tools,
|
||||
thinking split, long-context needle, streaming), `serve_probe.sh`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Hardware and co-residency
|
||||
|
||||
- **ana-ml2 = Blackwell (sm_120)**, 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths
|
||||
(`match_exact=True` on sm90) are **closed** here — do not plan around them.
|
||||
- **irv-ml1 = Ampere (sm_86)**, 3090 + A6000. **No native FP8/FP4** — 4-bit there is a VRAM saving
|
||||
only, not a speed win. Don't port a Blackwell scheme over and expect the throughput.
|
||||
- **GPU co-residency is a zero-sum budget, and a *smaller* model can break its neighbour.**
|
||||
`gpu-memory-utilization` is a fraction of the *whole card*, so when new weights are smaller the
|
||||
seat absorbs the slack as extra KV rather than releasing it. That is exactly how a −5.2 GB
|
||||
requant left the co-resident seat **0.18 GiB** short and crash-looping. **After any requant,
|
||||
re-check both seats' budgets** and hand the space back explicitly.
|
||||
|
||||
---
|
||||
|
||||
## 7. Superseded claims — do not follow these
|
||||
|
||||
Old docs stay for their history, but these specific claims are **false now** and will cost you a
|
||||
day if followed:
|
||||
|
||||
| claim | where | status |
|
||||
|---|---|---|
|
||||
| "Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance" | `docs/runbooks/heretic2-nvfp4-mtp-seat.md` §landmine 2 | **SUPERSEDED 2026-08-14.** The 0% was the missing `re:^mtp.*` ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.7–83.2% acceptance, live. Use compressed-tensors. |
|
||||
| "Abliteration desyncs the MTP head → uncensored models can't do MTP" | earlier auto-memory | **SUPERSEDED 2026-08-14.** A modest abliteration preserves MTP (83.7% at bf16). Test MTP on **bf16 first** to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source. |
|
||||
| "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). |
|
||||
|
||||
---
|
||||
|
||||
## 8. Measured negatives — don't re-chase
|
||||
|
||||
- **`num_speculative_tokens` = 3 is optimal** on the Qwen3.8-27B seat. Swept: n=2 → 77.1,
|
||||
**n=3 → 80.1**, n=4 → 78.7, n=5 → 75.9 tok/s. Higher n trades acceptance for draft width and
|
||||
loses. Re-sweep only if the drafter architecture changes.
|
||||
- **Uniform W4A4** — see §7 row 3.
|
||||
- **Dense-VL as the anatomy judge** — A/B'd, MoE retained. Don't re-propose.
|
||||
|
||||
---
|
||||
|
||||
## 9. Worked examples
|
||||
|
||||
Per-model artifacts. Read for *how a specific model went*, not for the general lessons — those are
|
||||
above, and where the two disagree, **this file wins**.
|
||||
|
||||
| artifact | what it is |
|
||||
|---|---|
|
||||
| `services/gen-seat-mixed-quant/` | **current reference.** Mixed NVFP4+FP8 on Qwen3.8-27B-Uncensored: scripts, acceptance harness, raw measurements. |
|
||||
| `stacks/gen-seat/README.md` | the live `gen` seat (7 LiteLLM aliases) |
|
||||
| `stacks/meromero-charrp/README.md` | Gemma-4 seat — the **tool-call/reasoning-parser** trap (a parser default that returns null `content` for all prose) |
|
||||
| `services/heretic2-nvfp4-quant/` | modelopt-format MTP seat — historical; see §7 before following it |
|
||||
| `tools/mistral-small4-nvfp4/` | MoE + native-convert path; source of §3.6 |
|
||||
| `docs/pfi/recommended-model-settings.md` | serve-time sampler/flag defaults (not quant) |
|
||||
|
||||
**A new model just dropped and needs requanting?** §1 → §2 → §4 → §5. Skim §3 first; it is the
|
||||
part that costs hours.
|
||||
@@ -1,5 +1,16 @@
|
||||
# Heretic2 NVFP4 + MTP fast char-rp-reasoning seat — the working recipe
|
||||
|
||||
> ⚠️ **PARTIALLY SUPERSEDED (2026-08-15). Read [`docs/pfi/model-quantization-playbook.md`](../pfi/model-quantization-playbook.md) first.**
|
||||
>
|
||||
> Specifically, **landmine 2 below is now false.** "compressed-tensors can't load the BF16 MTP
|
||||
> head → 0% acceptance" was a real symptom with the wrong cause: the head was missing from
|
||||
> `quantization_config.ignore`, not failed by the format. compressed-tensors + `re:^mtp.*` in
|
||||
> ignore gives 47.7–83.2% acceptance in production. **Use compressed-tensors / llm-compressor;
|
||||
> do not start a new quant on modelopt** (see the playbook §3.4 and §7).
|
||||
>
|
||||
> The rest — the loader-class trap, the GPU window ritual, the acceptance-verification method —
|
||||
> still holds and is generalized in the playbook.
|
||||
|
||||
**Status: WORKING (2026-07-14).** ~77 tok/s single-stream (vs GGUF NEO-CODE ~59.5, base
|
||||
NVFP4 ~53) — **~1.3× over GGUF**, MTP draft-acceptance **32–40%**, mean acceptance length
|
||||
**2.19**. This is a drop-in faster replacement for the GGUF NEO-CODE `char-rp-reasoning`
|
||||
|
||||
Reference in New Issue
Block a user