vh
a91cc3fb38
Quants are hard-fought and we keep re-paying for the same lessons. A survey found quant knowledge scattered across 18 files in four trees, with three documents having independently discovered and recorded overlapping "landmines" sections — and one of them now actively misleading. Adds docs/pfi/model-quantization-playbook.md as the single home for the TRANSFERABLE lessons, with per-model artifacts demoted to worked examples that link up to it. Contents: - scheme decision table, incl. that a literal "W4A8" NVFP4 checkpoint is unservable on vLLM (two legal activation settings, FP8 is not one) - the reference mixed-precision recipe and the three parts of it that are load-bearing and easy to drop - the recurring landmines, ordered by cost: the loader-class trap (rediscovered THREE times), the three separate ways to lose the MTP head, toolchain deadlocks, vision configs, memory/device placement - pipeline shape: prove targets before spending GPU time; mandatory post-steps that verify rather than assume - the acceptance gate, and the three ways measurement has lied to us — prefix caching faking both speed metrics, prompt_logprobs going uniform under speculative decoding, and a 0600 .env making compose silently no-op - hardware/co-residency, including that a SMALLER model can starve its neighbour because gpu-memory-utilization is a fraction of the whole card - a superseded-claims table, and measured negatives not to re-chase The superseded table earns its place immediately: the heretic2 runbook tells readers to use modelopt because "compressed-tensors can't load the BF16 MTP head, 0% acceptance". That symptom was real but the cause was not the format -- it was the missing re:^mtp.* ignore entry. compressed-tensors gives 47.7-83.2% acceptance in production. A fresh session following that doc would be sent down the modelopt path that current memory calls dependency hell, so the runbook now carries a stale-warning header pointing here. Wires discovery: an orientation.md "Where to look for what" row, pointers from the gen-seat / heretic2 / mistral artifacts, and a CLAUDE.md maintenance rule so the playbook gets fed instead of going stale -- model-agnostic lessons land in the playbook, model-specific ones stay put, and a wrong claim earns a dated superseded row rather than a silent edit. Motivated by Qwen3.8 having just released: the next model swap will need a requant, and this is what that session should read first.
156 lines
9.4 KiB
Markdown
156 lines
9.4 KiB
Markdown
# Heretic2 NVFP4 + MTP fast char-rp-reasoning seat — the working recipe
|
||
|
||
> ⚠️ **PARTIALLY SUPERSEDED (2026-08-15). Read [`docs/pfi/model-quantization-playbook.md`](../pfi/model-quantization-playbook.md) first.**
|
||
>
|
||
> Specifically, **landmine 2 below is now false.** "compressed-tensors can't load the BF16 MTP
|
||
> head → 0% acceptance" was a real symptom with the wrong cause: the head was missing from
|
||
> `quantization_config.ignore`, not failed by the format. compressed-tensors + `re:^mtp.*` in
|
||
> ignore gives 47.7–83.2% acceptance in production. **Use compressed-tensors / llm-compressor;
|
||
> do not start a new quant on modelopt** (see the playbook §3.4 and §7).
|
||
>
|
||
> The rest — the loader-class trap, the GPU window ritual, the acceptance-verification method —
|
||
> still holds and is generalized in the playbook.
|
||
|
||
**Status: WORKING (2026-07-14).** ~77 tok/s single-stream (vs GGUF NEO-CODE ~59.5, base
|
||
NVFP4 ~53) — **~1.3× over GGUF**, MTP draft-acceptance **32–40%**, mean acceptance length
|
||
**2.19**. This is a drop-in faster replacement for the GGUF NEO-CODE `char-rp-reasoning`
|
||
seat (same Heretic2/NEO-CODE model, NVFP4 + native MTP spec-decode).
|
||
|
||
This runbook exists because getting here was a multi-hour fire drill. **Every gotcha below
|
||
cost real time — read them before touching this.** The TL;DR: three things all had to be
|
||
right at once — (1) quant as the *multimodal* class, (2) use the *modelopt* format not
|
||
compressed-tensors, (3) work around a vLLM bug that quantizes the MTP draft head.
|
||
|
||
---
|
||
|
||
## What / where
|
||
|
||
- **Model:** NEO-CODE = `DavidAU/Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking` (dense
|
||
27B, `Qwen3_5` GDN-hybrid arch, multimodal `Qwen3_5ForConditionalGeneration`).
|
||
- **Runs only on ana-ml2 GPU0** (NVFP4 is Blackwell-only; irv-ml1 is Ampere).
|
||
- **Artifacts** (ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/`, root-owned):
|
||
- `heretic2-mtp-bf16/` — BF16 graft (Heretic2 + 15 base-Qwen3.6 MTP tensors). [graft input]
|
||
- `heretic2-modelopt-nvfp4/` — modelopt NVFP4 quant, single shard, **no mtp**. [quant output]
|
||
- `heretic2-modelopt-nvfp4-mtp/` — the above + spliced 15 BF16 mtp → **the seat**. [SERVE THIS]
|
||
- (superseded: `heretic2-nvfp4-cg*` = compressed-tensors path, coherent but MTP-inert;
|
||
`heretic2-mtp-nvfp4-prod` = original gibberish. Keep for diff, do not serve.)
|
||
- **Scripts** (eshpfi `services/heretic2-nvfp4-quant/`): `graft_mtp.py`, `quant_modelopt.py`,
|
||
`finalize_modelopt_mtp.py`, `serve_modelopt_mtp.sh`, `sitecustomize-mtp-workaround.py`.
|
||
- **Reference:** the MoE `gen` (`qwen36-35b-a3b-heretic-nvfp4`, `quant_method: modelopt`) and
|
||
the qwopus-122B `gen` both ran MTP before (qwopus +12% single-stream, archival-memory
|
||
2026-07-01) — dropped for `gen` because MTP *hurts concurrency*, which is why it belongs on
|
||
the single-stream RP seats, not `gen`.
|
||
|
||
## GPU window ritual
|
||
|
||
Base NVFP4 quant needs ~55 GB free on GPU0. `docker stop llama-charrp
|
||
llama-charrp-reasoning vllm-aeon-gen` (→ ~97 GB free); restore with `docker start …`
|
||
(~90–230 s to healthy). The GGUF NEO-CODE seat is the always-restorable fallback. Heads-up
|
||
wt-dev (their character / thoughtful-character / gen route through these) — unless told
|
||
otherwise. `ssh ana-ml2` = lkraven, in the docker group (no sudo needed for docker).
|
||
|
||
---
|
||
|
||
## The pipeline (4 steps)
|
||
|
||
### 1. GRAFT (CPU, seats up) — `graft_mtp.py`
|
||
Heretic2's finetune dropped the MTP head; graft the 15 BF16 `mtp.*` tensors from base
|
||
`Qwen/Qwen3.6-27B` (shards 13+15). Symlinks Heretic2 shards + one `model-mtp.safetensors`.
|
||
Idempotent, refuses to clobber. Output: `heretic2-mtp-bf16/`.
|
||
|
||
### 2. QUANT (GPU0 window, ~18 min) — `quant_modelopt.py` via `run_quant_modelopt.sh`
|
||
`nvidia-modelopt` PTQ → **modelopt** NVFP4 format. Three things this script gets right (each a
|
||
gotcha — see below): loads as **`AutoModelForImageTextToText`**, patches the modelopt↔transformers
|
||
**FusedMoE** bug, and forces **single-shard** export. Excludes `lm_head` + `visual` + all
|
||
`linear_attn` (GDN) → BF16, matching AEON. Calib = the 512-row workload-matched chat mix.
|
||
```bash
|
||
docker run -d --name vllm-heretic2-modelopt-quant --gpus '"device=0"' --ipc host \
|
||
-v /tank/aimodels:/tank/aimodels -v /home/lkraven:/lk \
|
||
--entrypoint bash vllm/vllm-openai:v0.24.0 -c '
|
||
set -e
|
||
pip install -q nvidia-modelopt tiktoken sentencepiece 2>&1 | tail -1
|
||
python3 /lk/quant_modelopt.py \
|
||
--model /tank/aimodels/heretic2-nvfp4-work/heretic2-mtp-bf16 \
|
||
--calib-mode chat --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
|
||
--num-samples 512 --seqlen 8192 \
|
||
--out /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4'
|
||
```
|
||
CPU dry-run (no GPU, tiny calib) to validate the pipeline without an outage: same command
|
||
minus `--gpus`, add `-e CUDA_VISIBLE_DEVICES=""`, `--num-samples 2 --seqlen 512`.
|
||
|
||
### 3. SPLICE (CPU) — `finalize_modelopt_mtp.py`
|
||
Copy `heretic2-modelopt-nvfp4` → `heretic2-modelopt-nvfp4-mtp`, splice the 15 BF16 `mtp.*`
|
||
tensors into the single shard (→ 1967 tensors). (The transformers load never builds an mtp
|
||
module, so mtp must be spliced post-quant — same as AEON/pantheon.)
|
||
|
||
### 4. SERVE (GPU0) — `serve_modelopt_mtp.sh` + the MTP workaround
|
||
```bash
|
||
docker run -d --name vllm-charrp-modelopt --gpus '"device=0"' --ipc host \
|
||
-v /tank/aimodels:/tank/aimodels \
|
||
-v <sitecustomize dir>:/lk_debug -e PYTHONPATH=/lk_debug \ # ← the MTP workaround, see below
|
||
-p 8018:8000 vllm/vllm-openai:v0.24.0 \
|
||
/tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4-mtp \
|
||
--quantization modelopt \
|
||
--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \
|
||
--language-model-only --mamba-cache-dtype float32 \
|
||
--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \
|
||
--served-model-name char-rp-reasoning --max-model-len 40960 --max-num-seqs 32 \
|
||
--gpu-memory-utilization 0.5 --trust-remote-code
|
||
```
|
||
`--language-model-only` skips the vision tower (RP seat doesn't need it; saves ~1–2 GB — the
|
||
tower is preserved BF16 in the weights, so multimodal is recoverable by dropping the flag).
|
||
|
||
---
|
||
|
||
## The four landmines (each cost hours)
|
||
|
||
1. **Load as `AutoModelForImageTextToText`, NEVER `AutoModelForCausalLM`.** The latter resolves
|
||
`qwen3_5` → text-only `Qwen3_5ForCausalLM` → flat `model.layers.*` keys. vLLM only serves
|
||
`Qwen3_5ForConditionalGeneration`, whose weight mapper needs `model.language_model.*` (+
|
||
`model.visual.*`). Wrong class → every layer weight silently fails to load → **`!!!!` gibberish**.
|
||
|
||
2. **Use the MODELOPT format (nvidia-modelopt), not compressed-tensors (llm-compressor).** On
|
||
compressed-tensors the MTP drafter can't load the BF16 mtp head at all (`not found in
|
||
params_dict`, **0% acceptance** — loads but never accelerates; this is what pantheon and the
|
||
"AEON RP seat" actually were). Base NVFP4 *alone* ≈ GGUF at batch-1 (no single-stream win) —
|
||
**the MTP multiplier is the entire point**, and it needs modelopt.
|
||
|
||
3. **modelopt 0.45 ↔ transformers 5.12.1 FusedMoE crash.** `mtq.quantize` dies with
|
||
`TypeError: issubclass() arg 2 must be a class` — modelopt registered transformers' `FusedMoE`
|
||
(a *function* in 5.x) as an nn class. `quant_modelopt.py` guards it (patches
|
||
`_DMRegistryCls._get_registered_nn_class` to skip non-class registry entries). Do **not**
|
||
pin `nvidia-modelopt[hf]==0.43` to dodge it — that drags transformers back to 4.57 which can't
|
||
load `qwen3_5` at all.
|
||
|
||
4. **⭐ THE BIG ONE — vLLM 0.24.0 does not propagate modelopt `exclude_modules` to the
|
||
spec-decode DRAFT model.** The MTP drafter builds its own `qkv_proj`/`gate_up_proj` as
|
||
*quantized* (NVFP4-packed) while the mtp head is BF16 → `AssertionError: param_data.shape ==
|
||
loaded_weight.shape` in `qwen3_5_mtp.py:256`. **No checkpoint config fixes this** — instrumenting
|
||
`is_layer_skipped` proved the drafter's exclude list contains only the *main* model's
|
||
`linear_attn` entries, never the mtp ones. Also note `is_layer_skipped` does **exact string
|
||
membership, not glob** — so wildcards like `mtp.layers.0.*` never match anything. **Fix = a
|
||
runtime patch** (`sitecustomize-mtp-workaround.py`, mounted on `PYTHONPATH`) that force-skips
|
||
any `mtp.*` prefix in `is_layer_skipped`, keeping the drafter BF16. This is a genuine vLLM bug —
|
||
**report upstream** (draft-model quant-config should inherit the target's exclude_modules).
|
||
|
||
## Verify it's actually accelerating
|
||
|
||
```bash
|
||
# coherence
|
||
curl -s :8018/v1/completions -d '{"model":"char-rp-reasoning","prompt":"The old tavern","max_tokens":40,"temperature":0}'
|
||
# drive tokens, then read acceptance from the seat log:
|
||
docker logs vllm-charrp-modelopt 2>&1 | grep SpecDecoding | tail -2
|
||
# -> "Mean acceptance length: 2.19 ... Avg Draft acceptance rate: 39.7%" [GOOD: >0%, ~2 length]
|
||
# -> "Avg Draft acceptance rate: 0.0%" [BAD: compressed-tensors, or mtp quantized]
|
||
```
|
||
`SpecDecoding` line only appears during active generation. 0% acceptance = you're on
|
||
compressed-tensors, or the workaround didn't load (check for `[ISLS] ... workaround installed`).
|
||
|
||
## Productionization TODO (not yet done)
|
||
- Bake the sitecustomize workaround into a compose stack (mount + `PYTHONPATH`), served-name
|
||
`char-rp-reasoning`, alongside/replacing the GGUF seat.
|
||
- brokkr P00 (soong 9-tool k5) — same base model as GGUF NEO-CODE so R36 should carry, but the
|
||
NVFP4-vs-Q5 quality + tool-path must be confirmed before cutover.
|
||
- Repoint gateway `char-rp-reasoning` alias + heads-up wt-dev.
|
||
- File the vLLM upstream bug (draft-model exclude non-inheritance).
|