vh
a91cc3fb38
Quants are hard-fought and we keep re-paying for the same lessons. A survey found quant knowledge scattered across 18 files in four trees, with three documents having independently discovered and recorded overlapping "landmines" sections — and one of them now actively misleading. Adds docs/pfi/model-quantization-playbook.md as the single home for the TRANSFERABLE lessons, with per-model artifacts demoted to worked examples that link up to it. Contents: - scheme decision table, incl. that a literal "W4A8" NVFP4 checkpoint is unservable on vLLM (two legal activation settings, FP8 is not one) - the reference mixed-precision recipe and the three parts of it that are load-bearing and easy to drop - the recurring landmines, ordered by cost: the loader-class trap (rediscovered THREE times), the three separate ways to lose the MTP head, toolchain deadlocks, vision configs, memory/device placement - pipeline shape: prove targets before spending GPU time; mandatory post-steps that verify rather than assume - the acceptance gate, and the three ways measurement has lied to us — prefix caching faking both speed metrics, prompt_logprobs going uniform under speculative decoding, and a 0600 .env making compose silently no-op - hardware/co-residency, including that a SMALLER model can starve its neighbour because gpu-memory-utilization is a fraction of the whole card - a superseded-claims table, and measured negatives not to re-chase The superseded table earns its place immediately: the heretic2 runbook tells readers to use modelopt because "compressed-tensors can't load the BF16 MTP head, 0% acceptance". That symptom was real but the cause was not the format -- it was the missing re:^mtp.* ignore entry. compressed-tensors gives 47.7-83.2% acceptance in production. A fresh session following that doc would be sent down the modelopt path that current memory calls dependency hell, so the runbook now carries a stale-warning header pointing here. Wires discovery: an orientation.md "Where to look for what" row, pointers from the gen-seat / heretic2 / mistral artifacts, and a CLAUDE.md maintenance rule so the playbook gets fed instead of going stale -- model-agnostic lessons land in the playbook, model-specific ones stay put, and a wrong claim earns a dated superseded row rather than a silent edit. Motivated by Qwen3.8 having just released: the next model swap will need a requant, and this is what that session should read first.
152 lines
7.5 KiB
Markdown
152 lines
7.5 KiB
Markdown
# gen-seat mixed-precision quant — NVFP4 W4A4 MLP + FP8 W8A8 attention
|
||
|
||
The pipeline that produced `/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed`, the
|
||
current fleet `gen` seat on ana-ml2 GPU0 `:8015`. **+18% decode** over the previous
|
||
weight-only NVFP4A16 build, at equal MTP acceptance and +1.7% perplexity.
|
||
|
||
> **General lessons live in [`docs/pfi/model-quantization-playbook.md`](../../docs/pfi/model-quantization-playbook.md)** —
|
||
> read that before starting a quant on a *different* model. This file is the worked example for
|
||
> Qwen3.8-27B-Uncensored specifically. Where the two disagree, the playbook wins.
|
||
|
||
## The headline correction: "W4A8" is not a thing you can serve
|
||
|
||
The queued task was "re-quant to NVFP4 weights + FP8 activations (W4A8) for ~20%
|
||
more decode". **That checkpoint cannot load.** vLLM 0.24's compressed-tensors
|
||
dispatcher (`compressed_tensors.py:704-713`) allows NVFP4 weights with exactly two
|
||
activation options:
|
||
|
||
| input_activations | scheme | kernel |
|
||
|---|---|---|
|
||
| `None` | W4A16 | **Marlin** (forced — `kernels/linear/__init__.py:881-883`) |
|
||
| NVFP4 | W4A4 | native Blackwell FP4 |
|
||
|
||
anything else — FP8 included — raises
|
||
|
||
```
|
||
ValueError: For NVFP4 weights, input quantization must also be NVFP4 format,
|
||
None for NVFP4A16
|
||
```
|
||
|
||
`CompressedTensorsW4A8Fp8` exists but is **INT4** weights (`W4A8_SUPPORTED_TYPES_MAP
|
||
= {4: int4}`) gated on `_check_scheme_supported(90, match_exact=True)` — Hopper only.
|
||
ana-ml2 is Blackwell (sm_120), so that path is doubly closed.
|
||
|
||
The ~20% intuition was right; the scheme name was wrong. The servable way to get FP8
|
||
into the mix is **per-layer-group**, which is exactly what `unsloth/Qwen3.8-27B-NVFP4`
|
||
does — and that build, measured on-box, ran +19.1% faster than ours at identical MTP
|
||
acceptance. This pipeline replicates its recipe on the abliterated weights.
|
||
|
||
## The recipe
|
||
|
||
| group | scheme | targets |
|
||
|---|---|---|
|
||
| `group_0` | **FP8 W8A8** — channel weights (static), per-token dynamic activations | `self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`, `lm_head`, **layers 56-63** MLPs |
|
||
| `group_1` | **NVFP4 W4A4** — tensor_group gsize16, fp8 scales, `imatrix_mse` weights, `dynamic:"local"` activations | **layers 0-55** MLP `{gate,up,down}_proj` |
|
||
| kv cache | FP8 static tensor | — |
|
||
| ignored | vision tower, `linear_attn.{norm,in_proj_a,in_proj_b}`, `re:^mtp.*` | — |
|
||
|
||
Keeping the **last 8 layers' MLPs at FP8** is the accuracy-preservation trick — late
|
||
layers are the sensitive ones. `conv1d` in `linear_attn` is not a Linear and stays BF16
|
||
in both our build and unsloth's.
|
||
|
||
Targets are deliberately **non-overlapping** (`group_1` enumerates layers 0-55 rather
|
||
than matching all MLPs) instead of relying on group precedence to resolve the 56-63
|
||
collision. `validate_targets.py` proves this against the real module names before any
|
||
GPU time is spent — run it first.
|
||
|
||
## Running it
|
||
|
||
```bash
|
||
# 0. prove the regexes hit what you think (free, no GPU)
|
||
python3 validate_targets.py /tank/aimodels/qwen38-27b-uncensored-bf16
|
||
# -> expect OVERLAP 0, MLP layer union covers 0-63 True
|
||
|
||
# 1. quant (~20 min on one Blackwell; needs llmcompressor, NOT modelopt)
|
||
python3 quant_mixed_nvfp4.py \
|
||
--model /tank/aimodels/qwen38-27b-uncensored-bf16 \
|
||
--calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
|
||
--out /tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed \
|
||
--num-samples 256 --seqlen 2048
|
||
|
||
# 2. MANDATORY post-steps — graft MTP, restore preprocessor, repair the mtp ignore
|
||
python3 post_quant.py <bf16-source> <out-dir>
|
||
```
|
||
|
||
`pip install llmcompressor` into the stock `vllm/vllm-openai:latest` image gives
|
||
llmcompressor 0.13.0 + compressed-tensors 0.18.0 without disturbing torch or
|
||
transformers. **Do not use modelopt 0.43** — dependency hell on `qwen3_5`.
|
||
|
||
## The foot-gun that has now cost three rounds
|
||
|
||
`llm-compressor` **prunes `ignore` entries that matched no module at quant time.**
|
||
The wrapper class (`Qwen3_5ForConditionalGeneration`) never loads the MTP head, so
|
||
`re:^mtp.*` matches nothing and is silently dropped from the saved config. vLLM then
|
||
treats the freshly grafted BF16 MTP head as quantized, brings it up **uninitialised,
|
||
and speculative decoding runs at 0% acceptance.**
|
||
|
||
`post_quant.py` re-injects the entry *after* the graft and re-verifies. It is not
|
||
optional, and it verifies rather than assumes — that check fired on this very run.
|
||
|
||
## Acceptance gate (`bench/`)
|
||
|
||
Speed alone does not justify cutting over a seat backing 7 LiteLLM aliases.
|
||
|
||
| metric | W4A16 (old) | mixed (new) | delta |
|
||
|---|---|---|---|
|
||
| decode tok/s, bs=1, cache-busted | 80.12 | **94.53** | **+18.0%** |
|
||
| **prefill tok/s, ~6.7k prompt** | 3,206 | **6,334** | **+98%** |
|
||
| **prefill tok/s, ~27k prompt** | 2,862 | **5,085** | **+78%** |
|
||
| TTFT on a ~27k-token doc | 9.43 s | **5.31 s** | −44% |
|
||
| MTP acceptance | 47.8% | 47.7% | unchanged |
|
||
| perplexity, 6 held-out passages | 6.941 | 7.059 | +1.7% worse |
|
||
| abliteration compliance | 4/4 | 4/4 | preserved |
|
||
| weights on disk | 27.7 GB | 22.5 GB | −19% |
|
||
|
||
**Prefill roughly doubled** — the bigger practical win, and exactly what theory predicts:
|
||
decode at bs=1 is memory-bandwidth-bound (weights are 4-bit either way, so little changes),
|
||
while prefill is compute-bound and is where Blackwell's native FP4 tensor cores replace the
|
||
Marlin dequant-to-BF16 path. This is what the `summarizer` / `summarizer-large` aliases feel
|
||
on long documents.
|
||
|
||
- `quickbench.py` — cache-busted bs=1 decode + MTP acceptance. **Bust the cache:** with
|
||
a fixed prompt, prefix caching returns byte-identical timings and you measure nothing.
|
||
- `prefill_bench.py` — TTFT on long prompts. Same trap, worse: a *seeded* nonce reproduces
|
||
the previous run's prompts verbatim, so prefix caching serves them and you read ~41k tok/s
|
||
of cache-hit instead of ~5k of real prefill. Uses `SystemRandom`; never seed it.
|
||
- `eval_quality.py` — perplexity, deterministic generations, abliteration survival.
|
||
**PPL must be measured with `--speculative-config` OFF**: under MTP, vLLM's
|
||
`prompt_logprobs` come back ~uniform over the vocab (median rank ~10^5, logprob
|
||
≈ log(1/vocab)). The harness raises rather than reporting the garbage.
|
||
- `surface_test.py` — the real gate: plain chat, vision, tool calling, thinking split,
|
||
36K-token needle retrieval, streaming. All six must pass before a cutover.
|
||
- `serve_probe.sh <model-dir> [nospec]` — serve a candidate on `:8017` without touching
|
||
the live seat.
|
||
|
||
## GPU0 budget
|
||
|
||
The mixed build's weights are 5.2 GB smaller. At the old `GEN_GPU_MEM_UTIL=0.45` the
|
||
seat absorbed that slack as extra KV (17.0 GiB / 477K tokens) and left `meromero-charrp`
|
||
**0.18 GiB short** of its 0.52 budget — it crash-looped on startup. Fixed by handing the
|
||
space back: `GEN_GPU_MEM_UTIL=0.43` → 15.1 GiB / 422K tokens, still 1.6× the 262K
|
||
context. Both seats co-resident at **89.8 / 97.9 GB**.
|
||
|
||
## Levers already measured — do not re-chase
|
||
|
||
`GEN_SPEC_TOKENS` swept on this seat: n=2 → 77.1, **n=3 → 80.1**, n=4 → 78.7,
|
||
n=5 → 75.9 tok/s. Three is the optimum; higher n trades acceptance for draft width and
|
||
loses.
|
||
|
||
## Rollback
|
||
|
||
The previous build is untouched at `/tank/aimodels/qwen38-27b-uncensored-nvfp4`.
|
||
|
||
```bash
|
||
ssh infra-ops@10.250.50.54
|
||
sudo cp /opt/docker/compose/gen-seat/.env.bak-w4a16-20260815 /opt/docker/compose/gen-seat/.env
|
||
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen
|
||
```
|
||
|
||
⚠ `gen-seat/.env` is mode 0600 and lkraven-owned — **every** `docker compose` call
|
||
against it needs `sudo`, or compose fails with `permission denied` reading `.env`,
|
||
leaves the old container running, and the change silently does not take.
|