docs(quant): consolidate quantization lessons into a durable playbook
Quants are hard-fought and we keep re-paying for the same lessons. A survey found quant knowledge scattered across 18 files in four trees, with three documents having independently discovered and recorded overlapping "landmines" sections — and one of them now actively misleading. Adds docs/pfi/model-quantization-playbook.md as the single home for the TRANSFERABLE lessons, with per-model artifacts demoted to worked examples that link up to it. Contents: - scheme decision table, incl. that a literal "W4A8" NVFP4 checkpoint is unservable on vLLM (two legal activation settings, FP8 is not one) - the reference mixed-precision recipe and the three parts of it that are load-bearing and easy to drop - the recurring landmines, ordered by cost: the loader-class trap (rediscovered THREE times), the three separate ways to lose the MTP head, toolchain deadlocks, vision configs, memory/device placement - pipeline shape: prove targets before spending GPU time; mandatory post-steps that verify rather than assume - the acceptance gate, and the three ways measurement has lied to us — prefix caching faking both speed metrics, prompt_logprobs going uniform under speculative decoding, and a 0600 .env making compose silently no-op - hardware/co-residency, including that a SMALLER model can starve its neighbour because gpu-memory-utilization is a fraction of the whole card - a superseded-claims table, and measured negatives not to re-chase The superseded table earns its place immediately: the heretic2 runbook tells readers to use modelopt because "compressed-tensors can't load the BF16 MTP head, 0% acceptance". That symptom was real but the cause was not the format -- it was the missing re:^mtp.* ignore entry. compressed-tensors gives 47.7-83.2% acceptance in production. A fresh session following that doc would be sent down the modelopt path that current memory calls dependency hell, so the runbook now carries a stale-warning header pointing here. Wires discovery: an orientation.md "Where to look for what" row, pointers from the gen-seat / heretic2 / mistral artifacts, and a CLAUDE.md maintenance rule so the playbook gets fed instead of going stale -- model-agnostic lessons land in the playbook, model-specific ones stay put, and a wrong claim earns a dated superseded row rather than a silent edit. Motivated by Qwen3.8 having just released: the next model swap will need a requant, and this is what that session should read first.
This commit is contained in:
@@ -1,5 +1,16 @@
|
||||
# Heretic2 NVFP4 + MTP fast char-rp-reasoning seat — the working recipe
|
||||
|
||||
> ⚠️ **PARTIALLY SUPERSEDED (2026-08-15). Read [`docs/pfi/model-quantization-playbook.md`](../pfi/model-quantization-playbook.md) first.**
|
||||
>
|
||||
> Specifically, **landmine 2 below is now false.** "compressed-tensors can't load the BF16 MTP
|
||||
> head → 0% acceptance" was a real symptom with the wrong cause: the head was missing from
|
||||
> `quantization_config.ignore`, not failed by the format. compressed-tensors + `re:^mtp.*` in
|
||||
> ignore gives 47.7–83.2% acceptance in production. **Use compressed-tensors / llm-compressor;
|
||||
> do not start a new quant on modelopt** (see the playbook §3.4 and §7).
|
||||
>
|
||||
> The rest — the loader-class trap, the GPU window ritual, the acceptance-verification method —
|
||||
> still holds and is generalized in the playbook.
|
||||
|
||||
**Status: WORKING (2026-07-14).** ~77 tok/s single-stream (vs GGUF NEO-CODE ~59.5, base
|
||||
NVFP4 ~53) — **~1.3× over GGUF**, MTP draft-acceptance **32–40%**, mean acceptance length
|
||||
**2.19**. This is a drop-in faster replacement for the GGUF NEO-CODE `char-rp-reasoning`
|
||||
|
||||
Reference in New Issue
Block a user