diff --git a/CLAUDE.md b/CLAUDE.md index 6b935ec..9004604 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -46,6 +46,22 @@ user can tell at a glance the session is parked on background work, not stalled on them. Hooks have no way to enumerate the bg-task list externally, so this is on the assistant. +## Model quantization + +Quants are hard-fought and we have repeatedly re-litigated the same lessons. +**`docs/pfi/model-quantization-playbook.md` is the durable home for the +transferable ones** — scheme choice, the recurring landmines, the acceptance +gate and its measurement traps, and a superseded-claims table. Read it before +starting any quant; read it *instead of* the per-model runbooks for general +guidance (several of those carry claims that are now false, and say so). + +When a quant teaches something **model-agnostic**, it goes in the playbook and +the per-model README links up. When it's **model-specific**, it stays in the +per-model artifact. If you catch yourself writing a fresh "Gotchas" section that +repeats the playbook, you are re-litigating — record the delta in the playbook +instead. When a playbook claim turns out wrong, don't just fix it: add a dated +row to its superseded-claims table so old docs stop misleading people. + ## Purpose - Inventory of servers and their state diff --git a/docs/orientation.md b/docs/orientation.md index bfc1fa3..365794f 100644 --- a/docs/orientation.md +++ b/docs/orientation.md @@ -212,6 +212,9 @@ These caught us once; don't let them catch you twice. | What's currently open / in-flight? | `STATUS.md` | | What do I need to know that isn't in current code? | `MEMORY.md` + the `.md` files it links | | Why did we do X? | Check memory files + `STATUS.md` session milestones at the bottom | +| **I need to quantize / requant a model** | **`docs/pfi/model-quantization-playbook.md` — READ IT FIRST.** Consolidated hard-won lessons (scheme choice, the recurring landmines, the acceptance gate, superseded claims). Per-model runbooks are worked examples, not the general guide. | +| What sampler/serve settings for model X? | `docs/pfi/recommended-model-settings.md` | +| Which model is on which GPU seat? | `servers/ana-ml2/README.md` + `stacks//README.md` | ## Inventory + automation scripts diff --git a/docs/pfi/model-quantization-playbook.md b/docs/pfi/model-quantization-playbook.md new file mode 100644 index 0000000..89d0483 --- /dev/null +++ b/docs/pfi/model-quantization-playbook.md @@ -0,0 +1,282 @@ +# Model quantization playbook — the lessons that keep costing us hours + +**Read this before starting any new quant.** Not the per-model runbooks — those are worked +examples of a *specific* model at a *specific* point in time, and several carry claims that are +now false (see §7). This file owns the **transferable** part: what recurs regardless of which +model dropped this week. + +Written 2026-08-15, after the fourth quant in five weeks re-discovered the third-known instance +of the same loader-class bug. Scope: NVFP4 / FP8 / mixed-precision on the Blackwell boxes +(ana-ml2), vLLM-served. Ampere (irv-ml1) has no native FP4/FP8 — see §6. + +**Maintenance rule.** When a quant teaches you something *model-agnostic*, it lands here and the +per-model README links up. When it's model-specific (this checkpoint's odd tensor names, this +finetune's missing config), it stays in the per-model artifact. If you find yourself writing a +"Gotchas" section that repeats §3, you are re-litigating — add the delta here instead. + +--- + +## 1. The 60-second decision: which scheme + +On Blackwell + vLLM, for a dense-or-hybrid VL model you intend to serve at long context: + +| want | scheme | notes | +|---|---|---| +| **default, best speed/accuracy** | **mixed: NVFP4 W4A4 bulk MLPs + FP8 W8A8 attention/`lm_head`/last-8-layer MLPs** | the current answer. §2. | +| max fidelity, don't care about prefill | NVFP4 **W4A16** (weight-only) | forces the **Marlin** kernel — ~half the prefill of native FP4 | +| small model, VRAM is free | FP8 **W8A8** | safe and simple; 2× the weight bytes of 4-bit | +| — | ~~"W4A8" = NVFP4 weights + FP8 activations~~ | **DOES NOT EXIST.** §3.1 | + +**Measured on Qwen3.8-27B (2026-08-15), W4A16 → mixed:** decode +18%, prefill **+78–98%**, +MTP acceptance unchanged, perplexity +1.7%, weights −19%. + +Note the shape of that: **decode barely moves, prefill nearly doubles.** Decode at batch-1 is +memory-bandwidth-bound and the weights are 4-bit under either scheme, so there is little to win; +prefill is compute-bound, which is where native FP4 tensor cores replace the Marlin +dequantize-to-BF16 path. If someone promises you a big *decode* win from a scheme change, be +skeptical — and go measure §5 before believing it. + +**The accuracy cost is real and is paid on purpose.** Operator ruling 2026-08-15: the ~1.7% +perplexity is an acceptable price for the speed. Settled — don't re-litigate. For correct +attribution: it is the **activation**-quantization cost (A4/A8 vs BF16 activations), *not* an MTP +cost. Turning MTP off does not recover it; only reverting the quant does. + +--- + +## 2. The reference recipe (mixed-precision) + +Lifted from `unsloth/Qwen3.8-27B-NVFP4` and replicated in-house. **Prefer replicating a published +recipe from a reputable quantizer over inventing one** — they have already paid for the +sensitivity analysis. + +| group | scheme | targets | +|---|---|---| +| `group_0` | FP8 W8A8 — channel weights (static) + per-token dynamic activations | `self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`, `lm_head`, **the last 8 layers' MLPs** | +| `group_1` | NVFP4 W4A4 — `tensor_group` gsize 16, fp8 scales, `imatrix_mse` weights, `dynamic:"local"` activations | **all remaining** MLP `{gate,up,down}_proj` | +| kv cache | FP8 static tensor | | +| ignore | vision tower, `linear_attn.{norm,in_proj_a,in_proj_b}`, `re:^mtp.*` | | + +Three things in there are load-bearing and easy to drop: + +- **Late layers stay FP8.** Holding the last ~8 layers' MLPs (and `lm_head`) at 8-bit is the + accuracy-preservation trick — late layers are the sensitive ones. Uniform W4A4 is what collapses. +- **`imatrix_mse` on the W4A4 weights**, not `memoryless_minmax`. Importance-weighted; needs + calibration data. +- **Group targets must be non-overlapping.** Do not let `group_1`'s `.*mlp\..*` also match the + late layers and rely on group precedence to sort it out. Enumerate the early layers explicitly + (`re:.*layers\.([0-9]|[1-4][0-9]|5[0-5])\.mlp\.…`) and **prove it** with a dry run (§4.1). + +**Toolchain:** `pip install llmcompressor` into stock `vllm/vllm-openai:latest` gives +llmcompressor 0.13 + compressed-tensors 0.18 without disturbing torch/transformers. +**Avoid nvidia-modelopt** — see §3.4. + +--- + +## 3. The recurring landmines + +Ordered by how much time each has cost. Every one of these has bitten more than once. + +### 3.1 "W4A8" is not a servable shape + +vLLM's compressed-tensors dispatcher (`compressed_tensors.py:704-713`) accepts NVFP4 weights with +**exactly two** activation settings: + +| `input_activations` | result | +|---|---| +| `None` | W4A16 — and it **forces the Marlin kernel** (`kernels/linear/__init__.py:881-883`) | +| NVFP4 | W4A4, native | + +Anything else — **FP8 included** — raises at load: + +``` +ValueError: For NVFP4 weights, input quantization must also be NVFP4 format, None for NVFP4A16 +``` + +`CompressedTensorsW4A8Fp8` exists but is **INT4** weights (`W4A8_SUPPORTED_TYPES_MAP = {4: int4}`) +gated on `_check_scheme_supported(90, match_exact=True)` — Hopper-exact, so on Blackwell (sm_120) +it is closed twice over. **FP8 enters per-layer-group, never as activations on NVFP4 weights.** + +*Cost: one queued task written against an impossible scheme.* + +### 3.2 Wrong loader class → silent weight-load failure + +**Rediscovered three times.** Load the model through the class vLLM actually serves — the +`…ForConditionalGeneration` / `…ForImageTextToText` **wrapper**, never `AutoModelForCausalLM`. + +`AutoModelForCausalLM` resolves a VL config to the text-only inner class and saves a **flat** +config with `model.layers.*` keys. vLLM's weight mapper wants `model.language_model.*` (+ +`model.visual.*`). The mismatch does not error — **every layer silently fails to load** and you +get `!!!!` gibberish, or an engine that rejects the checkpoint outright. + +*Bit: heretic2 (gibberish), Dark-Scarlett (both vLLM and SGLang refused the checkpoint), and the +2026-08 rounds.* + +### 3.3 The MTP head — three separate ways to lose it + +Speculative decoding is a large fraction of the seat's throughput. It fails **silently**: the +model serves fine, just at 0% acceptance. + +1. **The wrapper class does not instantiate `mtp.*`,** so the quant drops it. Post-quant you must + graft the BF16 `model-mtp.safetensors` back and register its tensors in the output index. +2. **`re:^mtp.*` must be in `quantization_config.ignore`** — else vLLM loads the grafted BF16 head + as though quantized, it comes up **uninitialised**, and acceptance is 0%. +3. **⭐ llm-compressor PRUNES `ignore` entries that matched no module at quant time.** Since the + wrapper never loaded `mtp.*`, the entry matches nothing and is **silently deleted from the + saved config — even though you put it in the recipe.** So it must be re-injected *after* the + graft, and then **verified, not assumed.** + +*Cost: three rounds. The verify step caught it live on the third.* + +There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not propagate +modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix +(needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4. + +### 3.4 Toolchain version deadlocks + +Both directions have burned us, so the resolution is: **use llm-compressor / compressed-tensors, +not nvidia-modelopt.** + +- modelopt **0.45** ↔ transformers 5.12: `mtq.quantize` dies `TypeError: issubclass() arg 2 must + be a class` (modelopt registers transformers' `FusedMoE`, a *function* in 5.x, as an nn class). +- modelopt **0.43** doesn't fix it — it drags transformers back to 4.57, which cannot load + `qwen3_5` at all. +- modelopt's config API also trails the current model families by a version. + +### 3.5 Vision tower and its configs + +- Keep the **vision tower in `ignore`** (BF16). Only the LLM backbone gets quantized. +- The wrapper-class save **drops `preprocessor_config.json`** (and the video one). Without it the + seat crash-loops `Can't load image processor`. Restore from the source — and if the upstream repo + omits it, **reconstruct it from `processor_config.json`'s `image_processor` sub-dict**. + +### 3.6 Memory and device placement (large models) + +- **`device_map=None`/`"cpu"`, never `"auto"`.** `auto` fills GPU0 and OOMs during un-fusing; + constraining with `max_memory` then offloads to the *meta* device, which cannot be `.copy_()`d. + CPU-resident keeps every tensor real; the sequential pipeline still onloads per-layer to GPU. +- **Avoid mmap on `/tank`.** `safetensors.safe_open()` mmaps a whole shard; on ZFS a 50 GB shard + ENOMEMs regardless of free RAM (MAP_SHARED never consults the commit limit). Read with plain + `read()` + `load(bytes)`, one shard cached at a time. +- **`vm.overcommit_memory=1`** on ana-ml2 (durable via `playbooks/ana-ml2-overcommit-memory.yaml`). + +--- + +## 4. Pipeline shape + +### 4.1 Prove the targets before spending GPU time + +Enumerate module names from the safetensors index and check your regexes against them: **zero +overlap between groups, and the union covers every layer you intended.** This is free, takes +seconds, and catches a mis-scoped regex that would otherwise surface as a mystery quality +regression hours later. Reference: `services/gen-seat-mixed-quant/validate_targets.py`. + +### 4.2 Quantize + +Calibration data matters for `imatrix_mse` + static activation observers. We use +`/tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl` (512 chat samples, RP/GM-flavoured +— appropriate for our seats). 256 samples @ 2048 tokens ≈ 20 min for a 27B on one Blackwell. + +### 4.3 The mandatory post-steps + +Never optional, always in this order, and the last one **verifies rather than assumes**: + +1. Graft `model-mtp.safetensors` + register its tensors in the output index. +2. Restore `preprocessor_config.json` / `processor_config.json` / `video_preprocessor_config.json`. +3. **Re-inject `re:^mtp.*` into `quantization_config.ignore` and confirm it is there** (§3.3). + +Reference implementation: `services/gen-seat-mixed-quant/post_quant.py`. + +### 4.4 Test on a temp port, never on the live seat + +Serve the candidate on an alt port with the live seat's **exact** flags, run the gate (§5), and +only then flip `.env`. Keep the previous build on disk; rollback is one `.env` line. + +--- + +## 5. The acceptance gate — and how measurement lies to you + +Speed alone does not justify cutting over a shared seat. Gate on **all** of: decode tok/s, MTP +acceptance, perplexity, a behavioural surface test, and — for an abliterated model — that the +abliteration survived. + +**Three ways the numbers have lied to us. All three produced confident, wrong results.** + +1. **Prefix caching fakes both speed metrics.** A fixed prompt returns byte-identical timings run + after run; you are measuring cache, not compute. Worse for prefill: a *seeded* nonce + regenerates the previous run's prompts verbatim and reads **~41k tok/s of cache-hit instead of + ~5k of real prefill**. Use a fresh unseeded nonce per request; never seed a cache-buster. +2. **`prompt_logprobs` are garbage while speculative decoding is on** — ~uniform over the vocab + (median rank ~10⁵; " Paris" after "The capital of France is" ranked 69698). **Perplexity must be + measured on a seat served without `--speculative-config`,** on both sides of the comparison. +3. **A 0600 `.env` makes `docker compose` silently no-op.** Without `sudo` it fails + `permission denied` reading `.env`, **leaves the old container running**, and reports success — + producing a full page of "benchmark results" that were just the unchanged baseline. + **Hard-verify the change landed against `docker inspect …Config.Cmd`.** + +**Re-measure the baseline before believing a target.** The 2026-08-15 handoff quoted ~68 tok/s; +cache-busted, the incumbent was already doing 80.1 — essentially the *target* of the work queued +against it. Had that not been re-measured, doing nothing would have looked like a 20% win. + +**Cheap shortcut worth taking first:** if a reputable published quant of the same architecture is +already on-box (or is a small pull), **serve it as a probe and measure it** before committing +hours to your own. It answers "is this gain even real?" in ten minutes *and* hands you the recipe. + +Harness: `services/gen-seat-mixed-quant/bench/` — `quickbench.py` (decode + acceptance), +`prefill_bench.py`, `eval_quality.py` (PPL + abliteration), `surface_test.py` (chat, vision, tools, +thinking split, long-context needle, streaming), `serve_probe.sh`. + +--- + +## 6. Hardware and co-residency + +- **ana-ml2 = Blackwell (sm_120)**, 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths + (`match_exact=True` on sm90) are **closed** here — do not plan around them. +- **irv-ml1 = Ampere (sm_86)**, 3090 + A6000. **No native FP8/FP4** — 4-bit there is a VRAM saving + only, not a speed win. Don't port a Blackwell scheme over and expect the throughput. +- **GPU co-residency is a zero-sum budget, and a *smaller* model can break its neighbour.** + `gpu-memory-utilization` is a fraction of the *whole card*, so when new weights are smaller the + seat absorbs the slack as extra KV rather than releasing it. That is exactly how a −5.2 GB + requant left the co-resident seat **0.18 GiB** short and crash-looping. **After any requant, + re-check both seats' budgets** and hand the space back explicitly. + +--- + +## 7. Superseded claims — do not follow these + +Old docs stay for their history, but these specific claims are **false now** and will cost you a +day if followed: + +| claim | where | status | +|---|---|---| +| "Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance" | `docs/runbooks/heretic2-nvfp4-mtp-seat.md` §landmine 2 | **SUPERSEDED 2026-08-14.** The 0% was the missing `re:^mtp.*` ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.7–83.2% acceptance, live. Use compressed-tensors. | +| "Abliteration desyncs the MTP head → uncensored models can't do MTP" | earlier auto-memory | **SUPERSEDED 2026-08-14.** A modest abliteration preserves MTP (83.7% at bf16). Test MTP on **bf16 first** to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source. | +| "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). | + +--- + +## 8. Measured negatives — don't re-chase + +- **`num_speculative_tokens` = 3 is optimal** on the Qwen3.8-27B seat. Swept: n=2 → 77.1, + **n=3 → 80.1**, n=4 → 78.7, n=5 → 75.9 tok/s. Higher n trades acceptance for draft width and + loses. Re-sweep only if the drafter architecture changes. +- **Uniform W4A4** — see §7 row 3. +- **Dense-VL as the anatomy judge** — A/B'd, MoE retained. Don't re-propose. + +--- + +## 9. Worked examples + +Per-model artifacts. Read for *how a specific model went*, not for the general lessons — those are +above, and where the two disagree, **this file wins**. + +| artifact | what it is | +|---|---| +| `services/gen-seat-mixed-quant/` | **current reference.** Mixed NVFP4+FP8 on Qwen3.8-27B-Uncensored: scripts, acceptance harness, raw measurements. | +| `stacks/gen-seat/README.md` | the live `gen` seat (7 LiteLLM aliases) | +| `stacks/meromero-charrp/README.md` | Gemma-4 seat — the **tool-call/reasoning-parser** trap (a parser default that returns null `content` for all prose) | +| `services/heretic2-nvfp4-quant/` | modelopt-format MTP seat — historical; see §7 before following it | +| `tools/mistral-small4-nvfp4/` | MoE + native-convert path; source of §3.6 | +| `docs/pfi/recommended-model-settings.md` | serve-time sampler/flag defaults (not quant) | + +**A new model just dropped and needs requanting?** §1 → §2 → §4 → §5. Skim §3 first; it is the +part that costs hours. diff --git a/docs/runbooks/heretic2-nvfp4-mtp-seat.md b/docs/runbooks/heretic2-nvfp4-mtp-seat.md index 633a854..427da4f 100644 --- a/docs/runbooks/heretic2-nvfp4-mtp-seat.md +++ b/docs/runbooks/heretic2-nvfp4-mtp-seat.md @@ -1,5 +1,16 @@ # Heretic2 NVFP4 + MTP fast char-rp-reasoning seat — the working recipe +> ⚠️ **PARTIALLY SUPERSEDED (2026-08-15). Read [`docs/pfi/model-quantization-playbook.md`](../pfi/model-quantization-playbook.md) first.** +> +> Specifically, **landmine 2 below is now false.** "compressed-tensors can't load the BF16 MTP +> head → 0% acceptance" was a real symptom with the wrong cause: the head was missing from +> `quantization_config.ignore`, not failed by the format. compressed-tensors + `re:^mtp.*` in +> ignore gives 47.7–83.2% acceptance in production. **Use compressed-tensors / llm-compressor; +> do not start a new quant on modelopt** (see the playbook §3.4 and §7). +> +> The rest — the loader-class trap, the GPU window ritual, the acceptance-verification method — +> still holds and is generalized in the playbook. + **Status: WORKING (2026-07-14).** ~77 tok/s single-stream (vs GGUF NEO-CODE ~59.5, base NVFP4 ~53) — **~1.3× over GGUF**, MTP draft-acceptance **32–40%**, mean acceptance length **2.19**. This is a drop-in faster replacement for the GGUF NEO-CODE `char-rp-reasoning` diff --git a/services/gen-seat-mixed-quant/README.md b/services/gen-seat-mixed-quant/README.md index adbb015..9d0670d 100644 --- a/services/gen-seat-mixed-quant/README.md +++ b/services/gen-seat-mixed-quant/README.md @@ -4,6 +4,10 @@ The pipeline that produced `/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed`, t current fleet `gen` seat on ana-ml2 GPU0 `:8015`. **+18% decode** over the previous weight-only NVFP4A16 build, at equal MTP acceptance and +1.7% perplexity. +> **General lessons live in [`docs/pfi/model-quantization-playbook.md`](../../docs/pfi/model-quantization-playbook.md)** — +> read that before starting a quant on a *different* model. This file is the worked example for +> Qwen3.8-27B-Uncensored specifically. Where the two disagree, the playbook wins. + ## The headline correction: "W4A8" is not a thing you can serve The queued task was "re-quant to NVFP4 weights + FP8 activations (W4A8) for ~20% diff --git a/tools/mistral-small4-nvfp4/README.md b/tools/mistral-small4-nvfp4/README.md index c9884c0..fd2c25a 100644 --- a/tools/mistral-small4-nvfp4/README.md +++ b/tools/mistral-small4-nvfp4/README.md @@ -1,5 +1,9 @@ # Mistral Small 4 → NVFP4 (vision-intact) build tooling +> General quant lessons are consolidated in +> [`docs/pfi/model-quantization-playbook.md`](../../docs/pfi/model-quantization-playbook.md); +> the memory/device gotchas below are generalized there as §3.6. + Quantize a **HF-format** `Mistral3ForConditionalGeneration` checkpoint (Mistral Small 4, 119B-total / 6.5B-active MoE) to **NVFP4** with the vision tower intact, then convert it to **Mistral native format** so vLLM can serve it.