vh
a91cc3fb38
Quants are hard-fought and we keep re-paying for the same lessons. A survey found quant knowledge scattered across 18 files in four trees, with three documents having independently discovered and recorded overlapping "landmines" sections — and one of them now actively misleading. Adds docs/pfi/model-quantization-playbook.md as the single home for the TRANSFERABLE lessons, with per-model artifacts demoted to worked examples that link up to it. Contents: - scheme decision table, incl. that a literal "W4A8" NVFP4 checkpoint is unservable on vLLM (two legal activation settings, FP8 is not one) - the reference mixed-precision recipe and the three parts of it that are load-bearing and easy to drop - the recurring landmines, ordered by cost: the loader-class trap (rediscovered THREE times), the three separate ways to lose the MTP head, toolchain deadlocks, vision configs, memory/device placement - pipeline shape: prove targets before spending GPU time; mandatory post-steps that verify rather than assume - the acceptance gate, and the three ways measurement has lied to us — prefix caching faking both speed metrics, prompt_logprobs going uniform under speculative decoding, and a 0600 .env making compose silently no-op - hardware/co-residency, including that a SMALLER model can starve its neighbour because gpu-memory-utilization is a fraction of the whole card - a superseded-claims table, and measured negatives not to re-chase The superseded table earns its place immediately: the heretic2 runbook tells readers to use modelopt because "compressed-tensors can't load the BF16 MTP head, 0% acceptance". That symptom was real but the cause was not the format -- it was the missing re:^mtp.* ignore entry. compressed-tensors gives 47.7-83.2% acceptance in production. A fresh session following that doc would be sent down the modelopt path that current memory calls dependency hell, so the runbook now carries a stale-warning header pointing here. Wires discovery: an orientation.md "Where to look for what" row, pointers from the gen-seat / heretic2 / mistral artifacts, and a CLAUDE.md maintenance rule so the playbook gets fed instead of going stale -- model-agnostic lessons land in the playbook, model-specific ones stay put, and a wrong claim earns a dated superseded row rather than a silent edit. Motivated by Qwen3.8 having just released: the next model swap will need a requant, and this is what that session should read first.
99 lines
5.0 KiB
Markdown
99 lines
5.0 KiB
Markdown
# Mistral Small 4 → NVFP4 (vision-intact) build tooling
|
|
|
|
> General quant lessons are consolidated in
|
|
> [`docs/pfi/model-quantization-playbook.md`](../../docs/pfi/model-quantization-playbook.md);
|
|
> the memory/device gotchas below are generalized there as §3.6.
|
|
|
|
Quantize a **HF-format** `Mistral3ForConditionalGeneration` checkpoint
|
|
(Mistral Small 4, 119B-total / 6.5B-active MoE) to **NVFP4** with the vision
|
|
tower intact, then convert it to **Mistral native format** so vLLM can serve it.
|
|
|
|
Built for the abliterated character-model successor
|
|
(`darkc0de/Mistral-Small-4-119B-2603-heretic`), validated end-to-end on ana-ml2
|
|
(2026-06-17). The official `mistralai/Mistral-Small-4-119B-2603-NVFP4` is the
|
|
naming reference the converter diffs against.
|
|
|
|
## Why both a quant *and* a convert step
|
|
|
|
vLLM serves Mistral Small 4 **only** through its native loader
|
|
(`--config-format mistral --load-format mistral --tokenizer-mode mistral`) —
|
|
there is no HF `Mistral4` serving path in any vLLM version. But `llm-compressor`
|
|
quantizes the **HF** checkpoint. So the pipeline is:
|
|
|
|
```
|
|
HF bf16 ──quant──▶ HF NVFP4 ──convert──▶ native NVFP4 ──serve──▶ vLLM
|
|
nvfp4_quant.py convert_hf_to_native.py (native loader)
|
|
```
|
|
|
|
## Pipeline (on ana-ml2, `/tank/aimodels/quant-work`, in a uv venv)
|
|
|
|
```bash
|
|
# 0. Pull the HF bf16 source (e.g. via huggingface-cli download).
|
|
|
|
# 1. Quantize HF bf16 -> HF NVFP4 (~65 GB out). GPU0 for compute, CPU-resident model.
|
|
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_VISIBLE_DEVICES=0 \
|
|
python nvfp4_quant.py <hf-bf16-dir> heretic-nvfp4 128
|
|
|
|
# 2. Dry-run the native convert (name-map check vs the official native reference).
|
|
python convert_hf_to_native.py --format nvfp4 \
|
|
--hf-dir heretic-nvfp4 \
|
|
--native-ref-dir <official native NVFP4 snapshot dir> \
|
|
--out-dir heretic-native-nvfp4 --dry-run
|
|
# Expect: unmapped=0, missing_from_output=0, extra_in_output=0.
|
|
|
|
# 3. Full native convert (~65 GB out, 5 shards).
|
|
python convert_hf_to_native.py --format nvfp4 \
|
|
--hf-dir heretic-nvfp4 --native-ref-dir <ref> \
|
|
--out-dir heretic-native-nvfp4 --max-shard-size-gb 15
|
|
|
|
# 4. Serve-test on vLLM (native loader, v0.22.0 = last vision-working pin).
|
|
docker run -d --name heretic-serve-test --ipc host --gpus '"device=0"' \
|
|
-p 8099:8000 -v $PWD/heretic-native-nvfp4:/model:ro \
|
|
vllm/vllm-openai:v0.22.0 /model --served-model-name heretic-test \
|
|
--host 0.0.0.0 --port 8000 \
|
|
--tokenizer-mode mistral --config-format mistral --load-format mistral \
|
|
--tensor-parallel-size 1 --gpu-memory-utilization 0.93 \
|
|
--max-model-len 16384 --attention-backend TRITON_MLA --max-num-seqs 8 --dtype auto
|
|
```
|
|
|
|
## Gotchas (each one cost a failed run)
|
|
|
|
- **`device_map="cpu"`, not `"auto"`** in the quant. `auto` fills GPU0 with the
|
|
205 GB model → OOM during MoE un-fusing; constraining with `max_memory` then
|
|
offloads experts to the *meta* device, which `copy_from_experts_module` can't
|
|
`.copy_()` (`Cannot copy out of meta tensor`). CPU-resident keeps every tensor
|
|
real; the sequential pipeline still onloads each layer to GPU0 for compute.
|
|
- **Non-mmap shard reads** in the converter. `safetensors.safe_open()` mmaps the
|
|
whole shard; on `/tank` (ZFS) a 50 GB shard mmap ENOMEMs regardless of free RAM
|
|
(MAP_SHARED never consults the commit limit). `read_tensor` reads with plain
|
|
`read()` + `safetensors.torch.load(bytes)`, caching one shard at a time — the
|
|
copy loop is sorted by shard so the cache doesn't thrash.
|
|
- **`vm.overcommit_memory=1`** on ana-ml2 (now durable — see
|
|
`playbooks/ana-ml2-overcommit-memory.yaml`). overcommit=0 + zero swap caps the
|
|
CommitLimit at ~RAM/2; the resident vLLM services eat the headroom and large
|
|
allocations fail despite free RAM.
|
|
- **NVFP4 output keeps the `model.` prefix.** llm-compressor's NVFP4 tensor names
|
|
are `model.language_model.model.layers.N...` (same prefix as bf16) — they are
|
|
*not* prefix-shifted. The only NVFP4 difference vs bf16 is per-expert-quantized
|
|
(`mlp.experts.E.{gate,up,down}_proj.{weight_packed,...}`) vs fused.
|
|
- **Vision tower stays bf16.** The IGNORE list excludes `vision_tower` +
|
|
`multi_modal_projector` (and all MLA attention, the MoE gate, embeddings,
|
|
lm_head) — only the expert FFN is NVFP4. So the vision encoder is byte-for-byte
|
|
full precision; any vision-quality nuance is the quantized LLM backbone, not the
|
|
tower.
|
|
|
|
## Serve-test results (2026-06-17, heretic native NVFP4)
|
|
|
|
- Loads on the vLLM native loader (v0.22.0), GPU0, ~91.9 GB at util 0.93.
|
|
- Text: correct (`2+2 = 4`, `capital of Japan = Tokyo`).
|
|
- Vision: tower functional — colors + spatial position accurate; exact shape
|
|
geometry fuzzy on small synthetic images (circle → pentagon). Evaluate real
|
|
vision quality during character tuning, against the official NVFP4 as baseline.
|
|
|
|
## Provenance
|
|
|
|
`convert_hf_to_native.py` originated with **worldtree-codex** (HF↔native bin
|
|
maps + the fused-expert split for the bf16 path). Fixed here: the NVFP4 layer
|
|
regexes (the `model.` prefix), and non-mmap shard reads. `nvfp4_quant.py` is
|
|
local. Homed in this infra repo per operator direction (not Worldtree).
|