vh
dd3a5c93fd
Quantize a HF-format Mistral Small 4 (Mistral3ForConditionalGeneration MoE) to NVFP4 with the vision tower intact, then convert HF NVFP4 -> Mistral native so vLLM can serve it (there is no HF Mistral4 serving path in any vLLM version). Built + validated end-to-end on ana-ml2 for the abliterated character-model successor (darkc0de/Mistral-Small-4-119B-2603-heretic): quant -> dry-run (clean vs the official native NVFP4 reference) -> convert -> serve-test (loads on the native loader, correct text, vision functional). Converter scaffold came from worldtree-codex (bf16 bin maps + fused-expert split); fixed here: NVFP4 layer regexes (keep the `model.` prefix) + non-mmap shard reads (ZFS large-mmap ENOMEM). nvfp4_quant.py is local. README documents the pipeline + every gotcha that cost a failed run. Homed here per operator direction (not Worldtree).
95 lines
4.8 KiB
Markdown
95 lines
4.8 KiB
Markdown
# Mistral Small 4 → NVFP4 (vision-intact) build tooling
|
|
|
|
Quantize a **HF-format** `Mistral3ForConditionalGeneration` checkpoint
|
|
(Mistral Small 4, 119B-total / 6.5B-active MoE) to **NVFP4** with the vision
|
|
tower intact, then convert it to **Mistral native format** so vLLM can serve it.
|
|
|
|
Built for the abliterated character-model successor
|
|
(`darkc0de/Mistral-Small-4-119B-2603-heretic`), validated end-to-end on ana-ml2
|
|
(2026-06-17). The official `mistralai/Mistral-Small-4-119B-2603-NVFP4` is the
|
|
naming reference the converter diffs against.
|
|
|
|
## Why both a quant *and* a convert step
|
|
|
|
vLLM serves Mistral Small 4 **only** through its native loader
|
|
(`--config-format mistral --load-format mistral --tokenizer-mode mistral`) —
|
|
there is no HF `Mistral4` serving path in any vLLM version. But `llm-compressor`
|
|
quantizes the **HF** checkpoint. So the pipeline is:
|
|
|
|
```
|
|
HF bf16 ──quant──▶ HF NVFP4 ──convert──▶ native NVFP4 ──serve──▶ vLLM
|
|
nvfp4_quant.py convert_hf_to_native.py (native loader)
|
|
```
|
|
|
|
## Pipeline (on ana-ml2, `/tank/aimodels/quant-work`, in a uv venv)
|
|
|
|
```bash
|
|
# 0. Pull the HF bf16 source (e.g. via huggingface-cli download).
|
|
|
|
# 1. Quantize HF bf16 -> HF NVFP4 (~65 GB out). GPU0 for compute, CPU-resident model.
|
|
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_VISIBLE_DEVICES=0 \
|
|
python nvfp4_quant.py <hf-bf16-dir> heretic-nvfp4 128
|
|
|
|
# 2. Dry-run the native convert (name-map check vs the official native reference).
|
|
python convert_hf_to_native.py --format nvfp4 \
|
|
--hf-dir heretic-nvfp4 \
|
|
--native-ref-dir <official native NVFP4 snapshot dir> \
|
|
--out-dir heretic-native-nvfp4 --dry-run
|
|
# Expect: unmapped=0, missing_from_output=0, extra_in_output=0.
|
|
|
|
# 3. Full native convert (~65 GB out, 5 shards).
|
|
python convert_hf_to_native.py --format nvfp4 \
|
|
--hf-dir heretic-nvfp4 --native-ref-dir <ref> \
|
|
--out-dir heretic-native-nvfp4 --max-shard-size-gb 15
|
|
|
|
# 4. Serve-test on vLLM (native loader, v0.22.0 = last vision-working pin).
|
|
docker run -d --name heretic-serve-test --ipc host --gpus '"device=0"' \
|
|
-p 8099:8000 -v $PWD/heretic-native-nvfp4:/model:ro \
|
|
vllm/vllm-openai:v0.22.0 /model --served-model-name heretic-test \
|
|
--host 0.0.0.0 --port 8000 \
|
|
--tokenizer-mode mistral --config-format mistral --load-format mistral \
|
|
--tensor-parallel-size 1 --gpu-memory-utilization 0.93 \
|
|
--max-model-len 16384 --attention-backend TRITON_MLA --max-num-seqs 8 --dtype auto
|
|
```
|
|
|
|
## Gotchas (each one cost a failed run)
|
|
|
|
- **`device_map="cpu"`, not `"auto"`** in the quant. `auto` fills GPU0 with the
|
|
205 GB model → OOM during MoE un-fusing; constraining with `max_memory` then
|
|
offloads experts to the *meta* device, which `copy_from_experts_module` can't
|
|
`.copy_()` (`Cannot copy out of meta tensor`). CPU-resident keeps every tensor
|
|
real; the sequential pipeline still onloads each layer to GPU0 for compute.
|
|
- **Non-mmap shard reads** in the converter. `safetensors.safe_open()` mmaps the
|
|
whole shard; on `/tank` (ZFS) a 50 GB shard mmap ENOMEMs regardless of free RAM
|
|
(MAP_SHARED never consults the commit limit). `read_tensor` reads with plain
|
|
`read()` + `safetensors.torch.load(bytes)`, caching one shard at a time — the
|
|
copy loop is sorted by shard so the cache doesn't thrash.
|
|
- **`vm.overcommit_memory=1`** on ana-ml2 (now durable — see
|
|
`playbooks/ana-ml2-overcommit-memory.yaml`). overcommit=0 + zero swap caps the
|
|
CommitLimit at ~RAM/2; the resident vLLM services eat the headroom and large
|
|
allocations fail despite free RAM.
|
|
- **NVFP4 output keeps the `model.` prefix.** llm-compressor's NVFP4 tensor names
|
|
are `model.language_model.model.layers.N...` (same prefix as bf16) — they are
|
|
*not* prefix-shifted. The only NVFP4 difference vs bf16 is per-expert-quantized
|
|
(`mlp.experts.E.{gate,up,down}_proj.{weight_packed,...}`) vs fused.
|
|
- **Vision tower stays bf16.** The IGNORE list excludes `vision_tower` +
|
|
`multi_modal_projector` (and all MLA attention, the MoE gate, embeddings,
|
|
lm_head) — only the expert FFN is NVFP4. So the vision encoder is byte-for-byte
|
|
full precision; any vision-quality nuance is the quantized LLM backbone, not the
|
|
tower.
|
|
|
|
## Serve-test results (2026-06-17, heretic native NVFP4)
|
|
|
|
- Loads on the vLLM native loader (v0.22.0), GPU0, ~91.9 GB at util 0.93.
|
|
- Text: correct (`2+2 = 4`, `capital of Japan = Tokyo`).
|
|
- Vision: tower functional — colors + spatial position accurate; exact shape
|
|
geometry fuzzy on small synthetic images (circle → pentagon). Evaluate real
|
|
vision quality during character tuning, against the official NVFP4 as baseline.
|
|
|
|
## Provenance
|
|
|
|
`convert_hf_to_native.py` originated with **worldtree-codex** (HF↔native bin
|
|
maps + the fused-expert split for the bf16 path). Fixed here: the NVFP4 layer
|
|
regexes (the `model.` prefix), and non-mmap shard reads. `nvfp4_quant.py` is
|
|
local. Homed in this infra repo per operator direction (not Worldtree).
|