esh-pfi-infrastructure/tools/mistral-small4-nvfp4

Mistral Small 4 → NVFP4 (vision-intact) build tooling

General quant lessons are consolidated in docs/pfi/model-quantization-playbook.md; the memory/device gotchas below are generalized there as §3.6.

Quantize a HF-format Mistral3ForConditionalGeneration checkpoint (Mistral Small 4, 119B-total / 6.5B-active MoE) to NVFP4 with the vision tower intact, then convert it to Mistral native format so vLLM can serve it.

Built for the abliterated character-model successor (darkc0de/Mistral-Small-4-119B-2603-heretic), validated end-to-end on ana-ml2 (2026-06-17). The official mistralai/Mistral-Small-4-119B-2603-NVFP4 is the naming reference the converter diffs against.

Why both a quant and a convert step

vLLM serves Mistral Small 4 only through its native loader (--config-format mistral --load-format mistral --tokenizer-mode mistral) — there is no HF Mistral4 serving path in any vLLM version. But llm-compressor quantizes the HF checkpoint. So the pipeline is:

HF bf16  ──quant──▶  HF NVFP4  ──convert──▶  native NVFP4  ──serve──▶  vLLM
        nvfp4_quant.py      convert_hf_to_native.py        (native loader)

Pipeline (on ana-ml2, /tank/aimodels/quant-work, in a uv venv)

# 0. Pull the HF bf16 source (e.g. via huggingface-cli download).

# 1. Quantize HF bf16 -> HF NVFP4  (~65 GB out). GPU0 for compute, CPU-resident model.
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_VISIBLE_DEVICES=0 \
  python nvfp4_quant.py <hf-bf16-dir> heretic-nvfp4 128

# 2. Dry-run the native convert (name-map check vs the official native reference).
python convert_hf_to_native.py --format nvfp4 \
  --hf-dir heretic-nvfp4 \
  --native-ref-dir <official native NVFP4 snapshot dir> \
  --out-dir heretic-native-nvfp4 --dry-run
#    Expect: unmapped=0, missing_from_output=0, extra_in_output=0.

# 3. Full native convert (~65 GB out, 5 shards).
python convert_hf_to_native.py --format nvfp4 \
  --hf-dir heretic-nvfp4 --native-ref-dir <ref> \
  --out-dir heretic-native-nvfp4 --max-shard-size-gb 15

# 4. Serve-test on vLLM (native loader, v0.22.0 = last vision-working pin).
docker run -d --name heretic-serve-test --ipc host --gpus '"device=0"' \
  -p 8099:8000 -v $PWD/heretic-native-nvfp4:/model:ro \
  vllm/vllm-openai:v0.22.0 /model --served-model-name heretic-test \
  --host 0.0.0.0 --port 8000 \
  --tokenizer-mode mistral --config-format mistral --load-format mistral \
  --tensor-parallel-size 1 --gpu-memory-utilization 0.93 \
  --max-model-len 16384 --attention-backend TRITON_MLA --max-num-seqs 8 --dtype auto

Gotchas (each one cost a failed run)

• device_map="cpu", not "auto" in the quant. auto fills GPU0 with the 205 GB model → OOM during MoE un-fusing; constraining with max_memory then offloads experts to the meta device, which copy_from_experts_module can't .copy_() (Cannot copy out of meta tensor). CPU-resident keeps every tensor real; the sequential pipeline still onloads each layer to GPU0 for compute.
• Non-mmap shard reads in the converter. safetensors.safe_open() mmaps the whole shard; on /tank (ZFS) a 50 GB shard mmap ENOMEMs regardless of free RAM (MAP_SHARED never consults the commit limit). read_tensor reads with plain read() + safetensors.torch.load(bytes), caching one shard at a time — the copy loop is sorted by shard so the cache doesn't thrash.
• vm.overcommit_memory=1 on ana-ml2 (now durable — see playbooks/ana-ml2-overcommit-memory.yaml). overcommit=0 + zero swap caps the CommitLimit at ~RAM/2; the resident vLLM services eat the headroom and large allocations fail despite free RAM.
• NVFP4 output keeps the model. prefix. llm-compressor's NVFP4 tensor names are model.language_model.model.layers.N... (same prefix as bf16) — they are not prefix-shifted. The only NVFP4 difference vs bf16 is per-expert-quantized (mlp.experts.E.{gate,up,down}_proj.{weight_packed,...}) vs fused.
• Vision tower stays bf16. The IGNORE list excludes vision_tower + multi_modal_projector (and all MLA attention, the MoE gate, embeddings, lm_head) — only the expert FFN is NVFP4. So the vision encoder is byte-for-byte full precision; any vision-quality nuance is the quantized LLM backbone, not the tower.

Serve-test results (2026-06-17, heretic native NVFP4)

• Loads on the vLLM native loader (v0.22.0), GPU0, ~91.9 GB at util 0.93.
• Text: correct (2+2 = 4, capital of Japan = Tokyo).
• Vision: tower functional — colors + spatial position accurate; exact shape geometry fuzzy on small synthetic images (circle → pentagon). Evaluate real vision quality during character tuning, against the official NVFP4 as baseline.

Provenance

convert_hf_to_native.py originated with worldtree-codex (HF↔native bin maps + the fused-expert split for the bf16 path). Fixed here: the NVFP4 layer regexes (the model. prefix), and non-mmap shard reads. nvfp4_quant.py is local. Homed in this infra repo per operator direction (not Worldtree).