# Mistral Small 4 → NVFP4 (vision-intact) build tooling

> General quant lessons are consolidated in
> [`docs/pfi/model-quantization-playbook.md`](../../docs/pfi/model-quantization-playbook.md);
> the memory/device gotchas below are generalized there as §3.6.

Quantize a **HF-format** `Mistral3ForConditionalGeneration` checkpoint
(Mistral Small 4, 119B-total / 6.5B-active MoE) to **NVFP4** with the vision
tower intact, then convert it to **Mistral native format** so vLLM can serve it.

Built for the abliterated character-model successor
(`darkc0de/Mistral-Small-4-119B-2603-heretic`), validated end-to-end on ana-ml2
(2026-06-17). The official `mistralai/Mistral-Small-4-119B-2603-NVFP4` is the
naming reference the converter diffs against.

## Why both a quant *and* a convert step

vLLM serves Mistral Small 4 **only** through its native loader
(`--config-format mistral --load-format mistral --tokenizer-mode mistral`) —
there is no HF `Mistral4` serving path in any vLLM version. But `llm-compressor`
quantizes the **HF** checkpoint. So the pipeline is:

```
HF bf16  ──quant──▶  HF NVFP4  ──convert──▶  native NVFP4  ──serve──▶  vLLM
        nvfp4_quant.py      convert_hf_to_native.py        (native loader)
```

## Pipeline (on ana-ml2, `/tank/aimodels/quant-work`, in a uv venv)

```bash
# 0. Pull the HF bf16 source (e.g. via huggingface-cli download).

# 1. Quantize HF bf16 -> HF NVFP4  (~65 GB out). GPU0 for compute, CPU-resident model.
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_VISIBLE_DEVICES=0 \
  python nvfp4_quant.py <hf-bf16-dir> heretic-nvfp4 128

# 2. Dry-run the native convert (name-map check vs the official native reference).
python convert_hf_to_native.py --format nvfp4 \
  --hf-dir heretic-nvfp4 \
  --native-ref-dir <official native NVFP4 snapshot dir> \
  --out-dir heretic-native-nvfp4 --dry-run
#    Expect: unmapped=0, missing_from_output=0, extra_in_output=0.

# 3. Full native convert (~65 GB out, 5 shards).
python convert_hf_to_native.py --format nvfp4 \
  --hf-dir heretic-nvfp4 --native-ref-dir <ref> \
  --out-dir heretic-native-nvfp4 --max-shard-size-gb 15

# 4. Serve-test on vLLM (native loader, v0.22.0 = last vision-working pin).
docker run -d --name heretic-serve-test --ipc host --gpus '"device=0"' \
  -p 8099:8000 -v $PWD/heretic-native-nvfp4:/model:ro \
  vllm/vllm-openai:v0.22.0 /model --served-model-name heretic-test \
  --host 0.0.0.0 --port 8000 \
  --tokenizer-mode mistral --config-format mistral --load-format mistral \
  --tensor-parallel-size 1 --gpu-memory-utilization 0.93 \
  --max-model-len 16384 --attention-backend TRITON_MLA --max-num-seqs 8 --dtype auto
```

## Gotchas (each one cost a failed run)

- **`device_map="cpu"`, not `"auto"`** in the quant. `auto` fills GPU0 with the
  205 GB model → OOM during MoE un-fusing; constraining with `max_memory` then
  offloads experts to the *meta* device, which `copy_from_experts_module` can't
  `.copy_()` (`Cannot copy out of meta tensor`). CPU-resident keeps every tensor
  real; the sequential pipeline still onloads each layer to GPU0 for compute.
- **Non-mmap shard reads** in the converter. `safetensors.safe_open()` mmaps the
  whole shard; on `/tank` (ZFS) a 50 GB shard mmap ENOMEMs regardless of free RAM
  (MAP_SHARED never consults the commit limit). `read_tensor` reads with plain
  `read()` + `safetensors.torch.load(bytes)`, caching one shard at a time — the
  copy loop is sorted by shard so the cache doesn't thrash.
- **`vm.overcommit_memory=1`** on ana-ml2 (now durable — see
  `playbooks/ana-ml2-overcommit-memory.yaml`). overcommit=0 + zero swap caps the
  CommitLimit at ~RAM/2; the resident vLLM services eat the headroom and large
  allocations fail despite free RAM.
- **NVFP4 output keeps the `model.` prefix.** llm-compressor's NVFP4 tensor names
  are `model.language_model.model.layers.N...` (same prefix as bf16) — they are
  *not* prefix-shifted. The only NVFP4 difference vs bf16 is per-expert-quantized
  (`mlp.experts.E.{gate,up,down}_proj.{weight_packed,...}`) vs fused.
- **Vision tower stays bf16.** The IGNORE list excludes `vision_tower` +
  `multi_modal_projector` (and all MLA attention, the MoE gate, embeddings,
  lm_head) — only the expert FFN is NVFP4. So the vision encoder is byte-for-byte
  full precision; any vision-quality nuance is the quantized LLM backbone, not the
  tower.

## Serve-test results (2026-06-17, heretic native NVFP4)

- Loads on the vLLM native loader (v0.22.0), GPU0, ~91.9 GB at util 0.93.
- Text: correct (`2+2 = 4`, `capital of Japan = Tokyo`).
- Vision: tower functional — colors + spatial position accurate; exact shape
  geometry fuzzy on small synthetic images (circle → pentagon). Evaluate real
  vision quality during character tuning, against the official NVFP4 as baseline.

## Provenance

`convert_hf_to_native.py` originated with **worldtree-codex** (HF↔native bin
maps + the fused-expert split for the bf16 path). Fixed here: the NVFP4 layer
regexes (the `model.` prefix), and non-mmap shard reads. `nvfp4_quant.py` is
local. Homed in this infra repo per operator direction (not Worldtree).