Commit Graph

1 Commits

Author SHA1 Message Date
vh dd3a5c93fd feat(tools): Mistral Small 4 NVFP4 build pipeline (quant + HF->native converter)
Quantize a HF-format Mistral Small 4 (Mistral3ForConditionalGeneration MoE) to
NVFP4 with the vision tower intact, then convert HF NVFP4 -> Mistral native so
vLLM can serve it (there is no HF Mistral4 serving path in any vLLM version).

Built + validated end-to-end on ana-ml2 for the abliterated character-model
successor (darkc0de/Mistral-Small-4-119B-2603-heretic): quant -> dry-run (clean
vs the official native NVFP4 reference) -> convert -> serve-test (loads on the
native loader, correct text, vision functional).

Converter scaffold came from worldtree-codex (bf16 bin maps + fused-expert
split); fixed here: NVFP4 layer regexes (keep the `model.` prefix) + non-mmap
shard reads (ZFS large-mmap ENOMEM). nvfp4_quant.py is local. README documents
the pipeline + every gotcha that cost a failed run. Homed here per operator
direction (not Worldtree).
2026-06-17 22:19:58 -07:00