feat(heretic2-nvfp4): WORKING modelopt NVFP4+MTP seat + full recipe runbook

The fast char-rp-reasoning seat works: ~77 tok/s (vs GGUF ~59.5, base NVFP4 ~53),
MTP draft-acceptance 32-40%, mean acceptance length 2.19. Same Heretic2/NEO-CODE
model, NVFP4 + native qwen3_5_mtp spec-decode.

Full end-to-end recipe + the four landmines in docs/runbooks/heretic2-nvfp4-mtp-seat.md:
(1) load as AutoModelForImageTextToText not AutoModelForCausalLM (namespace/gibberish);
(2) modelopt format not compressed-tensors (compressed-tensors MTP = 0% accept);
(3) modelopt 0.45 <-> transformers 5.12.1 FusedMoE crash (guarded in quant_modelopt.py);
(4) vLLM 0.24.0 does NOT propagate modelopt exclude_modules to the spec-decode draft
model -> BF16 mtp head gets quantized -> shape crash; no checkpoint config fixes it
(is_layer_skipped is exact-membership not glob) -> fix is a mounted sitecustomize that
force-skips mtp.* in is_layer_skipped (upstream vLLM bug to report).

Scripts: quant_modelopt.py (FusedMoE guard + single-shard export + multimodal load),
finalize_modelopt_mtp.py (splice bf16 mtp), serve_modelopt_mtp.sh, run_quant_modelopt.sh,
sitecustomize-mtp-workaround.py.
This commit is contained in:
vh
2026-07-14 14:41:48 -07:00
parent aca45393c2
commit 982c319d9f
7 changed files with 344 additions and 2 deletions
@@ -0,0 +1,20 @@
#!/bin/bash
# Launch the modelopt NVFP4 quant of the grafted Heretic2 on ana-ml2 GPU0 (detached, survives ssh
# drop). bare nvidia-modelopt (0.45). The FusedMoE-compat guard + single-shard export + the
# multimodal load class are all inside quant_modelopt.py. ~18 min. Output: heretic2-modelopt-nvfp4
# (no mtp yet — run finalize_modelopt_mtp.py after). quant_modelopt.py must be at /lk/quant_modelopt.py
# (mount /home/lkraven as /lk, or scp it there first).
set -euo pipefail
docker rm -f vllm-heretic2-modelopt-quant 2>/dev/null || true
rm -rf /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4 2>/dev/null || true
docker run -d --name vllm-heretic2-modelopt-quant --gpus '"device=0"' --ipc host \
-v /tank/aimodels:/tank/aimodels -v /home/lkraven:/lk \
--entrypoint bash vllm/vllm-openai:v0.24.0 -c '
set -e
pip install -q nvidia-modelopt tiktoken sentencepiece 2>&1 | tail -1
python3 /lk/quant_modelopt.py \
--model /tank/aimodels/heretic2-nvfp4-work/heretic2-mtp-bf16 \
--calib-mode chat --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
--num-samples 512 --seqlen 8192 \
--out /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4'
echo "LAUNCHED: $(docker ps --filter name=vllm-heretic2-modelopt-quant --format '{{.Status}}')"