feat(heretic2-nvfp4): WORKING modelopt NVFP4+MTP seat + full recipe runbook
The fast char-rp-reasoning seat works: ~77 tok/s (vs GGUF ~59.5, base NVFP4 ~53), MTP draft-acceptance 32-40%, mean acceptance length 2.19. Same Heretic2/NEO-CODE model, NVFP4 + native qwen3_5_mtp spec-decode. Full end-to-end recipe + the four landmines in docs/runbooks/heretic2-nvfp4-mtp-seat.md: (1) load as AutoModelForImageTextToText not AutoModelForCausalLM (namespace/gibberish); (2) modelopt format not compressed-tensors (compressed-tensors MTP = 0% accept); (3) modelopt 0.45 <-> transformers 5.12.1 FusedMoE crash (guarded in quant_modelopt.py); (4) vLLM 0.24.0 does NOT propagate modelopt exclude_modules to the spec-decode draft model -> BF16 mtp head gets quantized -> shape crash; no checkpoint config fixes it (is_layer_skipped is exact-membership not glob) -> fix is a mounted sitecustomize that force-skips mtp.* in is_layer_skipped (upstream vLLM bug to report). Scripts: quant_modelopt.py (FusedMoE guard + single-shard export + multimodal load), finalize_modelopt_mtp.py (splice bf16 mtp), serve_modelopt_mtp.sh, run_quant_modelopt.sh, sitecustomize-mtp-workaround.py.
This commit is contained in:
@@ -0,0 +1,20 @@
|
||||
#!/bin/bash
|
||||
# Launch the modelopt NVFP4 quant of the grafted Heretic2 on ana-ml2 GPU0 (detached, survives ssh
|
||||
# drop). bare nvidia-modelopt (0.45). The FusedMoE-compat guard + single-shard export + the
|
||||
# multimodal load class are all inside quant_modelopt.py. ~18 min. Output: heretic2-modelopt-nvfp4
|
||||
# (no mtp yet — run finalize_modelopt_mtp.py after). quant_modelopt.py must be at /lk/quant_modelopt.py
|
||||
# (mount /home/lkraven as /lk, or scp it there first).
|
||||
set -euo pipefail
|
||||
docker rm -f vllm-heretic2-modelopt-quant 2>/dev/null || true
|
||||
rm -rf /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4 2>/dev/null || true
|
||||
docker run -d --name vllm-heretic2-modelopt-quant --gpus '"device=0"' --ipc host \
|
||||
-v /tank/aimodels:/tank/aimodels -v /home/lkraven:/lk \
|
||||
--entrypoint bash vllm/vllm-openai:v0.24.0 -c '
|
||||
set -e
|
||||
pip install -q nvidia-modelopt tiktoken sentencepiece 2>&1 | tail -1
|
||||
python3 /lk/quant_modelopt.py \
|
||||
--model /tank/aimodels/heretic2-nvfp4-work/heretic2-mtp-bf16 \
|
||||
--calib-mode chat --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
|
||||
--num-samples 512 --seqlen 8192 \
|
||||
--out /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4'
|
||||
echo "LAUNCHED: $(docker ps --filter name=vllm-heretic2-modelopt-quant --format '{{.Status}}')"
|
||||
Reference in New Issue
Block a user