vh
982c319d9f
The fast char-rp-reasoning seat works: ~77 tok/s (vs GGUF ~59.5, base NVFP4 ~53), MTP draft-acceptance 32-40%, mean acceptance length 2.19. Same Heretic2/NEO-CODE model, NVFP4 + native qwen3_5_mtp spec-decode. Full end-to-end recipe + the four landmines in docs/runbooks/heretic2-nvfp4-mtp-seat.md: (1) load as AutoModelForImageTextToText not AutoModelForCausalLM (namespace/gibberish); (2) modelopt format not compressed-tensors (compressed-tensors MTP = 0% accept); (3) modelopt 0.45 <-> transformers 5.12.1 FusedMoE crash (guarded in quant_modelopt.py); (4) vLLM 0.24.0 does NOT propagate modelopt exclude_modules to the spec-decode draft model -> BF16 mtp head gets quantized -> shape crash; no checkpoint config fixes it (is_layer_skipped is exact-membership not glob) -> fix is a mounted sitecustomize that force-skips mtp.* in is_layer_skipped (upstream vLLM bug to report). Scripts: quant_modelopt.py (FusedMoE guard + single-shard export + multimodal load), finalize_modelopt_mtp.py (splice bf16 mtp), serve_modelopt_mtp.sh, run_quant_modelopt.sh, sitecustomize-mtp-workaround.py.
26 lines
1.5 KiB
Bash
26 lines
1.5 KiB
Bash
#!/bin/bash
|
|
# Serve the modelopt NVFP4 + MTP Heretic2 seat (the WORKING fast char-rp-reasoning seat).
|
|
# ~77 tok/s, MTP acceptance 32-40%. Requires: (1) heretic2-modelopt-nvfp4-mtp built (quant ->
|
|
# finalize), (2) the sitecustomize MTP workaround mounted on PYTHONPATH (vLLM 0.24 draft-model
|
|
# exclude bug — see runbook landmine #4; without it the engine crashes on a shape mismatch).
|
|
set -euo pipefail
|
|
MODEL="${1:-/tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4-mtp}"
|
|
# Dir containing sitecustomize.py (a copy of sitecustomize-mtp-workaround.py named sitecustomize.py):
|
|
WORKAROUND_DIR="${MTP_WORKAROUND_DIR:-/home/lkraven/isls_debug}"
|
|
docker rm -f vllm-charrp-modelopt 2>/dev/null || true
|
|
docker run -d --name vllm-charrp-modelopt --gpus '"device=0"' --ipc host \
|
|
-v /tank/aimodels:/tank/aimodels \
|
|
-v "${WORKAROUND_DIR}":/lk_debug -e PYTHONPATH=/lk_debug \
|
|
-p 8018:8000 \
|
|
vllm/vllm-openai:v0.24.0 \
|
|
"$MODEL" \
|
|
--quantization modelopt \
|
|
--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \
|
|
--language-model-only \
|
|
--mamba-cache-dtype float32 \
|
|
--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \
|
|
--served-model-name char-rp-reasoning \
|
|
--max-model-len 40960 --max-num-seqs 32 --gpu-memory-utilization 0.5 --trust-remote-code
|
|
echo "started: $(docker ps --filter name=vllm-charrp-modelopt --format '{{.Status}}')"
|
|
echo "verify MTP: docker logs vllm-charrp-modelopt 2>&1 | grep -E 'mtp-workaround|SpecDecoding'"
|