esh-pfi-infrastructure/docs/runbooks/heretic2-nvfp4-mtp-seat.md

Heretic2 NVFP4 + MTP fast char-rp-reasoning seat — the working recipe

Status: WORKING (2026-07-14). ~77 tok/s single-stream (vs GGUF NEO-CODE ~59.5, base NVFP4 ~53) — ~1.3× over GGUF, MTP draft-acceptance 32–40%, mean acceptance length 2.19. This is a drop-in faster replacement for the GGUF NEO-CODE char-rp-reasoning seat (same Heretic2/NEO-CODE model, NVFP4 + native MTP spec-decode).

This runbook exists because getting here was a multi-hour fire drill. Every gotcha below cost real time — read them before touching this. The TL;DR: three things all had to be right at once — (1) quant as the multimodal class, (2) use the modelopt format not compressed-tensors, (3) work around a vLLM bug that quantizes the MTP draft head.

---

What / where

• Model: NEO-CODE = DavidAU/Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking (dense 27B, Qwen3_5 GDN-hybrid arch, multimodal Qwen3_5ForConditionalGeneration).
• Runs only on ana-ml2 GPU0 (NVFP4 is Blackwell-only; irv-ml1 is Ampere).
• Artifacts (ana-ml2 /tank/aimodels/heretic2-nvfp4-work/, root-owned):
  • heretic2-mtp-bf16/ — BF16 graft (Heretic2 + 15 base-Qwen3.6 MTP tensors). [graft input]
  • heretic2-modelopt-nvfp4/ — modelopt NVFP4 quant, single shard, no mtp. [quant output]
  • heretic2-modelopt-nvfp4-mtp/ — the above + spliced 15 BF16 mtp → the seat. [SERVE THIS]
  • (superseded: heretic2-nvfp4-cg* = compressed-tensors path, coherent but MTP-inert; heretic2-mtp-nvfp4-prod = original gibberish. Keep for diff, do not serve.)
• Scripts (eshpfi services/heretic2-nvfp4-quant/): graft_mtp.py, quant_modelopt.py, finalize_modelopt_mtp.py, serve_modelopt_mtp.sh, sitecustomize-mtp-workaround.py.
• Reference: the MoE gen (qwen36-35b-a3b-heretic-nvfp4, quant_method: modelopt) and the qwopus-122B gen both ran MTP before (qwopus +12% single-stream, archival-memory 2026-07-01) — dropped for gen because MTP hurts concurrency, which is why it belongs on the single-stream RP seats, not gen.

GPU window ritual

Base NVFP4 quant needs ~55 GB free on GPU0. docker stop llama-charrp llama-charrp-reasoning vllm-aeon-gen (→ ~97 GB free); restore with docker start … (~90–230 s to healthy). The GGUF NEO-CODE seat is the always-restorable fallback. Heads-up wt-dev (their character / thoughtful-character / gen route through these) — unless told otherwise. ssh ana-ml2 = lkraven, in the docker group (no sudo needed for docker).

---

The pipeline (4 steps)

1. GRAFT (CPU, seats up) — graft_mtp.py

Heretic2's finetune dropped the MTP head; graft the 15 BF16 mtp.* tensors from base Qwen/Qwen3.6-27B (shards 13+15). Symlinks Heretic2 shards + one model-mtp.safetensors. Idempotent, refuses to clobber. Output: heretic2-mtp-bf16/.

2. QUANT (GPU0 window, ~18 min) — quant_modelopt.py via run_quant_modelopt.sh

nvidia-modelopt PTQ → modelopt NVFP4 format. Three things this script gets right (each a gotcha — see below): loads as AutoModelForImageTextToText, patches the modelopt↔transformers FusedMoE bug, and forces single-shard export. Excludes lm_head + visual + all linear_attn (GDN) → BF16, matching AEON. Calib = the 512-row workload-matched chat mix.

docker run -d --name vllm-heretic2-modelopt-quant --gpus '"device=0"' --ipc host \
  -v /tank/aimodels:/tank/aimodels -v /home/lkraven:/lk \
  --entrypoint bash vllm/vllm-openai:v0.24.0 -c '
    set -e
    pip install -q nvidia-modelopt tiktoken sentencepiece 2>&1 | tail -1
    python3 /lk/quant_modelopt.py \
      --model /tank/aimodels/heretic2-nvfp4-work/heretic2-mtp-bf16 \
      --calib-mode chat --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
      --num-samples 512 --seqlen 8192 \
      --out /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4'

CPU dry-run (no GPU, tiny calib) to validate the pipeline without an outage: same command minus --gpus, add -e CUDA_VISIBLE_DEVICES="", --num-samples 2 --seqlen 512.

3. SPLICE (CPU) — finalize_modelopt_mtp.py

Copy heretic2-modelopt-nvfp4 → heretic2-modelopt-nvfp4-mtp, splice the 15 BF16 mtp.* tensors into the single shard (→ 1967 tensors). (The transformers load never builds an mtp module, so mtp must be spliced post-quant — same as AEON/pantheon.)

4. SERVE (GPU0) — serve_modelopt_mtp.sh + the MTP workaround

docker run -d --name vllm-charrp-modelopt --gpus '"device=0"' --ipc host \
  -v /tank/aimodels:/tank/aimodels \
  -v <sitecustomize dir>:/lk_debug -e PYTHONPATH=/lk_debug \    # ← the MTP workaround, see below
  -p 8018:8000 vllm/vllm-openai:v0.24.0 \
  /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4-mtp \
  --quantization modelopt \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \
  --language-model-only --mamba-cache-dtype float32 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \
  --served-model-name char-rp-reasoning --max-model-len 40960 --max-num-seqs 32 \
  --gpu-memory-utilization 0.5 --trust-remote-code

--language-model-only skips the vision tower (RP seat doesn't need it; saves ~1–2 GB — the tower is preserved BF16 in the weights, so multimodal is recoverable by dropping the flag).

---

The four landmines (each cost hours)

1. Load as AutoModelForImageTextToText, NEVER AutoModelForCausalLM. The latter resolves qwen3_5 → text-only Qwen3_5ForCausalLM → flat model.layers.* keys. vLLM only serves Qwen3_5ForConditionalGeneration, whose weight mapper needs model.language_model.* (+ model.visual.*). Wrong class → every layer weight silently fails to load → !!!! gibberish.

2. Use the MODELOPT format (nvidia-modelopt), not compressed-tensors (llm-compressor). On compressed-tensors the MTP drafter can't load the BF16 mtp head at all (not found in params_dict, 0% acceptance — loads but never accelerates; this is what pantheon and the "AEON RP seat" actually were). Base NVFP4 alone ≈ GGUF at batch-1 (no single-stream win) — the MTP multiplier is the entire point, and it needs modelopt.

3. modelopt 0.45 ↔ transformers 5.12.1 FusedMoE crash. mtq.quantize dies with TypeError: issubclass() arg 2 must be a class — modelopt registered transformers' FusedMoE (a function in 5.x) as an nn class. quant_modelopt.py guards it (patches _DMRegistryCls._get_registered_nn_class to skip non-class registry entries). Do not pin nvidia-modelopt[hf]==0.43 to dodge it — that drags transformers back to 4.57 which can't load qwen3_5 at all.

4. ⭐ THE BIG ONE — vLLM 0.24.0 does not propagate modelopt exclude_modules to the spec-decode DRAFT model. The MTP drafter builds its own qkv_proj/gate_up_proj as quantized (NVFP4-packed) while the mtp head is BF16 → AssertionError: param_data.shape == loaded_weight.shape in qwen3_5_mtp.py:256. No checkpoint config fixes this — instrumenting is_layer_skipped proved the drafter's exclude list contains only the main model's linear_attn entries, never the mtp ones. Also note is_layer_skipped does exact string membership, not glob — so wildcards like mtp.layers.0.* never match anything. Fix = a runtime patch (sitecustomize-mtp-workaround.py, mounted on PYTHONPATH) that force-skips any mtp.* prefix in is_layer_skipped, keeping the drafter BF16. This is a genuine vLLM bug — report upstream (draft-model quant-config should inherit the target's exclude_modules).

Verify it's actually accelerating

# coherence
curl -s :8018/v1/completions -d '{"model":"char-rp-reasoning","prompt":"The old tavern","max_tokens":40,"temperature":0}'
# drive tokens, then read acceptance from the seat log:
docker logs vllm-charrp-modelopt 2>&1 | grep SpecDecoding | tail -2
#   -> "Mean acceptance length: 2.19 ... Avg Draft acceptance rate: 39.7%"   [GOOD: >0%, ~2 length]
#   -> "Avg Draft acceptance rate: 0.0%"                                       [BAD: compressed-tensors, or mtp quantized]

SpecDecoding line only appears during active generation. 0% acceptance = you're on compressed-tensors, or the workaround didn't load (check for [ISLS] ... workaround installed).

Productionization TODO (not yet done)

• Bake the sitecustomize workaround into a compose stack (mount + PYTHONPATH), served-name char-rp-reasoning, alongside/replacing the GGUF seat.
• brokkr P00 (soong 9-tool k5) — same base model as GGUF NEO-CODE so R36 should carry, but the NVFP4-vs-Q5 quality + tool-path must be confirmed before cutover.
• Repoint gateway char-rp-reasoning alias + heads-up wt-dev.
• File the vLLM upstream bug (draft-model exclude non-inheritance).