esh-pfi-infrastructure/docs/runbooks/heretic2-nvfp4-mtp-seat.md

# Heretic2 NVFP4 + MTP fast char-rp-reasoning seat — the working recipe

**Status: WORKING (2026-07-14).** ~77 tok/s single-stream (vs GGUF NEO-CODE ~59.5, base
NVFP4 ~53) — **~1.3× over GGUF**, MTP draft-acceptance **32–40%**, mean acceptance length
**2.19**. This is a drop-in faster replacement for the GGUF NEO-CODE `char-rp-reasoning`
seat (same Heretic2/NEO-CODE model, NVFP4 + native MTP spec-decode).

This runbook exists because getting here was a multi-hour fire drill. **Every gotcha below
cost real time — read them before touching this.** The TL;DR: three things all had to be
right at once — (1) quant as the *multimodal* class, (2) use the *modelopt* format not
compressed-tensors, (3) work around a vLLM bug that quantizes the MTP draft head.

---

## What / where

- **Model:** NEO-CODE = `DavidAU/Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking` (dense
  27B, `Qwen3_5` GDN-hybrid arch, multimodal `Qwen3_5ForConditionalGeneration`).
- **Runs only on ana-ml2 GPU0** (NVFP4 is Blackwell-only; irv-ml1 is Ampere).
- **Artifacts** (ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/`, root-owned):
  - `heretic2-mtp-bf16/` — BF16 graft (Heretic2 + 15 base-Qwen3.6 MTP tensors). [graft input]
  - `heretic2-modelopt-nvfp4/` — modelopt NVFP4 quant, single shard, **no mtp**. [quant output]
  - `heretic2-modelopt-nvfp4-mtp/` — the above + spliced 15 BF16 mtp → **the seat**. [SERVE THIS]
  - (superseded: `heretic2-nvfp4-cg*` = compressed-tensors path, coherent but MTP-inert;
    `heretic2-mtp-nvfp4-prod` = original gibberish. Keep for diff, do not serve.)
- **Scripts** (eshpfi `services/heretic2-nvfp4-quant/`): `graft_mtp.py`, `quant_modelopt.py`,
  `finalize_modelopt_mtp.py`, `serve_modelopt_mtp.sh`, `sitecustomize-mtp-workaround.py`.
- **Reference:** the MoE `gen` (`qwen36-35b-a3b-heretic-nvfp4`, `quant_method: modelopt`) and
  the qwopus-122B `gen` both ran MTP before (qwopus +12% single-stream, archival-memory
  2026-07-01) — dropped for `gen` because MTP *hurts concurrency*, which is why it belongs on
  the single-stream RP seats, not `gen`.

## GPU window ritual

Base NVFP4 quant needs ~55 GB free on GPU0. `docker stop llama-charrp
llama-charrp-reasoning vllm-aeon-gen` (→ ~97 GB free); restore with `docker start …`
(~90–230 s to healthy). The GGUF NEO-CODE seat is the always-restorable fallback. Heads-up
wt-dev (their character / thoughtful-character / gen route through these) — unless told
otherwise. `ssh ana-ml2` = lkraven, in the docker group (no sudo needed for docker).

---

## The pipeline (4 steps)

### 1. GRAFT (CPU, seats up) — `graft_mtp.py`
Heretic2's finetune dropped the MTP head; graft the 15 BF16 `mtp.*` tensors from base
`Qwen/Qwen3.6-27B` (shards 13+15). Symlinks Heretic2 shards + one `model-mtp.safetensors`.
Idempotent, refuses to clobber. Output: `heretic2-mtp-bf16/`.

### 2. QUANT (GPU0 window, ~18 min) — `quant_modelopt.py` via `run_quant_modelopt.sh`
`nvidia-modelopt` PTQ → **modelopt** NVFP4 format. Three things this script gets right (each a
gotcha — see below): loads as **`AutoModelForImageTextToText`**, patches the modelopt↔transformers
**FusedMoE** bug, and forces **single-shard** export. Excludes `lm_head` + `visual` + all
`linear_attn` (GDN) → BF16, matching AEON. Calib = the 512-row workload-matched chat mix.
```bash
docker run -d --name vllm-heretic2-modelopt-quant --gpus '"device=0"' --ipc host \
  -v /tank/aimodels:/tank/aimodels -v /home/lkraven:/lk \
  --entrypoint bash vllm/vllm-openai:v0.24.0 -c '
    set -e
    pip install -q nvidia-modelopt tiktoken sentencepiece 2>&1 | tail -1
    python3 /lk/quant_modelopt.py \
      --model /tank/aimodels/heretic2-nvfp4-work/heretic2-mtp-bf16 \
      --calib-mode chat --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
      --num-samples 512 --seqlen 8192 \
      --out /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4'
```
CPU dry-run (no GPU, tiny calib) to validate the pipeline without an outage: same command
minus `--gpus`, add `-e CUDA_VISIBLE_DEVICES=""`, `--num-samples 2 --seqlen 512`.

### 3. SPLICE (CPU) — `finalize_modelopt_mtp.py`
Copy `heretic2-modelopt-nvfp4` → `heretic2-modelopt-nvfp4-mtp`, splice the 15 BF16 `mtp.*`
tensors into the single shard (→ 1967 tensors). (The transformers load never builds an mtp
module, so mtp must be spliced post-quant — same as AEON/pantheon.)

### 4. SERVE (GPU0) — `serve_modelopt_mtp.sh` + the MTP workaround
```bash
docker run -d --name vllm-charrp-modelopt --gpus '"device=0"' --ipc host \
  -v /tank/aimodels:/tank/aimodels \
  -v <sitecustomize dir>:/lk_debug -e PYTHONPATH=/lk_debug \    # ← the MTP workaround, see below
  -p 8018:8000 vllm/vllm-openai:v0.24.0 \
  /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4-mtp \
  --quantization modelopt \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \
  --language-model-only --mamba-cache-dtype float32 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \
  --served-model-name char-rp-reasoning --max-model-len 40960 --max-num-seqs 32 \
  --gpu-memory-utilization 0.5 --trust-remote-code
```
`--language-model-only` skips the vision tower (RP seat doesn't need it; saves ~1–2 GB — the
tower is preserved BF16 in the weights, so multimodal is recoverable by dropping the flag).

---

## The four landmines (each cost hours)

1. **Load as `AutoModelForImageTextToText`, NEVER `AutoModelForCausalLM`.** The latter resolves
   `qwen3_5` → text-only `Qwen3_5ForCausalLM` → flat `model.layers.*` keys. vLLM only serves
   `Qwen3_5ForConditionalGeneration`, whose weight mapper needs `model.language_model.*` (+
   `model.visual.*`). Wrong class → every layer weight silently fails to load → **`!!!!` gibberish**.

2. **Use the MODELOPT format (nvidia-modelopt), not compressed-tensors (llm-compressor).** On
   compressed-tensors the MTP drafter can't load the BF16 mtp head at all (`not found in
   params_dict`, **0% acceptance** — loads but never accelerates; this is what pantheon and the
   "AEON RP seat" actually were). Base NVFP4 *alone* ≈ GGUF at batch-1 (no single-stream win) —
   **the MTP multiplier is the entire point**, and it needs modelopt.

3. **modelopt 0.45 ↔ transformers 5.12.1 FusedMoE crash.** `mtq.quantize` dies with
   `TypeError: issubclass() arg 2 must be a class` — modelopt registered transformers' `FusedMoE`
   (a *function* in 5.x) as an nn class. `quant_modelopt.py` guards it (patches
   `_DMRegistryCls._get_registered_nn_class` to skip non-class registry entries). Do **not**
   pin `nvidia-modelopt[hf]==0.43` to dodge it — that drags transformers back to 4.57 which can't
   load `qwen3_5` at all.

4. **⭐ THE BIG ONE — vLLM 0.24.0 does not propagate modelopt `exclude_modules` to the
   spec-decode DRAFT model.** The MTP drafter builds its own `qkv_proj`/`gate_up_proj` as
   *quantized* (NVFP4-packed) while the mtp head is BF16 → `AssertionError: param_data.shape ==
   loaded_weight.shape` in `qwen3_5_mtp.py:256`. **No checkpoint config fixes this** — instrumenting
   `is_layer_skipped` proved the drafter's exclude list contains only the *main* model's
   `linear_attn` entries, never the mtp ones. Also note `is_layer_skipped` does **exact string
   membership, not glob** — so wildcards like `mtp.layers.0.*` never match anything. **Fix = a
   runtime patch** (`sitecustomize-mtp-workaround.py`, mounted on `PYTHONPATH`) that force-skips
   any `mtp.*` prefix in `is_layer_skipped`, keeping the drafter BF16. This is a genuine vLLM bug —
   **report upstream** (draft-model quant-config should inherit the target's exclude_modules).

## Verify it's actually accelerating

```bash
# coherence
curl -s :8018/v1/completions -d '{"model":"char-rp-reasoning","prompt":"The old tavern","max_tokens":40,"temperature":0}'
# drive tokens, then read acceptance from the seat log:
docker logs vllm-charrp-modelopt 2>&1 | grep SpecDecoding | tail -2
#   -> "Mean acceptance length: 2.19 ... Avg Draft acceptance rate: 39.7%"   [GOOD: >0%, ~2 length]
#   -> "Avg Draft acceptance rate: 0.0%"                                       [BAD: compressed-tensors, or mtp quantized]
```
`SpecDecoding` line only appears during active generation. 0% acceptance = you're on
compressed-tensors, or the workaround didn't load (check for `[ISLS] ... workaround installed`).

## Productionization TODO (not yet done)
- Bake the sitecustomize workaround into a compose stack (mount + `PYTHONPATH`), served-name
  `char-rp-reasoning`, alongside/replacing the GGUF seat.
- brokkr P00 (soong 9-tool k5) — same base model as GGUF NEO-CODE so R36 should carry, but the
  NVFP4-vs-Q5 quality + tool-path must be confirmed before cutover.
- Repoint gateway `char-rp-reasoning` alias + heads-up wt-dev.
- File the vLLM upstream bug (draft-model exclude non-inheritance).