feat(heretic2-nvfp4): WORKING modelopt NVFP4+MTP seat + full recipe runbook

The fast char-rp-reasoning seat works: ~77 tok/s (vs GGUF ~59.5, base NVFP4 ~53),
MTP draft-acceptance 32-40%, mean acceptance length 2.19. Same Heretic2/NEO-CODE
model, NVFP4 + native qwen3_5_mtp spec-decode.

Full end-to-end recipe + the four landmines in docs/runbooks/heretic2-nvfp4-mtp-seat.md:
(1) load as AutoModelForImageTextToText not AutoModelForCausalLM (namespace/gibberish);
(2) modelopt format not compressed-tensors (compressed-tensors MTP = 0% accept);
(3) modelopt 0.45 <-> transformers 5.12.1 FusedMoE crash (guarded in quant_modelopt.py);
(4) vLLM 0.24.0 does NOT propagate modelopt exclude_modules to the spec-decode draft
model -> BF16 mtp head gets quantized -> shape crash; no checkpoint config fixes it
(is_layer_skipped is exact-membership not glob) -> fix is a mounted sitecustomize that
force-skips mtp.* in is_layer_skipped (upstream vLLM bug to report).

Scripts: quant_modelopt.py (FusedMoE guard + single-shard export + multimodal load),
finalize_modelopt_mtp.py (splice bf16 mtp), serve_modelopt_mtp.sh, run_quant_modelopt.sh,
sitecustomize-mtp-workaround.py.

docs/runbooks/heretic2-nvfp4-mtp-seat.md

@@ -0,0 +1,144 @@
+# Heretic2 NVFP4 + MTP fast char-rp-reasoning seat — the working recipe
+
+**Status: WORKING (2026-07-14).** ~77 tok/s single-stream (vs GGUF NEO-CODE ~59.5, base
+NVFP4 ~53) — **~1.3× over GGUF**, MTP draft-acceptance **32–40%**, mean acceptance length
+**2.19**. This is a drop-in faster replacement for the GGUF NEO-CODE `char-rp-reasoning`
+seat (same Heretic2/NEO-CODE model, NVFP4 + native MTP spec-decode).
+
+This runbook exists because getting here was a multi-hour fire drill. **Every gotcha below
+cost real time — read them before touching this.** The TL;DR: three things all had to be
+right at once — (1) quant as the *multimodal* class, (2) use the *modelopt* format not
+compressed-tensors, (3) work around a vLLM bug that quantizes the MTP draft head.
+
+---
+
+## What / where
+
+- **Model:** NEO-CODE = `DavidAU/Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking` (dense
+  27B, `Qwen3_5` GDN-hybrid arch, multimodal `Qwen3_5ForConditionalGeneration`).
+- **Runs only on ana-ml2 GPU0** (NVFP4 is Blackwell-only; irv-ml1 is Ampere).
+- **Artifacts** (ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/`, root-owned):
+  - `heretic2-mtp-bf16/` — BF16 graft (Heretic2 + 15 base-Qwen3.6 MTP tensors). [graft input]
+  - `heretic2-modelopt-nvfp4/` — modelopt NVFP4 quant, single shard, **no mtp**. [quant output]
+  - `heretic2-modelopt-nvfp4-mtp/` — the above + spliced 15 BF16 mtp → **the seat**. [SERVE THIS]
+  - (superseded: `heretic2-nvfp4-cg*` = compressed-tensors path, coherent but MTP-inert;
+    `heretic2-mtp-nvfp4-prod` = original gibberish. Keep for diff, do not serve.)
+- **Scripts** (eshpfi `services/heretic2-nvfp4-quant/`): `graft_mtp.py`, `quant_modelopt.py`,
+  `finalize_modelopt_mtp.py`, `serve_modelopt_mtp.sh`, `sitecustomize-mtp-workaround.py`.
+- **Reference:** the MoE `gen` (`qwen36-35b-a3b-heretic-nvfp4`, `quant_method: modelopt`) and
+  the qwopus-122B `gen` both ran MTP before (qwopus +12% single-stream, archival-memory
+  2026-07-01) — dropped for `gen` because MTP *hurts concurrency*, which is why it belongs on
+  the single-stream RP seats, not `gen`.
+
+## GPU window ritual
+
+Base NVFP4 quant needs ~55 GB free on GPU0. `docker stop llama-charrp
+llama-charrp-reasoning vllm-aeon-gen` (→ ~97 GB free); restore with `docker start …`
+(~90–230 s to healthy). The GGUF NEO-CODE seat is the always-restorable fallback. Heads-up
+wt-dev (their character / thoughtful-character / gen route through these) — unless told
+otherwise. `ssh ana-ml2` = lkraven, in the docker group (no sudo needed for docker).
+
+---
+
+## The pipeline (4 steps)
+
+### 1. GRAFT (CPU, seats up) — `graft_mtp.py`
+Heretic2's finetune dropped the MTP head; graft the 15 BF16 `mtp.*` tensors from base
+`Qwen/Qwen3.6-27B` (shards 13+15). Symlinks Heretic2 shards + one `model-mtp.safetensors`.
+Idempotent, refuses to clobber. Output: `heretic2-mtp-bf16/`.
+
+### 2. QUANT (GPU0 window, ~18 min) — `quant_modelopt.py` via `run_quant_modelopt.sh`
+`nvidia-modelopt` PTQ → **modelopt** NVFP4 format. Three things this script gets right (each a
+gotcha — see below): loads as **`AutoModelForImageTextToText`**, patches the modelopt↔transformers
+**FusedMoE** bug, and forces **single-shard** export. Excludes `lm_head` + `visual` + all
+`linear_attn` (GDN) → BF16, matching AEON. Calib = the 512-row workload-matched chat mix.
+```bash
+docker run -d --name vllm-heretic2-modelopt-quant --gpus '"device=0"' --ipc host \
+  -v /tank/aimodels:/tank/aimodels -v /home/lkraven:/lk \
+  --entrypoint bash vllm/vllm-openai:v0.24.0 -c '
+    set -e
+    pip install -q nvidia-modelopt tiktoken sentencepiece 2>&1 | tail -1
+    python3 /lk/quant_modelopt.py \
+      --model /tank/aimodels/heretic2-nvfp4-work/heretic2-mtp-bf16 \
+      --calib-mode chat --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
+      --num-samples 512 --seqlen 8192 \
+      --out /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4'
+```
+CPU dry-run (no GPU, tiny calib) to validate the pipeline without an outage: same command
+minus `--gpus`, add `-e CUDA_VISIBLE_DEVICES=""`, `--num-samples 2 --seqlen 512`.
+
+### 3. SPLICE (CPU) — `finalize_modelopt_mtp.py`
+Copy `heretic2-modelopt-nvfp4` → `heretic2-modelopt-nvfp4-mtp`, splice the 15 BF16 `mtp.*`
+tensors into the single shard (→ 1967 tensors). (The transformers load never builds an mtp
+module, so mtp must be spliced post-quant — same as AEON/pantheon.)
+
+### 4. SERVE (GPU0) — `serve_modelopt_mtp.sh` + the MTP workaround
+```bash
+docker run -d --name vllm-charrp-modelopt --gpus '"device=0"' --ipc host \
+  -v /tank/aimodels:/tank/aimodels \
+  -v <sitecustomize dir>:/lk_debug -e PYTHONPATH=/lk_debug \    # ← the MTP workaround, see below
+  -p 8018:8000 vllm/vllm-openai:v0.24.0 \
+  /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4-mtp \
+  --quantization modelopt \
+  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \
+  --language-model-only --mamba-cache-dtype float32 \
+  --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \
+  --served-model-name char-rp-reasoning --max-model-len 40960 --max-num-seqs 32 \
+  --gpu-memory-utilization 0.5 --trust-remote-code
+```
+`--language-model-only` skips the vision tower (RP seat doesn't need it; saves ~1–2 GB — the
+tower is preserved BF16 in the weights, so multimodal is recoverable by dropping the flag).
+
+---
+
+## The four landmines (each cost hours)
+
+1. **Load as `AutoModelForImageTextToText`, NEVER `AutoModelForCausalLM`.** The latter resolves
+   `qwen3_5` → text-only `Qwen3_5ForCausalLM` → flat `model.layers.*` keys. vLLM only serves
+   `Qwen3_5ForConditionalGeneration`, whose weight mapper needs `model.language_model.*` (+
+   `model.visual.*`). Wrong class → every layer weight silently fails to load → **`!!!!` gibberish**.
+
+2. **Use the MODELOPT format (nvidia-modelopt), not compressed-tensors (llm-compressor).** On
+   compressed-tensors the MTP drafter can't load the BF16 mtp head at all (`not found in
+   params_dict`, **0% acceptance** — loads but never accelerates; this is what pantheon and the
+   "AEON RP seat" actually were). Base NVFP4 *alone* ≈ GGUF at batch-1 (no single-stream win) —
+   **the MTP multiplier is the entire point**, and it needs modelopt.
+
+3. **modelopt 0.45 ↔ transformers 5.12.1 FusedMoE crash.** `mtq.quantize` dies with
+   `TypeError: issubclass() arg 2 must be a class` — modelopt registered transformers' `FusedMoE`
+   (a *function* in 5.x) as an nn class. `quant_modelopt.py` guards it (patches
+   `_DMRegistryCls._get_registered_nn_class` to skip non-class registry entries). Do **not**
+   pin `nvidia-modelopt[hf]==0.43` to dodge it — that drags transformers back to 4.57 which can't
+   load `qwen3_5` at all.
+
+4. **⭐ THE BIG ONE — vLLM 0.24.0 does not propagate modelopt `exclude_modules` to the
+   spec-decode DRAFT model.** The MTP drafter builds its own `qkv_proj`/`gate_up_proj` as
+   *quantized* (NVFP4-packed) while the mtp head is BF16 → `AssertionError: param_data.shape ==
+   loaded_weight.shape` in `qwen3_5_mtp.py:256`. **No checkpoint config fixes this** — instrumenting
+   `is_layer_skipped` proved the drafter's exclude list contains only the *main* model's
+   `linear_attn` entries, never the mtp ones. Also note `is_layer_skipped` does **exact string
+   membership, not glob** — so wildcards like `mtp.layers.0.*` never match anything. **Fix = a
+   runtime patch** (`sitecustomize-mtp-workaround.py`, mounted on `PYTHONPATH`) that force-skips
+   any `mtp.*` prefix in `is_layer_skipped`, keeping the drafter BF16. This is a genuine vLLM bug —
+   **report upstream** (draft-model quant-config should inherit the target's exclude_modules).
+
+## Verify it's actually accelerating
+
+```bash
+# coherence
+curl -s :8018/v1/completions -d '{"model":"char-rp-reasoning","prompt":"The old tavern","max_tokens":40,"temperature":0}'
+# drive tokens, then read acceptance from the seat log:
+docker logs vllm-charrp-modelopt 2>&1 | grep SpecDecoding | tail -2
+#   -> "Mean acceptance length: 2.19 ... Avg Draft acceptance rate: 39.7%"   [GOOD: >0%, ~2 length]
+#   -> "Avg Draft acceptance rate: 0.0%"                                       [BAD: compressed-tensors, or mtp quantized]
+```
+`SpecDecoding` line only appears during active generation. 0% acceptance = you're on
+compressed-tensors, or the workaround didn't load (check for `[ISLS] ... workaround installed`).
+
+## Productionization TODO (not yet done)
+- Bake the sitecustomize workaround into a compose stack (mount + `PYTHONPATH`), served-name
+  `char-rp-reasoning`, alongside/replacing the GGUF seat.
+- brokkr P00 (soong 9-tool k5) — same base model as GGUF NEO-CODE so R36 should carry, but the
+  NVFP4-vs-Q5 quality + tool-path must be confirmed before cutover.
+- Repoint gateway `char-rp-reasoning` alias + heads-up wt-dev.
+- File the vLLM upstream bug (draft-model exclude non-inheritance).