# heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP)

Local NVFP4 quant of **NEO-CODE = Heretic2-Thinking** (Qwen3.6-27B) with the
Qwen3.6 **MTP head grafted back**, for a ~2.5–4× faster vLLM/MTP
`char-rp-reasoning` seat (buys reasoning-budget headroom → better GM planning
inside soong's latency window). R36 fast-seat spike, 2026-07-14.

Fleet-first **local** NVFP4 quant — every other fleet NVFP4 model is *pulled*
pre-quantized; Heretic2 has none published, so we quantize it.

## 2026-07-14 status — gibberish FIXED, format PIVOTED to modelopt for MTP

- **Root cause of the `!!!!` was the quant NAMESPACE**, not calib/scheme: `quant_nvfp4.py`
  loaded via `AutoModelForCausalLM` → text-only `Qwen3_5ForCausalLM` → flat `model.layers.*`
  keys, but vLLM 0.24 serves only `Qwen3_5ForConditionalGeneration`, whose weight mapper needs
  `model.language_model.*`. **Fixed** by loading as `AutoModelForImageTextToText`
  (= `Qwen3_5ForConditionalGeneration`) → keys born `model.language_model.*` + `model.visual.*`.
  NVFP4 now serves **coherent**.
- **But this (llm-compressor / compressed-tensors) format can't deliver the speed goal:** base
  NVFP4 ≈ 53 tok/s ≈ the GGUF seat's ~59.5 at batch-1 (no single-stream win), and **MTP =
  0% acceptance** (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp head off a compressed-tensors
  main model). The mtp tensors are identical to AEON's; the blocker is purely the main-model format.
- **→ Working native MTP requires the MODELOPT format** (what AEON uses, ~3.3/3 accept). Use
  **`quant_modelopt.py`** (nvidia-modelopt PTQ; same graft + same `splice_mtp.py` + serve
  `--quantization modelopt`). AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` (`vllm-aeon-rp`) is the
  exact reference. `quant_nvfp4.py` (below) is kept for the coherent-but-MTP-inert compressed-tensors
  artifact and as the namespace-fix record.

## Fire sequence
1. **`graft_mtp.py`** — graft the 15 base-Qwen3.6 MTP tensors into Heretic2 BF16.
   CPU-only, no GPU window. (Heretic2's finetune dropped the head; config declares
   `mtp_num_hidden_layers=1` but ships 0 `mtp.*` tensors — verified.)
2. **`quant_nvfp4.py`** — llm-compressor NVFP4, Linear only; **GDN/vision/lm-head/
   norms/MTP kept BF16** (robbatt deckard-nvfp4 recipe + MTP). NEEDS a freed
   Blackwell GPU (~55 GB) + an llmcompressor env (run in a vLLM container:
   `pip install llmcompressor` on `vllm/vllm-openai:v0.24.0`).
   - baseline: `--calib-mode text --calib neuralmagic/calibration` (AEON control)
   - production: `--calib-mode chat --calib <512-row mix>` (brokkr/Dvalin) — rows
     rendered via `apply_chat_template(enable_thinking=True)` so the forward-pass
     sees the qwen3_coder tool-call XML = the seat's native activations.
3. **serve** (pantheon-27b-mtp-nvfp4 pattern): `vllm --quantization compressed-tensors
   --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
   --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice`.
4. **P00 acceptance** (brokkr): soong 9-tool k5 rig on the quant — must hold
   ~0.967 / perfect `attach_tool`. This is the **authoritative #355 check, NOT KL**
   (KL can pass while the structured-tool path regresses).

## Gates
- **GPU window** — both ana-ml2 Blackwell GPUs run ~full; NVFP4 is Blackwell-only
  (irv-ml1's Ampere can't). The ~30–60 min quant needs a brief off-peak window
  freeing a GPU. *Operator's call.*
- **Production calib** — brokkr/Dvalin assembling the 512-row mix; the tool-call-XML
  slice (128 rows, 53 `attach_tool`) is ready. The AEON-baseline is fireable now.

## Artifacts (on ana-ml2)
- Heretic2 BF16 (target): `/tank/aimodels/huggingface/hub/models--DavidAU--Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking`
- base Qwen3.6-27B (MTP source): `/tank/aimodels/huggingface/hub/models--Qwen--Qwen3.6-27B` (15 mtp.* tensors, shards 13+15)
- AEON-baseline calib: `neuralmagic/calibration` (HF)
- `soong-tools-v0.3.13.json` — live 9-tool schema (structure source-of-truth;
  **calib uses the PREFIXED `bifrost.soong-lab.*` runtime names** the seat emits)
- `extract_soong_tools.py` — how that schema was pulled from the deployed backend

## Serve target
Replaces the current llama.cpp GGUF `char-rp-reasoning` seat (~59.5 tok/s) once
P00 passes. Deckard stays staged as rollback; the GGUF seat is the fallback until
the NVFP4 seat is validated + cut over.