feat(heretic2-nvfp4): MTP-graft + NVFP4 quant scripts + pipeline README (fire-ready)

graft_mtp.py: grafts the 15 base-Qwen3.6 MTP tensors into Heretic2 BF16 (CPU-only).
quant_nvfp4.py: llm-compressor NVFP4 (Linear only; GDN/vision/lm-head/norms/MTP
kept BF16 per robbatt's deckard recipe + brokkr's spec); text (AEON-baseline) or
chat (production, apply_chat_template renders qwen3_coder XML) calib modes.
README: fire sequence + gates (GPU window, production calib) + artifacts.

Spike gated only on: (1) off-peak Blackwell GPU window, (2) brokkr's production calib.
This commit is contained in:
vh
2026-07-14 08:57:25 -07:00
parent bbbfe5502e
commit 920f9a3709
3 changed files with 305 additions and 0 deletions
+48
View File
@@ -0,0 +1,48 @@
# heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP)
Local NVFP4 quant of **NEO-CODE = Heretic2-Thinking** (Qwen3.6-27B) with the
Qwen3.6 **MTP head grafted back**, for a ~2.5–4× faster vLLM/MTP
`char-rp-reasoning` seat (buys reasoning-budget headroom → better GM planning
inside soong's latency window). R36 fast-seat spike, 2026-07-14.
Fleet-first **local** NVFP4 quant — every other fleet NVFP4 model is *pulled*
pre-quantized; Heretic2 has none published, so we quantize it.
## Fire sequence
1. **`graft_mtp.py`** — graft the 15 base-Qwen3.6 MTP tensors into Heretic2 BF16.
CPU-only, no GPU window. (Heretic2's finetune dropped the head; config declares
`mtp_num_hidden_layers=1` but ships 0 `mtp.*` tensors — verified.)
2. **`quant_nvfp4.py`** — llm-compressor NVFP4, Linear only; **GDN/vision/lm-head/
norms/MTP kept BF16** (robbatt deckard-nvfp4 recipe + MTP). NEEDS a freed
Blackwell GPU (~55 GB) + an llmcompressor env (run in a vLLM container:
`pip install llmcompressor` on `vllm/vllm-openai:v0.24.0`).
- baseline: `--calib-mode text --calib neuralmagic/calibration` (AEON control)
- production: `--calib-mode chat --calib <512-row mix>` (brokkr/Dvalin) — rows
rendered via `apply_chat_template(enable_thinking=True)` so the forward-pass
sees the qwen3_coder tool-call XML = the seat's native activations.
3. **serve** (pantheon-27b-mtp-nvfp4 pattern): `vllm --quantization compressed-tensors
--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice`.
4. **P00 acceptance** (brokkr): soong 9-tool k5 rig on the quant — must hold
~0.967 / perfect `attach_tool`. This is the **authoritative #355 check, NOT KL**
(KL can pass while the structured-tool path regresses).
## Gates
- **GPU window** — both ana-ml2 Blackwell GPUs run ~full; NVFP4 is Blackwell-only
(irv-ml1's Ampere can't). The ~30–60 min quant needs a brief off-peak window
freeing a GPU. *Operator's call.*
- **Production calib** — brokkr/Dvalin assembling the 512-row mix; the tool-call-XML
slice (128 rows, 53 `attach_tool`) is ready. The AEON-baseline is fireable now.
## Artifacts (on ana-ml2)
- Heretic2 BF16 (target): `/tank/aimodels/huggingface/hub/models--DavidAU--Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking`
- base Qwen3.6-27B (MTP source): `/tank/aimodels/huggingface/hub/models--Qwen--Qwen3.6-27B` (15 mtp.* tensors, shards 13+15)
- AEON-baseline calib: `neuralmagic/calibration` (HF)
- `soong-tools-v0.3.13.json` — live 9-tool schema (structure source-of-truth;
**calib uses the PREFIXED `bifrost.soong-lab.*` runtime names** the seat emits)
- `extract_soong_tools.py` — how that schema was pulled from the deployed backend
## Serve target
Replaces the current llama.cpp GGUF `char-rp-reasoning` seat (~59.5 tok/s) once
P00 passes. Deckard stays staged as rollback; the GGUF seat is the fallback until
the NVFP4 seat is validated + cut over.