feat(heretic2-nvfp4): MTP-graft + NVFP4 quant scripts + pipeline README (fire-ready)
graft_mtp.py: grafts the 15 base-Qwen3.6 MTP tensors into Heretic2 BF16 (CPU-only). quant_nvfp4.py: llm-compressor NVFP4 (Linear only; GDN/vision/lm-head/norms/MTP kept BF16 per robbatt's deckard recipe + brokkr's spec); text (AEON-baseline) or chat (production, apply_chat_template renders qwen3_coder XML) calib modes. README: fire sequence + gates (GPU window, production calib) + artifacts. Spike gated only on: (1) off-peak Blackwell GPU window, (2) brokkr's production calib.
This commit is contained in:
@@ -0,0 +1,48 @@
|
||||
# heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP)
|
||||
|
||||
Local NVFP4 quant of **NEO-CODE = Heretic2-Thinking** (Qwen3.6-27B) with the
|
||||
Qwen3.6 **MTP head grafted back**, for a ~2.5–4× faster vLLM/MTP
|
||||
`char-rp-reasoning` seat (buys reasoning-budget headroom → better GM planning
|
||||
inside soong's latency window). R36 fast-seat spike, 2026-07-14.
|
||||
|
||||
Fleet-first **local** NVFP4 quant — every other fleet NVFP4 model is *pulled*
|
||||
pre-quantized; Heretic2 has none published, so we quantize it.
|
||||
|
||||
## Fire sequence
|
||||
1. **`graft_mtp.py`** — graft the 15 base-Qwen3.6 MTP tensors into Heretic2 BF16.
|
||||
CPU-only, no GPU window. (Heretic2's finetune dropped the head; config declares
|
||||
`mtp_num_hidden_layers=1` but ships 0 `mtp.*` tensors — verified.)
|
||||
2. **`quant_nvfp4.py`** — llm-compressor NVFP4, Linear only; **GDN/vision/lm-head/
|
||||
norms/MTP kept BF16** (robbatt deckard-nvfp4 recipe + MTP). NEEDS a freed
|
||||
Blackwell GPU (~55 GB) + an llmcompressor env (run in a vLLM container:
|
||||
`pip install llmcompressor` on `vllm/vllm-openai:v0.24.0`).
|
||||
- baseline: `--calib-mode text --calib neuralmagic/calibration` (AEON control)
|
||||
- production: `--calib-mode chat --calib <512-row mix>` (brokkr/Dvalin) — rows
|
||||
rendered via `apply_chat_template(enable_thinking=True)` so the forward-pass
|
||||
sees the qwen3_coder tool-call XML = the seat's native activations.
|
||||
3. **serve** (pantheon-27b-mtp-nvfp4 pattern): `vllm --quantization compressed-tensors
|
||||
--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
|
||||
--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice`.
|
||||
4. **P00 acceptance** (brokkr): soong 9-tool k5 rig on the quant — must hold
|
||||
~0.967 / perfect `attach_tool`. This is the **authoritative #355 check, NOT KL**
|
||||
(KL can pass while the structured-tool path regresses).
|
||||
|
||||
## Gates
|
||||
- **GPU window** — both ana-ml2 Blackwell GPUs run ~full; NVFP4 is Blackwell-only
|
||||
(irv-ml1's Ampere can't). The ~30–60 min quant needs a brief off-peak window
|
||||
freeing a GPU. *Operator's call.*
|
||||
- **Production calib** — brokkr/Dvalin assembling the 512-row mix; the tool-call-XML
|
||||
slice (128 rows, 53 `attach_tool`) is ready. The AEON-baseline is fireable now.
|
||||
|
||||
## Artifacts (on ana-ml2)
|
||||
- Heretic2 BF16 (target): `/tank/aimodels/huggingface/hub/models--DavidAU--Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking`
|
||||
- base Qwen3.6-27B (MTP source): `/tank/aimodels/huggingface/hub/models--Qwen--Qwen3.6-27B` (15 mtp.* tensors, shards 13+15)
|
||||
- AEON-baseline calib: `neuralmagic/calibration` (HF)
|
||||
- `soong-tools-v0.3.13.json` — live 9-tool schema (structure source-of-truth;
|
||||
**calib uses the PREFIXED `bifrost.soong-lab.*` runtime names** the seat emits)
|
||||
- `extract_soong_tools.py` — how that schema was pulled from the deployed backend
|
||||
|
||||
## Serve target
|
||||
Replaces the current llama.cpp GGUF `char-rp-reasoning` seat (~59.5 tok/s) once
|
||||
P00 passes. Deckard stays staged as rollback; the GGUF seat is the fallback until
|
||||
the NVFP4 seat is validated + cut over.
|
||||
Reference in New Issue
Block a user