esh-pfi-infrastructure/services/heretic2-nvfp4-quant

heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP)

Local NVFP4 quant of NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) with the Qwen3.6 MTP head grafted back, for a ~2.5–4× faster vLLM/MTP char-rp-reasoning seat (buys reasoning-budget headroom → better GM planning inside soong's latency window). R36 fast-seat spike, 2026-07-14.

Fleet-first local NVFP4 quant — every other fleet NVFP4 model is pulled pre-quantized; Heretic2 has none published, so we quantize it.

Fire sequence

1. graft_mtp.py — graft the 15 base-Qwen3.6 MTP tensors into Heretic2 BF16. CPU-only, no GPU window. (Heretic2's finetune dropped the head; config declares mtp_num_hidden_layers=1 but ships 0 mtp.* tensors — verified.)
2. quant_nvfp4.py — llm-compressor NVFP4, Linear only; GDN/vision/lm-head/ norms/MTP kept BF16 (robbatt deckard-nvfp4 recipe + MTP). NEEDS a freed Blackwell GPU (~55 GB) + an llmcompressor env (run in a vLLM container: pip install llmcompressor on vllm/vllm-openai:v0.24.0).
   • baseline: --calib-mode text --calib neuralmagic/calibration (AEON control)
   • production: --calib-mode chat --calib <512-row mix> (brokkr/Dvalin) — rows rendered via apply_chat_template(enable_thinking=True) so the forward-pass sees the qwen3_coder tool-call XML = the seat's native activations.
3. serve (pantheon-27b-mtp-nvfp4 pattern): vllm --quantization compressed-tensors --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice.
4. P00 acceptance (brokkr): soong 9-tool k5 rig on the quant — must hold ~0.967 / perfect attach_tool. This is the authoritative #355 check, NOT KL (KL can pass while the structured-tool path regresses).

Gates

• GPU window — both ana-ml2 Blackwell GPUs run ~full; NVFP4 is Blackwell-only (irv-ml1's Ampere can't). The ~30–60 min quant needs a brief off-peak window freeing a GPU. Operator's call.
• Production calib — brokkr/Dvalin assembling the 512-row mix; the tool-call-XML slice (128 rows, 53 attach_tool) is ready. The AEON-baseline is fireable now.

Artifacts (on ana-ml2)

• Heretic2 BF16 (target): /tank/aimodels/huggingface/hub/models--DavidAU--Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking
• base Qwen3.6-27B (MTP source): /tank/aimodels/huggingface/hub/models--Qwen--Qwen3.6-27B (15 mtp.* tensors, shards 13+15)
• AEON-baseline calib: neuralmagic/calibration (HF)
• soong-tools-v0.3.13.json — live 9-tool schema (structure source-of-truth; calib uses the PREFIXED bifrost.soong-lab.* runtime names the seat emits)
• extract_soong_tools.py — how that schema was pulled from the deployed backend

Serve target

Replaces the current llama.cpp GGUF char-rp-reasoning seat (~59.5 tok/s) once P00 passes. Deckard stays staged as rollback; the GGUF seat is the fallback until the NVFP4 seat is validated + cut over.