Files

T

vh aca45393c2 fix(heretic2-nvfp4): quant as ConditionalGeneration (namespace fix) + modelopt recipe for working MTP

Root-caused the NVFP4 gibberish to a quant-namespace bug: quant_nvfp4.py loaded
via AutoModelForCausalLM -> text-only Qwen3_5ForCausalLM -> flat model.layers.* keys,
but vLLM 0.24 serves only Qwen3_5ForConditionalGeneration (whose weight mapper needs
model.language_model.*). Fixed by loading as AutoModelForImageTextToText; NVFP4 now
serves coherent (validated greedy on ana-ml2 GPU0).

Base NVFP4 (compressed-tensors) measured ~53 tok/s (~= GGUF at batch-1, no single-stream
win) and its MTP is 0% acceptance (vLLM's Qwen3_5MTP drafter loads the bf16 mtp head only
off a modelopt main-model checkpoint). Added quant_modelopt.py (nvidia-modelopt PTQ,
matches AEON's NVFP4 W4A4 g16 + lm_head/linear_attn/visual exclusions) as the path to
working native MTP; graft + splice + serve otherwise unchanged.

2026-07-14 13:10:25 -07:00

4.3 KiB

Raw Blame History

heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP)

Local NVFP4 quant of NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) with the Qwen3.6 MTP head grafted back, for a ~2.5–4× faster vLLM/MTP char-rp-reasoning seat (buys reasoning-budget headroom → better GM planning inside soong's latency window). R36 fast-seat spike, 2026-07-14.

Fleet-first local NVFP4 quant — every other fleet NVFP4 model is pulled pre-quantized; Heretic2 has none published, so we quantize it.

2026-07-14 status — gibberish FIXED, format PIVOTED to modelopt for MTP

Root cause of the !!!! was the quant NAMESPACE, not calib/scheme: quant_nvfp4.py loaded via AutoModelForCausalLM → text-only Qwen3_5ForCausalLM → flat model.layers.* keys, but vLLM 0.24 serves only Qwen3_5ForConditionalGeneration, whose weight mapper needs model.language_model.*. Fixed by loading as AutoModelForImageTextToText (= Qwen3_5ForConditionalGeneration) → keys born model.language_model.* + model.visual.*. NVFP4 now serves coherent.
But this (llm-compressor / compressed-tensors) format can't deliver the speed goal: base NVFP4 ≈ 53 tok/s ≈ the GGUF seat's ~59.5 at batch-1 (no single-stream win), and MTP = 0% acceptance (vLLM's Qwen3_5MTP drafter won't load the bf16 mtp head off a compressed-tensors main model). The mtp tensors are identical to AEON's; the blocker is purely the main-model format.
→ Working native MTP requires the MODELOPT format (what AEON uses, ~3.3/3 accept). Use quant_modelopt.py (nvidia-modelopt PTQ; same graft + same splice_mtp.py + serve --quantization modelopt). AEON /tank/aimodels/qwen36-27b-aeon-nvfp4 (vllm-aeon-rp) is the exact reference. quant_nvfp4.py (below) is kept for the coherent-but-MTP-inert compressed-tensors artifact and as the namespace-fix record.

Fire sequence

graft_mtp.py — graft the 15 base-Qwen3.6 MTP tensors into Heretic2 BF16. CPU-only, no GPU window. (Heretic2's finetune dropped the head; config declares mtp_num_hidden_layers=1 but ships 0 mtp.* tensors — verified.)
quant_nvfp4.py — llm-compressor NVFP4, Linear only; GDN/vision/lm-head/ norms/MTP kept BF16 (robbatt deckard-nvfp4 recipe + MTP). NEEDS a freed Blackwell GPU (~55 GB) + an llmcompressor env (run in a vLLM container: pip install llmcompressor on vllm/vllm-openai:v0.24.0).
- baseline: --calib-mode text --calib neuralmagic/calibration (AEON control)
- production: --calib-mode chat --calib <512-row mix> (brokkr/Dvalin) — rows rendered via apply_chat_template(enable_thinking=True) so the forward-pass sees the qwen3_coder tool-call XML = the seat's native activations.
serve (pantheon-27b-mtp-nvfp4 pattern): vllm --quantization compressed-tensors --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice.
P00 acceptance (brokkr): soong 9-tool k5 rig on the quant — must hold ~0.967 / perfect attach_tool. This is the authoritative #355 check, NOT KL (KL can pass while the structured-tool path regresses).

Gates

GPU window — both ana-ml2 Blackwell GPUs run ~full; NVFP4 is Blackwell-only (irv-ml1's Ampere can't). The ~30–60 min quant needs a brief off-peak window freeing a GPU. Operator's call.
Production calib — brokkr/Dvalin assembling the 512-row mix; the tool-call-XML slice (128 rows, 53 attach_tool) is ready. The AEON-baseline is fireable now.

Artifacts (on ana-ml2)

Heretic2 BF16 (target): /tank/aimodels/huggingface/hub/models--DavidAU--Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking
base Qwen3.6-27B (MTP source): /tank/aimodels/huggingface/hub/models--Qwen--Qwen3.6-27B (15 mtp.* tensors, shards 13+15)
AEON-baseline calib: neuralmagic/calibration (HF)
soong-tools-v0.3.13.json — live 9-tool schema (structure source-of-truth; calib uses the PREFIXED bifrost.soong-lab.* runtime names the seat emits)
extract_soong_tools.py — how that schema was pulled from the deployed backend

Serve target

Replaces the current llama.cpp GGUF char-rp-reasoning seat (~59.5 tok/s) once P00 passes. Deckard stays staged as rollback; the GGUF seat is the fallback until the NVFP4 seat is validated + cut over.

4.3 KiB Raw Blame History Unescape Escape