vh
982c319d9f
The fast char-rp-reasoning seat works: ~77 tok/s (vs GGUF ~59.5, base NVFP4 ~53), MTP draft-acceptance 32-40%, mean acceptance length 2.19. Same Heretic2/NEO-CODE model, NVFP4 + native qwen3_5_mtp spec-decode. Full end-to-end recipe + the four landmines in docs/runbooks/heretic2-nvfp4-mtp-seat.md: (1) load as AutoModelForImageTextToText not AutoModelForCausalLM (namespace/gibberish); (2) modelopt format not compressed-tensors (compressed-tensors MTP = 0% accept); (3) modelopt 0.45 <-> transformers 5.12.1 FusedMoE crash (guarded in quant_modelopt.py); (4) vLLM 0.24.0 does NOT propagate modelopt exclude_modules to the spec-decode draft model -> BF16 mtp head gets quantized -> shape crash; no checkpoint config fixes it (is_layer_skipped is exact-membership not glob) -> fix is a mounted sitecustomize that force-skips mtp.* in is_layer_skipped (upstream vLLM bug to report). Scripts: quant_modelopt.py (FusedMoE guard + single-shard export + multimodal load), finalize_modelopt_mtp.py (splice bf16 mtp), serve_modelopt_mtp.sh, run_quant_modelopt.sh, sitecustomize-mtp-workaround.py.
heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP)
Local NVFP4 quant of NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) with the
Qwen3.6 MTP head grafted back, for a ~2.5–4× faster vLLM/MTP
char-rp-reasoning seat (buys reasoning-budget headroom → better GM planning
inside soong's latency window). R36 fast-seat spike, 2026-07-14.
Fleet-first local NVFP4 quant — every other fleet NVFP4 model is pulled pre-quantized; Heretic2 has none published, so we quantize it.
2026-07-14 status — gibberish FIXED, format PIVOTED to modelopt for MTP
- Root cause of the
!!!!was the quant NAMESPACE, not calib/scheme:quant_nvfp4.pyloaded viaAutoModelForCausalLM→ text-onlyQwen3_5ForCausalLM→ flatmodel.layers.*keys, but vLLM 0.24 serves onlyQwen3_5ForConditionalGeneration, whose weight mapper needsmodel.language_model.*. Fixed by loading asAutoModelForImageTextToText(=Qwen3_5ForConditionalGeneration) → keys bornmodel.language_model.*+model.visual.*. NVFP4 now serves coherent. - But this (llm-compressor / compressed-tensors) format can't deliver the speed goal: base
NVFP4 ≈ 53 tok/s ≈ the GGUF seat's ~59.5 at batch-1 (no single-stream win), and MTP =
0% acceptance (vLLM's
Qwen3_5MTPdrafter won't load the bf16 mtp head off a compressed-tensors main model). The mtp tensors are identical to AEON's; the blocker is purely the main-model format. - → Working native MTP requires the MODELOPT format (what AEON uses, ~3.3/3 accept). Use
quant_modelopt.py(nvidia-modelopt PTQ; same graft + samesplice_mtp.py+ serve--quantization modelopt). AEON/tank/aimodels/qwen36-27b-aeon-nvfp4(vllm-aeon-rp) is the exact reference.quant_nvfp4.py(below) is kept for the coherent-but-MTP-inert compressed-tensors artifact and as the namespace-fix record.
Fire sequence
graft_mtp.py— graft the 15 base-Qwen3.6 MTP tensors into Heretic2 BF16. CPU-only, no GPU window. (Heretic2's finetune dropped the head; config declaresmtp_num_hidden_layers=1but ships 0mtp.*tensors — verified.)quant_nvfp4.py— llm-compressor NVFP4, Linear only; GDN/vision/lm-head/ norms/MTP kept BF16 (robbatt deckard-nvfp4 recipe + MTP). NEEDS a freed Blackwell GPU (~55 GB) + an llmcompressor env (run in a vLLM container:pip install llmcompressoronvllm/vllm-openai:v0.24.0).- baseline:
--calib-mode text --calib neuralmagic/calibration(AEON control) - production:
--calib-mode chat --calib <512-row mix>(brokkr/Dvalin) — rows rendered viaapply_chat_template(enable_thinking=True)so the forward-pass sees the qwen3_coder tool-call XML = the seat's native activations.
- baseline:
- serve (pantheon-27b-mtp-nvfp4 pattern):
vllm --quantization compressed-tensors --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice. - P00 acceptance (brokkr): soong 9-tool k5 rig on the quant — must hold
~0.967 / perfect
attach_tool. This is the authoritative #355 check, NOT KL (KL can pass while the structured-tool path regresses).
Gates
- GPU window — both ana-ml2 Blackwell GPUs run ~full; NVFP4 is Blackwell-only (irv-ml1's Ampere can't). The ~30–60 min quant needs a brief off-peak window freeing a GPU. Operator's call.
- Production calib — brokkr/Dvalin assembling the 512-row mix; the tool-call-XML
slice (128 rows, 53
attach_tool) is ready. The AEON-baseline is fireable now.
Artifacts (on ana-ml2)
- Heretic2 BF16 (target):
/tank/aimodels/huggingface/hub/models--DavidAU--Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking - base Qwen3.6-27B (MTP source):
/tank/aimodels/huggingface/hub/models--Qwen--Qwen3.6-27B(15 mtp.* tensors, shards 13+15) - AEON-baseline calib:
neuralmagic/calibration(HF) soong-tools-v0.3.13.json— live 9-tool schema (structure source-of-truth; calib uses the PREFIXEDbifrost.soong-lab.*runtime names the seat emits)extract_soong_tools.py— how that schema was pulled from the deployed backend
Serve target
Replaces the current llama.cpp GGUF char-rp-reasoning seat (~59.5 tok/s) once
P00 passes. Deckard stays staged as rollback; the GGUF seat is the fallback until
the NVFP4 seat is validated + cut over.