vh
aca45393c2
Root-caused the NVFP4 gibberish to a quant-namespace bug: quant_nvfp4.py loaded via AutoModelForCausalLM -> text-only Qwen3_5ForCausalLM -> flat model.layers.* keys, but vLLM 0.24 serves only Qwen3_5ForConditionalGeneration (whose weight mapper needs model.language_model.*). Fixed by loading as AutoModelForImageTextToText; NVFP4 now serves coherent (validated greedy on ana-ml2 GPU0). Base NVFP4 (compressed-tensors) measured ~53 tok/s (~= GGUF at batch-1, no single-stream win) and its MTP is 0% acceptance (vLLM's Qwen3_5MTP drafter loads the bf16 mtp head only off a modelopt main-model checkpoint). Added quant_modelopt.py (nvidia-modelopt PTQ, matches AEON's NVFP4 W4A4 g16 + lm_head/linear_attn/visual exclusions) as the path to working native MTP; graft + splice + serve otherwise unchanged.
67 lines
4.3 KiB
Markdown
67 lines
4.3 KiB
Markdown
# heretic2-nvfp4-quant — fast char-rp-reasoning seat (NVFP4 + MTP)
|
||
|
||
Local NVFP4 quant of **NEO-CODE = Heretic2-Thinking** (Qwen3.6-27B) with the
|
||
Qwen3.6 **MTP head grafted back**, for a ~2.5–4× faster vLLM/MTP
|
||
`char-rp-reasoning` seat (buys reasoning-budget headroom → better GM planning
|
||
inside soong's latency window). R36 fast-seat spike, 2026-07-14.
|
||
|
||
Fleet-first **local** NVFP4 quant — every other fleet NVFP4 model is *pulled*
|
||
pre-quantized; Heretic2 has none published, so we quantize it.
|
||
|
||
## 2026-07-14 status — gibberish FIXED, format PIVOTED to modelopt for MTP
|
||
|
||
- **Root cause of the `!!!!` was the quant NAMESPACE**, not calib/scheme: `quant_nvfp4.py`
|
||
loaded via `AutoModelForCausalLM` → text-only `Qwen3_5ForCausalLM` → flat `model.layers.*`
|
||
keys, but vLLM 0.24 serves only `Qwen3_5ForConditionalGeneration`, whose weight mapper needs
|
||
`model.language_model.*`. **Fixed** by loading as `AutoModelForImageTextToText`
|
||
(= `Qwen3_5ForConditionalGeneration`) → keys born `model.language_model.*` + `model.visual.*`.
|
||
NVFP4 now serves **coherent**.
|
||
- **But this (llm-compressor / compressed-tensors) format can't deliver the speed goal:** base
|
||
NVFP4 ≈ 53 tok/s ≈ the GGUF seat's ~59.5 at batch-1 (no single-stream win), and **MTP =
|
||
0% acceptance** (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp head off a compressed-tensors
|
||
main model). The mtp tensors are identical to AEON's; the blocker is purely the main-model format.
|
||
- **→ Working native MTP requires the MODELOPT format** (what AEON uses, ~3.3/3 accept). Use
|
||
**`quant_modelopt.py`** (nvidia-modelopt PTQ; same graft + same `splice_mtp.py` + serve
|
||
`--quantization modelopt`). AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` (`vllm-aeon-rp`) is the
|
||
exact reference. `quant_nvfp4.py` (below) is kept for the coherent-but-MTP-inert compressed-tensors
|
||
artifact and as the namespace-fix record.
|
||
|
||
## Fire sequence
|
||
1. **`graft_mtp.py`** — graft the 15 base-Qwen3.6 MTP tensors into Heretic2 BF16.
|
||
CPU-only, no GPU window. (Heretic2's finetune dropped the head; config declares
|
||
`mtp_num_hidden_layers=1` but ships 0 `mtp.*` tensors — verified.)
|
||
2. **`quant_nvfp4.py`** — llm-compressor NVFP4, Linear only; **GDN/vision/lm-head/
|
||
norms/MTP kept BF16** (robbatt deckard-nvfp4 recipe + MTP). NEEDS a freed
|
||
Blackwell GPU (~55 GB) + an llmcompressor env (run in a vLLM container:
|
||
`pip install llmcompressor` on `vllm/vllm-openai:v0.24.0`).
|
||
- baseline: `--calib-mode text --calib neuralmagic/calibration` (AEON control)
|
||
- production: `--calib-mode chat --calib <512-row mix>` (brokkr/Dvalin) — rows
|
||
rendered via `apply_chat_template(enable_thinking=True)` so the forward-pass
|
||
sees the qwen3_coder tool-call XML = the seat's native activations.
|
||
3. **serve** (pantheon-27b-mtp-nvfp4 pattern): `vllm --quantization compressed-tensors
|
||
--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
|
||
--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice`.
|
||
4. **P00 acceptance** (brokkr): soong 9-tool k5 rig on the quant — must hold
|
||
~0.967 / perfect `attach_tool`. This is the **authoritative #355 check, NOT KL**
|
||
(KL can pass while the structured-tool path regresses).
|
||
|
||
## Gates
|
||
- **GPU window** — both ana-ml2 Blackwell GPUs run ~full; NVFP4 is Blackwell-only
|
||
(irv-ml1's Ampere can't). The ~30–60 min quant needs a brief off-peak window
|
||
freeing a GPU. *Operator's call.*
|
||
- **Production calib** — brokkr/Dvalin assembling the 512-row mix; the tool-call-XML
|
||
slice (128 rows, 53 `attach_tool`) is ready. The AEON-baseline is fireable now.
|
||
|
||
## Artifacts (on ana-ml2)
|
||
- Heretic2 BF16 (target): `/tank/aimodels/huggingface/hub/models--DavidAU--Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking`
|
||
- base Qwen3.6-27B (MTP source): `/tank/aimodels/huggingface/hub/models--Qwen--Qwen3.6-27B` (15 mtp.* tensors, shards 13+15)
|
||
- AEON-baseline calib: `neuralmagic/calibration` (HF)
|
||
- `soong-tools-v0.3.13.json` — live 9-tool schema (structure source-of-truth;
|
||
**calib uses the PREFIXED `bifrost.soong-lab.*` runtime names** the seat emits)
|
||
- `extract_soong_tools.py` — how that schema was pulled from the deployed backend
|
||
|
||
## Serve target
|
||
Replaces the current llama.cpp GGUF `char-rp-reasoning` seat (~59.5 tok/s) once
|
||
P00 passes. Deckard stays staged as rollback; the GGUF seat is the fallback until
|
||
the NVFP4 seat is validated + cut over.
|