fix(heretic2-nvfp4): quant as ConditionalGeneration (namespace fix) + modelopt recipe for working MTP

Root-caused the NVFP4 gibberish to a quant-namespace bug: quant_nvfp4.py loaded
via AutoModelForCausalLM -> text-only Qwen3_5ForCausalLM -> flat model.layers.* keys,
but vLLM 0.24 serves only Qwen3_5ForConditionalGeneration (whose weight mapper needs
model.language_model.*). Fixed by loading as AutoModelForImageTextToText; NVFP4 now
serves coherent (validated greedy on ana-ml2 GPU0).

Base NVFP4 (compressed-tensors) measured ~53 tok/s (~= GGUF at batch-1, no single-stream
win) and its MTP is 0% acceptance (vLLM's Qwen3_5MTP drafter loads the bf16 mtp head only
off a modelopt main-model checkpoint). Added quant_modelopt.py (nvidia-modelopt PTQ,
matches AEON's NVFP4 W4A4 g16 + lm_head/linear_attn/visual exclusions) as the path to
working native MTP; graft + splice + serve otherwise unchanged.

persistent-memory.md

@@ -102,9 +102,25 @@ no longer deployed sidecars here. See Recent decisions.)
 
 ## Current state / in-flight
 
-_As of 2026-07-14 — ONE active task: finish the NVFP4+MTP fast char-rp-reasoning seat (the "nvfp4 quant chase"). Everything else this session LANDED: the char-rp-reasoning Deckard→NEO-CODE swap (#355 resolved), the Worldtree deploy-speed PR, and the soong-lab webhook fix — see Recent decisions + git; the old #355/deploy-speed detail below is kept as history (both DONE)._
+_As of 2026-07-14 — ONE active task: the NVFP4 fast char-rp-reasoning seat. GIBBERISH RESOLVED (quant-namespace bug — NVFP4 now serves coherent) and the speed premise DISPROVEN on the llm-compressor/compressed-tensors format (base NVFP4 ≈ GGUF at batch-1; MTP 0%-accept). PIVOTED (operator) to a **modelopt-format re-quant for working MTP** — scoped + de-risked, needs `quant_modelopt.py` + one more GPU0 window (see ★ section). Everything else this session LANDED: the char-rp-reasoning Deckard→NEO-CODE swap (#355 resolved), the Worldtree deploy-speed PR, and the soong-lab webhook fix — see Recent decisions + git; the old #355/deploy-speed detail below is kept as history (both DONE)._
 
-### ★ ACTIVE — NVFP4+MTP fast char-rp-reasoning seat (the quant chase)
+### ✅ RESOLVED (gibberish) → ⏭ PIVOT: modelopt-format re-quant for working MTP
+
+**2026-07-14 WINDOW OUTCOME (this session).** Ran the diagnostic ladder in one clean ~40-min GPU0 window.
+- **GIBBERISH ROOT CAUSE = quant NAMESPACE (found from config diffs + vLLM source, ZERO GPU time).** `quant_nvfp4.py` loaded via `AutoModelForCausalLM` → resolves qwen3_5 to the text-only `Qwen3_5ForCausalLM` → weight keys born flat `model.layers.*` (no vision). But vLLM 0.24 registers ONLY `Qwen3_5ForConditionalGeneration` (registry.py:566), whose `hf_to_vllm_mapper` (qwen3_vl.py:1692) remaps `model.language_model.*`→`language_model.model.*` and has NO rule for a bare `model.layers.` prefix → every transformer-layer weight fails to match → uninitialized → `!!!!`. The step-4 config-merge to ConditionalGeneration was a doomed patch over a wrong-namespace checkpoint. **FIX = load as `AutoModelForImageTextToText`** (resolves qwen3_5 → `Qwen3_5ForConditionalGeneration` → keys born `model.language_model.*` + `model.visual.*`, pantheon namespace). One-class swap; committed to `quant_nvfp4.py`.
+- **VALIDATED COHERENT.** Re-quant → `heretic2-nvfp4-cg` (arch ConditionalGeneration, nvfp4-pack-quantized, 1952 keys: 1618 language_model + 333 visual + 0 bare model.layers, 447-entry ignore = pantheon-shaped). Greedy `/v1/completions` = fluent ("...tavern in Baldur's Gate...", "Paris."). Spliced 15 BF16 mtp → `heretic2-nvfp4-cg-mtp` (1967 tensors, mtp key-set identical to pantheon+AEON).
+- **BUT THE SPEED PREMISE IS DISPROVEN on this (llm-compressor / compressed-tensors) format:** (1) base NVFP4 no-MTP ≈ **53 tok/s decode ≈ the GGUF seat's ~59.5 at batch-1** (NO single-stream win — this GDN-hybrid is bandwidth-bound at batch=1 with the BF16 linear_attn layers dominating; NVFP4's edge is concurrency, not batch-1; llama.cpp is highly optimized single-stream). (2) MTP spec-decode = **0% acceptance** — vLLM's Qwen3_5MTP drafter can't load the bf16 mtp weights off a compressed-tensors main model (`Parameter ... not found in params_dict`); pantheon is identical (its "working MTP" was never real acceleration). KNOWN limit: the bf16 MTP head only loads on the **MODELOPT** main-model format.
+- **PIVOT (operator 2026-07-14): pursue a modelopt-format re-quant for working MTP** — the ONLY path to the 2-4×.
+
+**⏭ MODELOPT PLAN (scoped + de-risked this session, ready for the next window):**
+- **REFERENCE = AEON** `/tank/aimodels/qwen36-27b-aeon-nvfp4` (served by `vllm-aeon-rp`, `/opt/docker/compose/qwen36-27b-aeon/compose.yaml`). Confirmed: arch `Qwen3_5ForConditionalGeneration`, `quant_method: modelopt` (config.json quantization_config + a separate `hf_quant_config.json`), 1967 tensors, **15 BF16 `mtp.*` tensors with byte-identical keys to our graft** (in `exclude_modules`), namespace `model.language_model.*`. Weights stored as `weight`+`weight_scale`+`input_scale` (modelopt) vs compressed-tensors packing — THAT format diff is why MTP loads on modelopt but not compressed-tensors (mtp tensors themselves are identical).
+- **THE PIPELINE IS UNCHANGED except the quant step:** reuse the graft (`heretic2-mtp-bf16`) → **quant with `nvidia-modelopt` instead of llm-compressor** → reuse `splice_mtp.py` (bf16 mtp) → serve `--quantization modelopt` + `--speculative-config qwen3_5_mtp` (AEON's `vllm-aeon-rp` is the exact serve reference).
+- **modelopt API de-risked:** `pip install nvidia-modelopt` → **v0.45.0 installs** in the vLLM container; `mtq.quantize(model, config, forward_loop)` + `NVFP4_DEFAULT_CFG` + `export_hf_checkpoint(model, export_dir=…)` all present (the AEON PTQ→HF-export path). Load as `AutoModelForImageTextToText`; exclude visual/linear_attn/mtp/lm_head/embed to match AEON's `exclude_modules`; 512-row chat calib as the `forward_loop`.
+- **RESIDUAL RISKS:** (1) modelopt 0.45 warns "not tested with transformers 5.12.1" → try `nvidia-modelopt[hf]` or a transformers pin; (2) modelopt PTQ tracing the GDN-hybrid arch for calibration (AEON proves feasible); (3) building the exclude/quant config to match AEON exactly. (4) needs another ~30-min GPU0 window (PTQ calibration runs on GPU).
+- **NEXT STEPS:** write `quant_modelopt.py` (adapt `quant_nvfp4.py`'s load+calib, swap the quant body to `mtq.quantize`+`export_hf_checkpoint`) → GPU0 window → serve modelopt+MTP → verify acceptance >0% → brokkr P00.
+- **Seats restored + healthy; char-rp-reasoning STAYS on the validated GGUF NEO-CODE seat** (~59.5 tok/s, R36-passing) until a modelopt-MTP seat proves a real win. GGUF is backstopped by worldtree's 300s wall-clock. Artifacts kept on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/`: `heretic2-nvfp4-cg` (coherent no-MTP), `heretic2-nvfp4-cg-mtp` (spliced, MTP-inert on compressed-tensors), `heretic2-mtp-nvfp4-prod` (original gibberish, keep for diff). Fixed `quant_nvfp4.py` staged at `ana-ml2:~/quant_nvfp4_cg.py`.
+
+### ★ (historical, SUPERSEDED by the above) NVFP4+MTP compressed-tensors recipe — the quant chase
 
 **GOAL:** NVFP4-quantize NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) + graft the Qwen3.6 MTP head → a ~2-4× faster **vLLM** char-rp-reasoning seat replacing the ~59.5 tok/s GGUF NEO-CODE seat (buys reasoning-budget headroom for soong's 45s client timeout). NVFP4 is **Blackwell-only** → ana-ml2 GPU0 is the ONLY place it runs (irv-ml1 is Ampere).
 
@@ -168,6 +184,7 @@ _As of 2026-07-14 — ONE active task: finish the NVFP4+MTP fast char-rp-reasoni
 
 ## Recent decisions
 
+- `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section.
 - `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`.
 - `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]].
 - `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528.
@@ -414,6 +431,7 @@ _142 older entries archived to archival-memory.md._
 
 ## Tried and abandoned
 
+- `[2026-07-14]` **NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept.** Measured base NVFP4 no-MTP ≈53 tok/s decode vs the GGUF NEO-CODE seat ~59.5 (llama.cpp wins single-stream; NVFP4's edge is concurrency, and this hybrid is bandwidth-bound at batch-1 with the BF16 linear_attn/GDN layers dominating). MTP spec-decode = 0% acceptance (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp weights off a compressed-tensors main model → `Parameter … not found in params_dict`, `Avg Draft acceptance rate: 0.0%`). Pantheon is identical — its "working NVFP4+MTP" was working *structure*, never real acceleration. Working native MTP needs the **modelopt** main-model format (AEON, ~3.3/3 accept). LESSON: don't expect a faster single-stream seat from an llm-compressor NVFP4 quant of this arch; the MTP multiplier is the whole point and it requires modelopt.
 - `[2026-07-14]` **NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent.** Chased 6 sequential serve-config fixes (entrypoint doubled `serve`, arch `ForCausalLM`→`ConditionalGeneration`, `--language-model-only`, mamba-cache/`max-num-seqs`) across a **2.5hr GPU window** (quoted 30-60 min) — only to find the served model gibbers (`!!!!`). LESSON: smoke a PLAIN `/v1/completions` coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the `re:mtp.*` ignore fix that turned out moot). Diagnostic ladder in Current state.
 - `[2026-07-14]` **MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained`** — the `Qwen3_5ForCausalLM` class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip.
 - `[2026-07-14]` **gitea "test-delivery 204" is NOT proof a webhook works** (204 = gitea *queuing*, not the listener receiving) — and a proxy test signing with the listener's OWN secret proves the listener, not gitea's real delivery. Both red herrings cost a round of the soong-lab webhook diagnosis. Diagnose from BOTH ends: sender (`docker logs gitea | grep webhook` → the `deny '<ip>'` line) AND an instrumented receiver.