fix(heretic2-nvfp4): quant as ConditionalGeneration (namespace fix) + modelopt recipe for working MTP

Root-caused the NVFP4 gibberish to a quant-namespace bug: quant_nvfp4.py loaded
via AutoModelForCausalLM -> text-only Qwen3_5ForCausalLM -> flat model.layers.* keys,
but vLLM 0.24 serves only Qwen3_5ForConditionalGeneration (whose weight mapper needs
model.language_model.*). Fixed by loading as AutoModelForImageTextToText; NVFP4 now
serves coherent (validated greedy on ana-ml2 GPU0).

Base NVFP4 (compressed-tensors) measured ~53 tok/s (~= GGUF at batch-1, no single-stream
win) and its MTP is 0% acceptance (vLLM's Qwen3_5MTP drafter loads the bf16 mtp head only
off a modelopt main-model checkpoint). Added quant_modelopt.py (nvidia-modelopt PTQ,
matches AEON's NVFP4 W4A4 g16 + lm_head/linear_attn/visual exclusions) as the path to
working native MTP; graft + splice + serve otherwise unchanged.

persistent-memory.md

@@ -102,9 +102,25 @@ no longer deployed sidecars here. See Recent decisions.)
 
 ## Current state / in-flight
 
-_As of 2026-07-14 — ONE active task: finish the NVFP4+MTP fast char-rp-reasoning seat (the "nvfp4 quant chase"). Everything else this session LANDED: the char-rp-reasoning Deckard→NEO-CODE swap (#355 resolved), the Worldtree deploy-speed PR, and the soong-lab webhook fix — see Recent decisions + git; the old #355/deploy-speed detail below is kept as history (both DONE)._
+_As of 2026-07-14 — ONE active task: the NVFP4 fast char-rp-reasoning seat. GIBBERISH RESOLVED (quant-namespace bug — NVFP4 now serves coherent) and the speed premise DISPROVEN on the llm-compressor/compressed-tensors format (base NVFP4 ≈ GGUF at batch-1; MTP 0%-accept). PIVOTED (operator) to a **modelopt-format re-quant for working MTP** — scoped + de-risked, needs `quant_modelopt.py` + one more GPU0 window (see ★ section). Everything else this session LANDED: the char-rp-reasoning Deckard→NEO-CODE swap (#355 resolved), the Worldtree deploy-speed PR, and the soong-lab webhook fix — see Recent decisions + git; the old #355/deploy-speed detail below is kept as history (both DONE)._
 
-### ★ ACTIVE — NVFP4+MTP fast char-rp-reasoning seat (the quant chase)
+### ✅ RESOLVED (gibberish) → ⏭ PIVOT: modelopt-format re-quant for working MTP
+
+**2026-07-14 WINDOW OUTCOME (this session).** Ran the diagnostic ladder in one clean ~40-min GPU0 window.
+- **GIBBERISH ROOT CAUSE = quant NAMESPACE (found from config diffs + vLLM source, ZERO GPU time).** `quant_nvfp4.py` loaded via `AutoModelForCausalLM` → resolves qwen3_5 to the text-only `Qwen3_5ForCausalLM` → weight keys born flat `model.layers.*` (no vision). But vLLM 0.24 registers ONLY `Qwen3_5ForConditionalGeneration` (registry.py:566), whose `hf_to_vllm_mapper` (qwen3_vl.py:1692) remaps `model.language_model.*`→`language_model.model.*` and has NO rule for a bare `model.layers.` prefix → every transformer-layer weight fails to match → uninitialized → `!!!!`. The step-4 config-merge to ConditionalGeneration was a doomed patch over a wrong-namespace checkpoint. **FIX = load as `AutoModelForImageTextToText`** (resolves qwen3_5 → `Qwen3_5ForConditionalGeneration` → keys born `model.language_model.*` + `model.visual.*`, pantheon namespace). One-class swap; committed to `quant_nvfp4.py`.
+- **VALIDATED COHERENT.** Re-quant → `heretic2-nvfp4-cg` (arch ConditionalGeneration, nvfp4-pack-quantized, 1952 keys: 1618 language_model + 333 visual + 0 bare model.layers, 447-entry ignore = pantheon-shaped). Greedy `/v1/completions` = fluent ("...tavern in Baldur's Gate...", "Paris."). Spliced 15 BF16 mtp → `heretic2-nvfp4-cg-mtp` (1967 tensors, mtp key-set identical to pantheon+AEON).
+- **BUT THE SPEED PREMISE IS DISPROVEN on this (llm-compressor / compressed-tensors) format:** (1) base NVFP4 no-MTP ≈ **53 tok/s decode ≈ the GGUF seat's ~59.5 at batch-1** (NO single-stream win — this GDN-hybrid is bandwidth-bound at batch=1 with the BF16 linear_attn layers dominating; NVFP4's edge is concurrency, not batch-1; llama.cpp is highly optimized single-stream). (2) MTP spec-decode = **0% acceptance** — vLLM's Qwen3_5MTP drafter can't load the bf16 mtp weights off a compressed-tensors main model (`Parameter ... not found in params_dict`); pantheon is identical (its "working MTP" was never real acceleration). KNOWN limit: the bf16 MTP head only loads on the **MODELOPT** main-model format.
+- **PIVOT (operator 2026-07-14): pursue a modelopt-format re-quant for working MTP** — the ONLY path to the 2-4×.
+
+**⏭ MODELOPT PLAN (scoped + de-risked this session, ready for the next window):**
+- **REFERENCE = AEON** `/tank/aimodels/qwen36-27b-aeon-nvfp4` (served by `vllm-aeon-rp`, `/opt/docker/compose/qwen36-27b-aeon/compose.yaml`). Confirmed: arch `Qwen3_5ForConditionalGeneration`, `quant_method: modelopt` (config.json quantization_config + a separate `hf_quant_config.json`), 1967 tensors, **15 BF16 `mtp.*` tensors with byte-identical keys to our graft** (in `exclude_modules`), namespace `model.language_model.*`. Weights stored as `weight`+`weight_scale`+`input_scale` (modelopt) vs compressed-tensors packing — THAT format diff is why MTP loads on modelopt but not compressed-tensors (mtp tensors themselves are identical).
+- **THE PIPELINE IS UNCHANGED except the quant step:** reuse the graft (`heretic2-mtp-bf16`) → **quant with `nvidia-modelopt` instead of llm-compressor** → reuse `splice_mtp.py` (bf16 mtp) → serve `--quantization modelopt` + `--speculative-config qwen3_5_mtp` (AEON's `vllm-aeon-rp` is the exact serve reference).
+- **modelopt API de-risked:** `pip install nvidia-modelopt` → **v0.45.0 installs** in the vLLM container; `mtq.quantize(model, config, forward_loop)` + `NVFP4_DEFAULT_CFG` + `export_hf_checkpoint(model, export_dir=…)` all present (the AEON PTQ→HF-export path). Load as `AutoModelForImageTextToText`; exclude visual/linear_attn/mtp/lm_head/embed to match AEON's `exclude_modules`; 512-row chat calib as the `forward_loop`.
+- **RESIDUAL RISKS:** (1) modelopt 0.45 warns "not tested with transformers 5.12.1" → try `nvidia-modelopt[hf]` or a transformers pin; (2) modelopt PTQ tracing the GDN-hybrid arch for calibration (AEON proves feasible); (3) building the exclude/quant config to match AEON exactly. (4) needs another ~30-min GPU0 window (PTQ calibration runs on GPU).
+- **NEXT STEPS:** write `quant_modelopt.py` (adapt `quant_nvfp4.py`'s load+calib, swap the quant body to `mtq.quantize`+`export_hf_checkpoint`) → GPU0 window → serve modelopt+MTP → verify acceptance >0% → brokkr P00.
+- **Seats restored + healthy; char-rp-reasoning STAYS on the validated GGUF NEO-CODE seat** (~59.5 tok/s, R36-passing) until a modelopt-MTP seat proves a real win. GGUF is backstopped by worldtree's 300s wall-clock. Artifacts kept on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/`: `heretic2-nvfp4-cg` (coherent no-MTP), `heretic2-nvfp4-cg-mtp` (spliced, MTP-inert on compressed-tensors), `heretic2-mtp-nvfp4-prod` (original gibberish, keep for diff). Fixed `quant_nvfp4.py` staged at `ana-ml2:~/quant_nvfp4_cg.py`.
+
+### ★ (historical, SUPERSEDED by the above) NVFP4+MTP compressed-tensors recipe — the quant chase
 
 **GOAL:** NVFP4-quantize NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) + graft the Qwen3.6 MTP head → a ~2-4× faster **vLLM** char-rp-reasoning seat replacing the ~59.5 tok/s GGUF NEO-CODE seat (buys reasoning-budget headroom for soong's 45s client timeout). NVFP4 is **Blackwell-only** → ana-ml2 GPU0 is the ONLY place it runs (irv-ml1 is Ampere).
 
@@ -168,6 +184,7 @@ _As of 2026-07-14 — ONE active task: finish the NVFP4+MTP fast char-rp-reasoni
 
 ## Recent decisions
 
+- `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section.
 - `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`.
 - `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]].
 - `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528.
@@ -414,6 +431,7 @@ _142 older entries archived to archival-memory.md._
 
 ## Tried and abandoned
 
+- `[2026-07-14]` **NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept.** Measured base NVFP4 no-MTP ≈53 tok/s decode vs the GGUF NEO-CODE seat ~59.5 (llama.cpp wins single-stream; NVFP4's edge is concurrency, and this hybrid is bandwidth-bound at batch-1 with the BF16 linear_attn/GDN layers dominating). MTP spec-decode = 0% acceptance (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp weights off a compressed-tensors main model → `Parameter … not found in params_dict`, `Avg Draft acceptance rate: 0.0%`). Pantheon is identical — its "working NVFP4+MTP" was working *structure*, never real acceleration. Working native MTP needs the **modelopt** main-model format (AEON, ~3.3/3 accept). LESSON: don't expect a faster single-stream seat from an llm-compressor NVFP4 quant of this arch; the MTP multiplier is the whole point and it requires modelopt.
 - `[2026-07-14]` **NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent.** Chased 6 sequential serve-config fixes (entrypoint doubled `serve`, arch `ForCausalLM`→`ConditionalGeneration`, `--language-model-only`, mamba-cache/`max-num-seqs`) across a **2.5hr GPU window** (quoted 30-60 min) — only to find the served model gibbers (`!!!!`). LESSON: smoke a PLAIN `/v1/completions` coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the `re:mtp.*` ignore fix that turned out moot). Diagnostic ladder in Current state.
 - `[2026-07-14]` **MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained`** — the `Qwen3_5ForCausalLM` class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip.
 - `[2026-07-14]` **gitea "test-delivery 204" is NOT proof a webhook works** (204 = gitea *queuing*, not the listener receiving) — and a proxy test signing with the listener's OWN secret proves the listener, not gitea's real delivery. Both red herrings cost a round of the soong-lab webhook diagnosis. Diagnose from BOTH ends: sender (`docker logs gitea | grep webhook` → the `deny '<ip>'` line) AND an instrumented receiver.

services/heretic2-nvfp4-quant/README.md

@@ -8,6 +8,24 @@ inside soong's latency window). R36 fast-seat spike, 2026-07-14.
 Fleet-first **local** NVFP4 quant — every other fleet NVFP4 model is *pulled*
 pre-quantized; Heretic2 has none published, so we quantize it.
 
+## 2026-07-14 status — gibberish FIXED, format PIVOTED to modelopt for MTP
+
+- **Root cause of the `!!!!` was the quant NAMESPACE**, not calib/scheme: `quant_nvfp4.py`
+  loaded via `AutoModelForCausalLM` → text-only `Qwen3_5ForCausalLM` → flat `model.layers.*`
+  keys, but vLLM 0.24 serves only `Qwen3_5ForConditionalGeneration`, whose weight mapper needs
+  `model.language_model.*`. **Fixed** by loading as `AutoModelForImageTextToText`
+  (= `Qwen3_5ForConditionalGeneration`) → keys born `model.language_model.*` + `model.visual.*`.
+  NVFP4 now serves **coherent**.
+- **But this (llm-compressor / compressed-tensors) format can't deliver the speed goal:** base
+  NVFP4 ≈ 53 tok/s ≈ the GGUF seat's ~59.5 at batch-1 (no single-stream win), and **MTP =
+  0% acceptance** (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp head off a compressed-tensors
+  main model). The mtp tensors are identical to AEON's; the blocker is purely the main-model format.
+- **→ Working native MTP requires the MODELOPT format** (what AEON uses, ~3.3/3 accept). Use
+  **`quant_modelopt.py`** (nvidia-modelopt PTQ; same graft + same `splice_mtp.py` + serve
+  `--quantization modelopt`). AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` (`vllm-aeon-rp`) is the
+  exact reference. `quant_nvfp4.py` (below) is kept for the coherent-but-MTP-inert compressed-tensors
+  artifact and as the namespace-fix record.
+
 ## Fire sequence
 1. **`graft_mtp.py`** — graft the 15 base-Qwen3.6 MTP tensors into Heretic2 BF16.
    CPU-only, no GPU window. (Heretic2's finetune dropped the head; config declares

services/heretic2-nvfp4-quant/quant_modelopt.py

@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+"""NVFP4-quantize the MTP-grafted Heretic2 model via NVIDIA nvidia-modelopt (MODELOPT format).
+
+Sibling of quant_nvfp4.py (llm-compressor / compressed-tensors) but produces the **modelopt**
+NVFP4 format instead. Why it exists: the compressed-tensors path (quant_nvfp4.py) serves COHERENT
+but its MTP is 0% acceptance — vLLM's `Qwen3_5MTP` speculative-decode drafter only loads the bf16
+MTP head off a **modelopt** main-model checkpoint (the mtp tensors are byte-identical between the
+two formats; the difference is purely how the main model's quantized weights/scales are stored).
+So working native MTP (the ~2-4x goal) requires this format. (2026-07-14 modelopt pivot.)
+
+Reference = AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` (served by vllm-aeon-rp). Its
+hf_quant_config.json says: quant_algo NVFP4 (W4A4, group_size 16), targets Linear, exclude
+`lm_head` + `model.visual*` + every `linear_attn*` (the GDN). This script matches that EXACTLY.
+`embed_tokens` is an Embedding (not a Linear target) so it is never quantized — no need to exclude.
+
+Pipeline (unchanged except THIS quant step swaps llm-compressor -> modelopt):
+  graft_mtp.py  ->  quant_modelopt.py (this)  ->  splice_mtp.py (bf16 mtp)  ->  serve
+  serve: vllm --quantization modelopt --speculative-config
+         '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'  (AEON vllm-aeon-rp is the ref)
+
+The MTP head is NOT in the module tree at load (transformers builds no mtp module for either
+Qwen3_5 class), so the 15 bf16 mtp.* tensors are spliced back AFTER export — same as pantheon/AEON.
+
+Load class is AutoModelForImageTextToText (= Qwen3_5ForConditionalGeneration) so weight keys are
+born `model.language_model.*` + `model.visual.*` — the namespace vLLM's ConditionalGeneration
+loader expects. (The compressed-tensors-path gibberish was a text-only-namespace bug; same fix.)
+
+Run in a vLLM container on the freed GPU0 (nvidia-modelopt[hf] per the transformers-compat warning):
+  docker run --gpus '"device=0"' --ipc host -v /tank/aimodels:/tank/aimodels -v /home/lkraven:/lk \
+    --entrypoint bash vllm/vllm-openai:v0.24.0 -c \
+    "pip install -q 'nvidia-modelopt[hf]' tiktoken sentencepiece && python3 /lk/quant_modelopt.py \
+       --model /tank/aimodels/heretic2-nvfp4-work/heretic2-mtp-bf16 \
+       --calib-mode chat --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
+       --num-samples 512 --seqlen 8192 \
+       --out /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4"
+"""
+import argparse
+import copy
+import json
+import sys
+
+
+def load_calib_text(name, tokenizer, n, seqlen):
+    from datasets import load_dataset
+    ds = load_dataset(name, split="train").shuffle(seed=42).select(range(min(n, 100000)))
+    col = "text" if "text" in ds.column_names else ds.column_names[0]
+    return [tokenizer(x[col], truncation=True, max_length=seqlen) for x in ds.select(range(n))]
+
+
+def load_calib_chat(path, tokenizer, seqlen, n):
+    # Identical to quant_nvfp4.py: render each row via the Qwen3.6 chat template with thinking on
+    # so the forward-pass sees the seat's native tool-call XML activations. tool_call arguments may
+    # arrive as OpenAI wire-form JSON strings; the template does .items() on them -> parse to dict.
+    rows = [json.loads(l) for l in open(path) if l.strip()][:n]
+    out = []
+    for r in rows:
+        for m in r["messages"]:
+            for tc in (m.get("tool_calls") or []):
+                a = tc.get("function", {}).get("arguments")
+                if isinstance(a, str):
+                    tc["function"]["arguments"] = json.loads(a)
+        text = tokenizer.apply_chat_template(
+            r["messages"], tools=r.get("tools"),
+            tokenize=False, add_generation_prompt=False, enable_thinking=True,
+        )
+        out.append(tokenizer(text, truncation=True, max_length=seqlen))
+    return out
+
+
+def build_nvfp4_cfg(mtq):
+    """NVFP4 W4A4 (group_size 16) matching AEON's exclusions. NVFP4_DEFAULT_CFG already disables
+    lm_head + linear_attn.conv1d/in_proj_a/in_proj_b + mlp.gate; append the FULL linear_attn (GDN)
+    and the vision tower so only the standard attn/MLP Linears get NVFP4 (later entries override)."""
+    cfg = copy.deepcopy(mtq.NVFP4_DEFAULT_CFG)
+    cfg["quant_cfg"].append({"quantizer_name": "*linear_attn*", "enable": False})
+    cfg["quant_cfg"].append({"quantizer_name": "*visual*", "enable": False})
+    cfg["quant_cfg"].append({"quantizer_name": "*mtp*", "enable": False})  # moot (not in tree); belt+suspenders
+    return cfg
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--model", required=True)
+    ap.add_argument("--calib-mode", choices=["text", "chat"], required=True)
+    ap.add_argument("--calib", required=True)
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--num-samples", type=int, default=512)
+    ap.add_argument("--seqlen", type=int, default=8192)
+    args = ap.parse_args()
+
+    import torch
+    from transformers import AutoModelForImageTextToText, AutoTokenizer
+    import modelopt.torch.quantization as mtq
+    from modelopt.torch.export import export_hf_checkpoint
+
+    print(f"loading grafted model (multimodal ConditionalGeneration): {args.model}", flush=True)
+    model = AutoModelForImageTextToText.from_pretrained(
+        args.model, torch_dtype="auto", device_map="auto", trust_remote_code=True,
+    )
+    model.eval()
+    tok = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True)
+
+    print(f"building calibration ({args.calib_mode}, up to {args.num_samples} @ seq {args.seqlen})", flush=True)
+    if args.calib_mode == "text":
+        calib = load_calib_text(args.calib, tok, args.num_samples, args.seqlen)
+    else:
+        calib = load_calib_chat(args.calib, tok, args.seqlen, args.num_samples)
+    print(f"  {len(calib)} calibration rows", flush=True)
+
+    def forward_loop(m):
+        with torch.no_grad():
+            for i, row in enumerate(calib):
+                ids = torch.tensor([row["input_ids"]], device=m.device)
+                m(input_ids=ids)
+                if (i + 1) % 64 == 0:
+                    print(f"  calib {i + 1}/{len(calib)}", flush=True)
+
+    cfg = build_nvfp4_cfg(mtq)
+    print("running modelopt NVFP4 PTQ (W4A4 g16; lm_head/linear_attn/visual kept BF16)", flush=True)
+    mtq.quantize(model, cfg, forward_loop=forward_loop)
+
+    print(f"exporting modelopt HF checkpoint -> {args.out}", flush=True)
+    export_hf_checkpoint(model, export_dir=args.out)
+    tok.save_pretrained(args.out)
+    print("DONE. Next: splice_mtp.py <out> <graft>  then serve "
+          "--quantization modelopt --speculative-config "
+          "'{\"method\":\"qwen3_5_mtp\",\"num_speculative_tokens\":3}' "
+          "--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice", flush=True)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

services/heretic2-nvfp4-quant/quant_nvfp4.py

@@ -82,12 +82,24 @@ def main() -> int:
     ap.add_argument("--seqlen", type=int, default=8192)
     args = ap.parse_args()
 
-    from transformers import AutoModelForCausalLM, AutoTokenizer
+    from transformers import AutoModelForImageTextToText, AutoTokenizer
     from llmcompressor import oneshot
     from llmcompressor.modifiers.quantization import QuantizationModifier
 
     print(f"loading grafted model: {args.model}", flush=True)
-    model = AutoModelForCausalLM.from_pretrained(
+    # Load as the FULL multimodal Qwen3_5ForConditionalGeneration (NOT AutoModelForCausalLM).
+    # AutoModelForCausalLM resolves qwen3_5 -> Qwen3_5ForCausalLM (text-only), whose weight
+    # keys are flat `model.layers.*` with no vision tower. But vLLM 0.24 only registers
+    # Qwen3_5ForConditionalGeneration, and its hf_to_vllm_mapper expects the checkpoint keyed
+    # `model.language_model.layers.*` (+ `model.visual.*`) — a bare `model.layers.` prefix has
+    # NO mapping rule, so every transformer-layer weight fails to load -> uninitialized weights
+    # -> degenerate `!!!!` output. AutoModelForImageTextToText resolves qwen3_5 ->
+    # Qwen3_5ForConditionalGeneration, so keys are born `model.language_model.*` / `model.visual.*`
+    # matching the working pantheon-27b-mtp-nvfp4 reference. The vision tower loads in BF16 and is
+    # ignored by the quant (re:.*visual.*); calibration is text-only (no pixel_values needed).
+    # (R36 fast-seat namespace fix, 2026-07-14 — the config-merge in the prior recipe was a doomed
+    # patch over a checkpoint quantized in the wrong namespace.)
+    model = AutoModelForImageTextToText.from_pretrained(
         args.model, torch_dtype="auto", device_map="auto", trust_remote_code=True,
     )
     tok = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True)