diff --git a/persistent-memory.md b/persistent-memory.md index e001322..5e992d9 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -102,9 +102,25 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-07-14 — ONE active task: finish the NVFP4+MTP fast char-rp-reasoning seat (the "nvfp4 quant chase"). Everything else this session LANDED: the char-rp-reasoning Deckard→NEO-CODE swap (#355 resolved), the Worldtree deploy-speed PR, and the soong-lab webhook fix — see Recent decisions + git; the old #355/deploy-speed detail below is kept as history (both DONE)._ +_As of 2026-07-14 — ONE active task: the NVFP4 fast char-rp-reasoning seat. GIBBERISH RESOLVED (quant-namespace bug — NVFP4 now serves coherent) and the speed premise DISPROVEN on the llm-compressor/compressed-tensors format (base NVFP4 ≈ GGUF at batch-1; MTP 0%-accept). PIVOTED (operator) to a **modelopt-format re-quant for working MTP** — scoped + de-risked, needs `quant_modelopt.py` + one more GPU0 window (see ★ section). Everything else this session LANDED: the char-rp-reasoning Deckard→NEO-CODE swap (#355 resolved), the Worldtree deploy-speed PR, and the soong-lab webhook fix — see Recent decisions + git; the old #355/deploy-speed detail below is kept as history (both DONE)._ -### ★ ACTIVE — NVFP4+MTP fast char-rp-reasoning seat (the quant chase) +### ✅ RESOLVED (gibberish) → ⏭ PIVOT: modelopt-format re-quant for working MTP + +**2026-07-14 WINDOW OUTCOME (this session).** Ran the diagnostic ladder in one clean ~40-min GPU0 window. +- **GIBBERISH ROOT CAUSE = quant NAMESPACE (found from config diffs + vLLM source, ZERO GPU time).** `quant_nvfp4.py` loaded via `AutoModelForCausalLM` → resolves qwen3_5 to the text-only `Qwen3_5ForCausalLM` → weight keys born flat `model.layers.*` (no vision). But vLLM 0.24 registers ONLY `Qwen3_5ForConditionalGeneration` (registry.py:566), whose `hf_to_vllm_mapper` (qwen3_vl.py:1692) remaps `model.language_model.*`→`language_model.model.*` and has NO rule for a bare `model.layers.` prefix → every transformer-layer weight fails to match → uninitialized → `!!!!`. The step-4 config-merge to ConditionalGeneration was a doomed patch over a wrong-namespace checkpoint. **FIX = load as `AutoModelForImageTextToText`** (resolves qwen3_5 → `Qwen3_5ForConditionalGeneration` → keys born `model.language_model.*` + `model.visual.*`, pantheon namespace). One-class swap; committed to `quant_nvfp4.py`. +- **VALIDATED COHERENT.** Re-quant → `heretic2-nvfp4-cg` (arch ConditionalGeneration, nvfp4-pack-quantized, 1952 keys: 1618 language_model + 333 visual + 0 bare model.layers, 447-entry ignore = pantheon-shaped). Greedy `/v1/completions` = fluent ("...tavern in Baldur's Gate...", "Paris."). Spliced 15 BF16 mtp → `heretic2-nvfp4-cg-mtp` (1967 tensors, mtp key-set identical to pantheon+AEON). +- **BUT THE SPEED PREMISE IS DISPROVEN on this (llm-compressor / compressed-tensors) format:** (1) base NVFP4 no-MTP ≈ **53 tok/s decode ≈ the GGUF seat's ~59.5 at batch-1** (NO single-stream win — this GDN-hybrid is bandwidth-bound at batch=1 with the BF16 linear_attn layers dominating; NVFP4's edge is concurrency, not batch-1; llama.cpp is highly optimized single-stream). (2) MTP spec-decode = **0% acceptance** — vLLM's Qwen3_5MTP drafter can't load the bf16 mtp weights off a compressed-tensors main model (`Parameter ... not found in params_dict`); pantheon is identical (its "working MTP" was never real acceleration). KNOWN limit: the bf16 MTP head only loads on the **MODELOPT** main-model format. +- **PIVOT (operator 2026-07-14): pursue a modelopt-format re-quant for working MTP** — the ONLY path to the 2-4×. + +**⏭ MODELOPT PLAN (scoped + de-risked this session, ready for the next window):** +- **REFERENCE = AEON** `/tank/aimodels/qwen36-27b-aeon-nvfp4` (served by `vllm-aeon-rp`, `/opt/docker/compose/qwen36-27b-aeon/compose.yaml`). Confirmed: arch `Qwen3_5ForConditionalGeneration`, `quant_method: modelopt` (config.json quantization_config + a separate `hf_quant_config.json`), 1967 tensors, **15 BF16 `mtp.*` tensors with byte-identical keys to our graft** (in `exclude_modules`), namespace `model.language_model.*`. Weights stored as `weight`+`weight_scale`+`input_scale` (modelopt) vs compressed-tensors packing — THAT format diff is why MTP loads on modelopt but not compressed-tensors (mtp tensors themselves are identical). +- **THE PIPELINE IS UNCHANGED except the quant step:** reuse the graft (`heretic2-mtp-bf16`) → **quant with `nvidia-modelopt` instead of llm-compressor** → reuse `splice_mtp.py` (bf16 mtp) → serve `--quantization modelopt` + `--speculative-config qwen3_5_mtp` (AEON's `vllm-aeon-rp` is the exact serve reference). +- **modelopt API de-risked:** `pip install nvidia-modelopt` → **v0.45.0 installs** in the vLLM container; `mtq.quantize(model, config, forward_loop)` + `NVFP4_DEFAULT_CFG` + `export_hf_checkpoint(model, export_dir=…)` all present (the AEON PTQ→HF-export path). Load as `AutoModelForImageTextToText`; exclude visual/linear_attn/mtp/lm_head/embed to match AEON's `exclude_modules`; 512-row chat calib as the `forward_loop`. +- **RESIDUAL RISKS:** (1) modelopt 0.45 warns "not tested with transformers 5.12.1" → try `nvidia-modelopt[hf]` or a transformers pin; (2) modelopt PTQ tracing the GDN-hybrid arch for calibration (AEON proves feasible); (3) building the exclude/quant config to match AEON exactly. (4) needs another ~30-min GPU0 window (PTQ calibration runs on GPU). +- **NEXT STEPS:** write `quant_modelopt.py` (adapt `quant_nvfp4.py`'s load+calib, swap the quant body to `mtq.quantize`+`export_hf_checkpoint`) → GPU0 window → serve modelopt+MTP → verify acceptance >0% → brokkr P00. +- **Seats restored + healthy; char-rp-reasoning STAYS on the validated GGUF NEO-CODE seat** (~59.5 tok/s, R36-passing) until a modelopt-MTP seat proves a real win. GGUF is backstopped by worldtree's 300s wall-clock. Artifacts kept on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/`: `heretic2-nvfp4-cg` (coherent no-MTP), `heretic2-nvfp4-cg-mtp` (spliced, MTP-inert on compressed-tensors), `heretic2-mtp-nvfp4-prod` (original gibberish, keep for diff). Fixed `quant_nvfp4.py` staged at `ana-ml2:~/quant_nvfp4_cg.py`. + +### ★ (historical, SUPERSEDED by the above) NVFP4+MTP compressed-tensors recipe — the quant chase **GOAL:** NVFP4-quantize NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) + graft the Qwen3.6 MTP head → a ~2-4× faster **vLLM** char-rp-reasoning seat replacing the ~59.5 tok/s GGUF NEO-CODE seat (buys reasoning-budget headroom for soong's 45s client timeout). NVFP4 is **Blackwell-only** → ana-ml2 GPU0 is the ONLY place it runs (irv-ml1 is Ampere). @@ -168,6 +184,7 @@ _As of 2026-07-14 — ONE active task: finish the NVFP4+MTP fast char-rp-reasoni ## Recent decisions +- `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section. - `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`. - `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]]. - `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528. @@ -414,6 +431,7 @@ _142 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-07-14]` **NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept.** Measured base NVFP4 no-MTP ≈53 tok/s decode vs the GGUF NEO-CODE seat ~59.5 (llama.cpp wins single-stream; NVFP4's edge is concurrency, and this hybrid is bandwidth-bound at batch-1 with the BF16 linear_attn/GDN layers dominating). MTP spec-decode = 0% acceptance (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp weights off a compressed-tensors main model → `Parameter … not found in params_dict`, `Avg Draft acceptance rate: 0.0%`). Pantheon is identical — its "working NVFP4+MTP" was working *structure*, never real acceleration. Working native MTP needs the **modelopt** main-model format (AEON, ~3.3/3 accept). LESSON: don't expect a faster single-stream seat from an llm-compressor NVFP4 quant of this arch; the MTP multiplier is the whole point and it requires modelopt. - `[2026-07-14]` **NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent.** Chased 6 sequential serve-config fixes (entrypoint doubled `serve`, arch `ForCausalLM`→`ConditionalGeneration`, `--language-model-only`, mamba-cache/`max-num-seqs`) across a **2.5hr GPU window** (quoted 30-60 min) — only to find the served model gibbers (`!!!!`). LESSON: smoke a PLAIN `/v1/completions` coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the `re:mtp.*` ignore fix that turned out moot). Diagnostic ladder in Current state. - `[2026-07-14]` **MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained`** — the `Qwen3_5ForCausalLM` class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip. - `[2026-07-14]` **gitea "test-delivery 204" is NOT proof a webhook works** (204 = gitea *queuing*, not the listener receiving) — and a proxy test signing with the listener's OWN secret proves the listener, not gitea's real delivery. Both red herrings cost a round of the soong-lab webhook diagnosis. Diagnose from BOTH ends: sender (`docker logs gitea | grep webhook` → the `deny ''` line) AND an instrumented receiver. diff --git a/services/heretic2-nvfp4-quant/README.md b/services/heretic2-nvfp4-quant/README.md index d521f96..0dfcb3f 100644 --- a/services/heretic2-nvfp4-quant/README.md +++ b/services/heretic2-nvfp4-quant/README.md @@ -8,6 +8,24 @@ inside soong's latency window). R36 fast-seat spike, 2026-07-14. Fleet-first **local** NVFP4 quant — every other fleet NVFP4 model is *pulled* pre-quantized; Heretic2 has none published, so we quantize it. +## 2026-07-14 status — gibberish FIXED, format PIVOTED to modelopt for MTP + +- **Root cause of the `!!!!` was the quant NAMESPACE**, not calib/scheme: `quant_nvfp4.py` + loaded via `AutoModelForCausalLM` → text-only `Qwen3_5ForCausalLM` → flat `model.layers.*` + keys, but vLLM 0.24 serves only `Qwen3_5ForConditionalGeneration`, whose weight mapper needs + `model.language_model.*`. **Fixed** by loading as `AutoModelForImageTextToText` + (= `Qwen3_5ForConditionalGeneration`) → keys born `model.language_model.*` + `model.visual.*`. + NVFP4 now serves **coherent**. +- **But this (llm-compressor / compressed-tensors) format can't deliver the speed goal:** base + NVFP4 ≈ 53 tok/s ≈ the GGUF seat's ~59.5 at batch-1 (no single-stream win), and **MTP = + 0% acceptance** (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp head off a compressed-tensors + main model). The mtp tensors are identical to AEON's; the blocker is purely the main-model format. +- **→ Working native MTP requires the MODELOPT format** (what AEON uses, ~3.3/3 accept). Use + **`quant_modelopt.py`** (nvidia-modelopt PTQ; same graft + same `splice_mtp.py` + serve + `--quantization modelopt`). AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` (`vllm-aeon-rp`) is the + exact reference. `quant_nvfp4.py` (below) is kept for the coherent-but-MTP-inert compressed-tensors + artifact and as the namespace-fix record. + ## Fire sequence 1. **`graft_mtp.py`** — graft the 15 base-Qwen3.6 MTP tensors into Heretic2 BF16. CPU-only, no GPU window. (Heretic2's finetune dropped the head; config declares diff --git a/services/heretic2-nvfp4-quant/quant_modelopt.py b/services/heretic2-nvfp4-quant/quant_modelopt.py new file mode 100644 index 0000000..d880ff5 --- /dev/null +++ b/services/heretic2-nvfp4-quant/quant_modelopt.py @@ -0,0 +1,133 @@ +#!/usr/bin/env python3 +"""NVFP4-quantize the MTP-grafted Heretic2 model via NVIDIA nvidia-modelopt (MODELOPT format). + +Sibling of quant_nvfp4.py (llm-compressor / compressed-tensors) but produces the **modelopt** +NVFP4 format instead. Why it exists: the compressed-tensors path (quant_nvfp4.py) serves COHERENT +but its MTP is 0% acceptance — vLLM's `Qwen3_5MTP` speculative-decode drafter only loads the bf16 +MTP head off a **modelopt** main-model checkpoint (the mtp tensors are byte-identical between the +two formats; the difference is purely how the main model's quantized weights/scales are stored). +So working native MTP (the ~2-4x goal) requires this format. (2026-07-14 modelopt pivot.) + +Reference = AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` (served by vllm-aeon-rp). Its +hf_quant_config.json says: quant_algo NVFP4 (W4A4, group_size 16), targets Linear, exclude +`lm_head` + `model.visual*` + every `linear_attn*` (the GDN). This script matches that EXACTLY. +`embed_tokens` is an Embedding (not a Linear target) so it is never quantized — no need to exclude. + +Pipeline (unchanged except THIS quant step swaps llm-compressor -> modelopt): + graft_mtp.py -> quant_modelopt.py (this) -> splice_mtp.py (bf16 mtp) -> serve + serve: vllm --quantization modelopt --speculative-config + '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' (AEON vllm-aeon-rp is the ref) + +The MTP head is NOT in the module tree at load (transformers builds no mtp module for either +Qwen3_5 class), so the 15 bf16 mtp.* tensors are spliced back AFTER export — same as pantheon/AEON. + +Load class is AutoModelForImageTextToText (= Qwen3_5ForConditionalGeneration) so weight keys are +born `model.language_model.*` + `model.visual.*` — the namespace vLLM's ConditionalGeneration +loader expects. (The compressed-tensors-path gibberish was a text-only-namespace bug; same fix.) + +Run in a vLLM container on the freed GPU0 (nvidia-modelopt[hf] per the transformers-compat warning): + docker run --gpus '"device=0"' --ipc host -v /tank/aimodels:/tank/aimodels -v /home/lkraven:/lk \ + --entrypoint bash vllm/vllm-openai:v0.24.0 -c \ + "pip install -q 'nvidia-modelopt[hf]' tiktoken sentencepiece && python3 /lk/quant_modelopt.py \ + --model /tank/aimodels/heretic2-nvfp4-work/heretic2-mtp-bf16 \ + --calib-mode chat --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \ + --num-samples 512 --seqlen 8192 \ + --out /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4" +""" +import argparse +import copy +import json +import sys + + +def load_calib_text(name, tokenizer, n, seqlen): + from datasets import load_dataset + ds = load_dataset(name, split="train").shuffle(seed=42).select(range(min(n, 100000))) + col = "text" if "text" in ds.column_names else ds.column_names[0] + return [tokenizer(x[col], truncation=True, max_length=seqlen) for x in ds.select(range(n))] + + +def load_calib_chat(path, tokenizer, seqlen, n): + # Identical to quant_nvfp4.py: render each row via the Qwen3.6 chat template with thinking on + # so the forward-pass sees the seat's native tool-call XML activations. tool_call arguments may + # arrive as OpenAI wire-form JSON strings; the template does .items() on them -> parse to dict. + rows = [json.loads(l) for l in open(path) if l.strip()][:n] + out = [] + for r in rows: + for m in r["messages"]: + for tc in (m.get("tool_calls") or []): + a = tc.get("function", {}).get("arguments") + if isinstance(a, str): + tc["function"]["arguments"] = json.loads(a) + text = tokenizer.apply_chat_template( + r["messages"], tools=r.get("tools"), + tokenize=False, add_generation_prompt=False, enable_thinking=True, + ) + out.append(tokenizer(text, truncation=True, max_length=seqlen)) + return out + + +def build_nvfp4_cfg(mtq): + """NVFP4 W4A4 (group_size 16) matching AEON's exclusions. NVFP4_DEFAULT_CFG already disables + lm_head + linear_attn.conv1d/in_proj_a/in_proj_b + mlp.gate; append the FULL linear_attn (GDN) + and the vision tower so only the standard attn/MLP Linears get NVFP4 (later entries override).""" + cfg = copy.deepcopy(mtq.NVFP4_DEFAULT_CFG) + cfg["quant_cfg"].append({"quantizer_name": "*linear_attn*", "enable": False}) + cfg["quant_cfg"].append({"quantizer_name": "*visual*", "enable": False}) + cfg["quant_cfg"].append({"quantizer_name": "*mtp*", "enable": False}) # moot (not in tree); belt+suspenders + return cfg + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--model", required=True) + ap.add_argument("--calib-mode", choices=["text", "chat"], required=True) + ap.add_argument("--calib", required=True) + ap.add_argument("--out", required=True) + ap.add_argument("--num-samples", type=int, default=512) + ap.add_argument("--seqlen", type=int, default=8192) + args = ap.parse_args() + + import torch + from transformers import AutoModelForImageTextToText, AutoTokenizer + import modelopt.torch.quantization as mtq + from modelopt.torch.export import export_hf_checkpoint + + print(f"loading grafted model (multimodal ConditionalGeneration): {args.model}", flush=True) + model = AutoModelForImageTextToText.from_pretrained( + args.model, torch_dtype="auto", device_map="auto", trust_remote_code=True, + ) + model.eval() + tok = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True) + + print(f"building calibration ({args.calib_mode}, up to {args.num_samples} @ seq {args.seqlen})", flush=True) + if args.calib_mode == "text": + calib = load_calib_text(args.calib, tok, args.num_samples, args.seqlen) + else: + calib = load_calib_chat(args.calib, tok, args.seqlen, args.num_samples) + print(f" {len(calib)} calibration rows", flush=True) + + def forward_loop(m): + with torch.no_grad(): + for i, row in enumerate(calib): + ids = torch.tensor([row["input_ids"]], device=m.device) + m(input_ids=ids) + if (i + 1) % 64 == 0: + print(f" calib {i + 1}/{len(calib)}", flush=True) + + cfg = build_nvfp4_cfg(mtq) + print("running modelopt NVFP4 PTQ (W4A4 g16; lm_head/linear_attn/visual kept BF16)", flush=True) + mtq.quantize(model, cfg, forward_loop=forward_loop) + + print(f"exporting modelopt HF checkpoint -> {args.out}", flush=True) + export_hf_checkpoint(model, export_dir=args.out) + tok.save_pretrained(args.out) + print("DONE. Next: splice_mtp.py then serve " + "--quantization modelopt --speculative-config " + "'{\"method\":\"qwen3_5_mtp\",\"num_speculative_tokens\":3}' " + "--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice", flush=True) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/services/heretic2-nvfp4-quant/quant_nvfp4.py b/services/heretic2-nvfp4-quant/quant_nvfp4.py index bd9c6ac..f15e9cd 100644 --- a/services/heretic2-nvfp4-quant/quant_nvfp4.py +++ b/services/heretic2-nvfp4-quant/quant_nvfp4.py @@ -82,12 +82,24 @@ def main() -> int: ap.add_argument("--seqlen", type=int, default=8192) args = ap.parse_args() - from transformers import AutoModelForCausalLM, AutoTokenizer + from transformers import AutoModelForImageTextToText, AutoTokenizer from llmcompressor import oneshot from llmcompressor.modifiers.quantization import QuantizationModifier print(f"loading grafted model: {args.model}", flush=True) - model = AutoModelForCausalLM.from_pretrained( + # Load as the FULL multimodal Qwen3_5ForConditionalGeneration (NOT AutoModelForCausalLM). + # AutoModelForCausalLM resolves qwen3_5 -> Qwen3_5ForCausalLM (text-only), whose weight + # keys are flat `model.layers.*` with no vision tower. But vLLM 0.24 only registers + # Qwen3_5ForConditionalGeneration, and its hf_to_vllm_mapper expects the checkpoint keyed + # `model.language_model.layers.*` (+ `model.visual.*`) — a bare `model.layers.` prefix has + # NO mapping rule, so every transformer-layer weight fails to load -> uninitialized weights + # -> degenerate `!!!!` output. AutoModelForImageTextToText resolves qwen3_5 -> + # Qwen3_5ForConditionalGeneration, so keys are born `model.language_model.*` / `model.visual.*` + # matching the working pantheon-27b-mtp-nvfp4 reference. The vision tower loads in BF16 and is + # ignored by the quant (re:.*visual.*); calibration is text-only (no pixel_values needed). + # (R36 fast-seat namespace fix, 2026-07-14 — the config-merge in the prior recipe was a doomed + # patch over a checkpoint quantized in the wrong namespace.) + model = AutoModelForImageTextToText.from_pretrained( args.model, torch_dtype="auto", device_map="auto", trust_remote_code=True, ) tok = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True)