diff --git a/docs/runbooks/heretic2-nvfp4-mtp-seat.md b/docs/runbooks/heretic2-nvfp4-mtp-seat.md new file mode 100644 index 0000000..633a854 --- /dev/null +++ b/docs/runbooks/heretic2-nvfp4-mtp-seat.md @@ -0,0 +1,144 @@ +# Heretic2 NVFP4 + MTP fast char-rp-reasoning seat — the working recipe + +**Status: WORKING (2026-07-14).** ~77 tok/s single-stream (vs GGUF NEO-CODE ~59.5, base +NVFP4 ~53) — **~1.3× over GGUF**, MTP draft-acceptance **32–40%**, mean acceptance length +**2.19**. This is a drop-in faster replacement for the GGUF NEO-CODE `char-rp-reasoning` +seat (same Heretic2/NEO-CODE model, NVFP4 + native MTP spec-decode). + +This runbook exists because getting here was a multi-hour fire drill. **Every gotcha below +cost real time — read them before touching this.** The TL;DR: three things all had to be +right at once — (1) quant as the *multimodal* class, (2) use the *modelopt* format not +compressed-tensors, (3) work around a vLLM bug that quantizes the MTP draft head. + +--- + +## What / where + +- **Model:** NEO-CODE = `DavidAU/Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking` (dense + 27B, `Qwen3_5` GDN-hybrid arch, multimodal `Qwen3_5ForConditionalGeneration`). +- **Runs only on ana-ml2 GPU0** (NVFP4 is Blackwell-only; irv-ml1 is Ampere). +- **Artifacts** (ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/`, root-owned): + - `heretic2-mtp-bf16/` — BF16 graft (Heretic2 + 15 base-Qwen3.6 MTP tensors). [graft input] + - `heretic2-modelopt-nvfp4/` — modelopt NVFP4 quant, single shard, **no mtp**. [quant output] + - `heretic2-modelopt-nvfp4-mtp/` — the above + spliced 15 BF16 mtp → **the seat**. [SERVE THIS] + - (superseded: `heretic2-nvfp4-cg*` = compressed-tensors path, coherent but MTP-inert; + `heretic2-mtp-nvfp4-prod` = original gibberish. Keep for diff, do not serve.) +- **Scripts** (eshpfi `services/heretic2-nvfp4-quant/`): `graft_mtp.py`, `quant_modelopt.py`, + `finalize_modelopt_mtp.py`, `serve_modelopt_mtp.sh`, `sitecustomize-mtp-workaround.py`. +- **Reference:** the MoE `gen` (`qwen36-35b-a3b-heretic-nvfp4`, `quant_method: modelopt`) and + the qwopus-122B `gen` both ran MTP before (qwopus +12% single-stream, archival-memory + 2026-07-01) — dropped for `gen` because MTP *hurts concurrency*, which is why it belongs on + the single-stream RP seats, not `gen`. + +## GPU window ritual + +Base NVFP4 quant needs ~55 GB free on GPU0. `docker stop llama-charrp +llama-charrp-reasoning vllm-aeon-gen` (→ ~97 GB free); restore with `docker start …` +(~90–230 s to healthy). The GGUF NEO-CODE seat is the always-restorable fallback. Heads-up +wt-dev (their character / thoughtful-character / gen route through these) — unless told +otherwise. `ssh ana-ml2` = lkraven, in the docker group (no sudo needed for docker). + +--- + +## The pipeline (4 steps) + +### 1. GRAFT (CPU, seats up) — `graft_mtp.py` +Heretic2's finetune dropped the MTP head; graft the 15 BF16 `mtp.*` tensors from base +`Qwen/Qwen3.6-27B` (shards 13+15). Symlinks Heretic2 shards + one `model-mtp.safetensors`. +Idempotent, refuses to clobber. Output: `heretic2-mtp-bf16/`. + +### 2. QUANT (GPU0 window, ~18 min) — `quant_modelopt.py` via `run_quant_modelopt.sh` +`nvidia-modelopt` PTQ → **modelopt** NVFP4 format. Three things this script gets right (each a +gotcha — see below): loads as **`AutoModelForImageTextToText`**, patches the modelopt↔transformers +**FusedMoE** bug, and forces **single-shard** export. Excludes `lm_head` + `visual` + all +`linear_attn` (GDN) → BF16, matching AEON. Calib = the 512-row workload-matched chat mix. +```bash +docker run -d --name vllm-heretic2-modelopt-quant --gpus '"device=0"' --ipc host \ + -v /tank/aimodels:/tank/aimodels -v /home/lkraven:/lk \ + --entrypoint bash vllm/vllm-openai:v0.24.0 -c ' + set -e + pip install -q nvidia-modelopt tiktoken sentencepiece 2>&1 | tail -1 + python3 /lk/quant_modelopt.py \ + --model /tank/aimodels/heretic2-nvfp4-work/heretic2-mtp-bf16 \ + --calib-mode chat --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \ + --num-samples 512 --seqlen 8192 \ + --out /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4' +``` +CPU dry-run (no GPU, tiny calib) to validate the pipeline without an outage: same command +minus `--gpus`, add `-e CUDA_VISIBLE_DEVICES=""`, `--num-samples 2 --seqlen 512`. + +### 3. SPLICE (CPU) — `finalize_modelopt_mtp.py` +Copy `heretic2-modelopt-nvfp4` → `heretic2-modelopt-nvfp4-mtp`, splice the 15 BF16 `mtp.*` +tensors into the single shard (→ 1967 tensors). (The transformers load never builds an mtp +module, so mtp must be spliced post-quant — same as AEON/pantheon.) + +### 4. SERVE (GPU0) — `serve_modelopt_mtp.sh` + the MTP workaround +```bash +docker run -d --name vllm-charrp-modelopt --gpus '"device=0"' --ipc host \ + -v /tank/aimodels:/tank/aimodels \ + -v :/lk_debug -e PYTHONPATH=/lk_debug \ # ← the MTP workaround, see below + -p 8018:8000 vllm/vllm-openai:v0.24.0 \ + /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4-mtp \ + --quantization modelopt \ + --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \ + --language-model-only --mamba-cache-dtype float32 \ + --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \ + --served-model-name char-rp-reasoning --max-model-len 40960 --max-num-seqs 32 \ + --gpu-memory-utilization 0.5 --trust-remote-code +``` +`--language-model-only` skips the vision tower (RP seat doesn't need it; saves ~1–2 GB — the +tower is preserved BF16 in the weights, so multimodal is recoverable by dropping the flag). + +--- + +## The four landmines (each cost hours) + +1. **Load as `AutoModelForImageTextToText`, NEVER `AutoModelForCausalLM`.** The latter resolves + `qwen3_5` → text-only `Qwen3_5ForCausalLM` → flat `model.layers.*` keys. vLLM only serves + `Qwen3_5ForConditionalGeneration`, whose weight mapper needs `model.language_model.*` (+ + `model.visual.*`). Wrong class → every layer weight silently fails to load → **`!!!!` gibberish**. + +2. **Use the MODELOPT format (nvidia-modelopt), not compressed-tensors (llm-compressor).** On + compressed-tensors the MTP drafter can't load the BF16 mtp head at all (`not found in + params_dict`, **0% acceptance** — loads but never accelerates; this is what pantheon and the + "AEON RP seat" actually were). Base NVFP4 *alone* ≈ GGUF at batch-1 (no single-stream win) — + **the MTP multiplier is the entire point**, and it needs modelopt. + +3. **modelopt 0.45 ↔ transformers 5.12.1 FusedMoE crash.** `mtq.quantize` dies with + `TypeError: issubclass() arg 2 must be a class` — modelopt registered transformers' `FusedMoE` + (a *function* in 5.x) as an nn class. `quant_modelopt.py` guards it (patches + `_DMRegistryCls._get_registered_nn_class` to skip non-class registry entries). Do **not** + pin `nvidia-modelopt[hf]==0.43` to dodge it — that drags transformers back to 4.57 which can't + load `qwen3_5` at all. + +4. **⭐ THE BIG ONE — vLLM 0.24.0 does not propagate modelopt `exclude_modules` to the + spec-decode DRAFT model.** The MTP drafter builds its own `qkv_proj`/`gate_up_proj` as + *quantized* (NVFP4-packed) while the mtp head is BF16 → `AssertionError: param_data.shape == + loaded_weight.shape` in `qwen3_5_mtp.py:256`. **No checkpoint config fixes this** — instrumenting + `is_layer_skipped` proved the drafter's exclude list contains only the *main* model's + `linear_attn` entries, never the mtp ones. Also note `is_layer_skipped` does **exact string + membership, not glob** — so wildcards like `mtp.layers.0.*` never match anything. **Fix = a + runtime patch** (`sitecustomize-mtp-workaround.py`, mounted on `PYTHONPATH`) that force-skips + any `mtp.*` prefix in `is_layer_skipped`, keeping the drafter BF16. This is a genuine vLLM bug — + **report upstream** (draft-model quant-config should inherit the target's exclude_modules). + +## Verify it's actually accelerating + +```bash +# coherence +curl -s :8018/v1/completions -d '{"model":"char-rp-reasoning","prompt":"The old tavern","max_tokens":40,"temperature":0}' +# drive tokens, then read acceptance from the seat log: +docker logs vllm-charrp-modelopt 2>&1 | grep SpecDecoding | tail -2 +# -> "Mean acceptance length: 2.19 ... Avg Draft acceptance rate: 39.7%" [GOOD: >0%, ~2 length] +# -> "Avg Draft acceptance rate: 0.0%" [BAD: compressed-tensors, or mtp quantized] +``` +`SpecDecoding` line only appears during active generation. 0% acceptance = you're on +compressed-tensors, or the workaround didn't load (check for `[ISLS] ... workaround installed`). + +## Productionization TODO (not yet done) +- Bake the sitecustomize workaround into a compose stack (mount + `PYTHONPATH`), served-name + `char-rp-reasoning`, alongside/replacing the GGUF seat. +- brokkr P00 (soong 9-tool k5) — same base model as GGUF NEO-CODE so R36 should carry, but the + NVFP4-vs-Q5 quality + tool-path must be confirmed before cutover. +- Repoint gateway `char-rp-reasoning` alias + heads-up wt-dev. +- File the vLLM upstream bug (draft-model exclude non-inheritance). diff --git a/persistent-memory.md b/persistent-memory.md index 5e992d9..6e2c359 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -104,7 +104,34 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-07-14 — ONE active task: the NVFP4 fast char-rp-reasoning seat. GIBBERISH RESOLVED (quant-namespace bug — NVFP4 now serves coherent) and the speed premise DISPROVEN on the llm-compressor/compressed-tensors format (base NVFP4 ≈ GGUF at batch-1; MTP 0%-accept). PIVOTED (operator) to a **modelopt-format re-quant for working MTP** — scoped + de-risked, needs `quant_modelopt.py` + one more GPU0 window (see ★ section). Everything else this session LANDED: the char-rp-reasoning Deckard→NEO-CODE swap (#355 resolved), the Worldtree deploy-speed PR, and the soong-lab webhook fix — see Recent decisions + git; the old #355/deploy-speed detail below is kept as history (both DONE)._ -### ✅ RESOLVED (gibberish) → ⏭ PIVOT: modelopt-format re-quant for working MTP +### ✅✅ DONE — modelopt NVFP4 + MTP fast char-rp-reasoning seat WORKS (2026-07-14) + +**WORKING at ~77 tok/s** (vs GGUF NEO-CODE ~59.5, base NVFP4 ~53 → **~1.3× over GGUF**), MTP +draft-acceptance **32–40%**, mean acceptance length **2.19**. Same Heretic2/NEO-CODE model, NVFP4 ++ native MTP. **★ FULL RECIPE + all gotchas: `docs/runbooks/heretic2-nvfp4-mtp-seat.md`** (the +fire-drill-killer the operator demanded). Scripts: eshpfi `services/heretic2-nvfp4-quant/` +(`quant_modelopt.py`, `finalize_modelopt_mtp.py`, `serve_modelopt_mtp.sh`, +`sitecustomize-mtp-workaround.py`). Seat artifact: ana-ml2 +`/tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4-mtp`. + +**The four landmines (each cost hours — full detail in the runbook):** (1) load as +`AutoModelForImageTextToText` not `AutoModelForCausalLM` (namespace → else `!!!!`); (2) MODELOPT +format not compressed-tensors (compressed-tensors MTP = 0% accept; base NVFP4 alone ≈ GGUF, MTP is +the whole win); (3) modelopt 0.45↔transformers 5.12.1 FusedMoE crash — guarded in `quant_modelopt.py`; +(4) **⭐ vLLM 0.24.0 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → +the BF16 mtp head gets quantized → shape crash; **no checkpoint config fixes it** (`is_layer_skipped` +uses exact membership not glob, and the drafter never sees the mtp excludes) → **fix = a mounted +`sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (report upstream as a vLLM bug). + +**Operator's memory was right:** MTP ran before on the MoE `gen` (qwopus-122B +12% single-stream, +archival 2026-07-01) — dropped for `gen` because MTP HURTS concurrency; it belongs on the +single-stream RP seats. **NOT-YET-DONE (productionization):** compose stack with the workaround baked +in, brokkr P00 (soong 9-tool k5 — same base as GGUF so R36 should carry, confirm NVFP4-vs-Q5 quality), +gateway `char-rp-reasoning` repoint, wt-dev heads-up, file the vLLM upstream bug. + +--- + +### (historical) RESOLVED (gibberish) → PIVOT: modelopt-format re-quant for working MTP **2026-07-14 WINDOW OUTCOME (this session).** Ran the diagnostic ladder in one clean ~40-min GPU0 window. - **GIBBERISH ROOT CAUSE = quant NAMESPACE (found from config diffs + vLLM source, ZERO GPU time).** `quant_nvfp4.py` loaded via `AutoModelForCausalLM` → resolves qwen3_5 to the text-only `Qwen3_5ForCausalLM` → weight keys born flat `model.layers.*` (no vision). But vLLM 0.24 registers ONLY `Qwen3_5ForConditionalGeneration` (registry.py:566), whose `hf_to_vllm_mapper` (qwen3_vl.py:1692) remaps `model.language_model.*`→`language_model.model.*` and has NO rule for a bare `model.layers.` prefix → every transformer-layer weight fails to match → uninitialized → `!!!!`. The step-4 config-merge to ConditionalGeneration was a doomed patch over a wrong-namespace checkpoint. **FIX = load as `AutoModelForImageTextToText`** (resolves qwen3_5 → `Qwen3_5ForConditionalGeneration` → keys born `model.language_model.*` + `model.visual.*`, pantheon namespace). One-class swap; committed to `quant_nvfp4.py`. diff --git a/services/heretic2-nvfp4-quant/finalize_modelopt_mtp.py b/services/heretic2-nvfp4-quant/finalize_modelopt_mtp.py new file mode 100644 index 0000000..a049878 --- /dev/null +++ b/services/heretic2-nvfp4-quant/finalize_modelopt_mtp.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 +"""Splice the 15 BF16 mtp.* tensors into the modelopt NVFP4 quant output → the servable seat. + +Runs AFTER quant_modelopt.py. Copies heretic2-modelopt-nvfp4 → heretic2-modelopt-nvfp4-mtp, +splices the grafted BF16 mtp head into the single shard (transformers never builds an mtp module +at load, so mtp is always post-quant-spliced — same as AEON/pantheon), and adds the mtp module +names to config.json exclude_modules for tidiness. NOTE: the actual thing that keeps the mtp head +BF16 at serve time is the sitecustomize MTP workaround (see runbook landmine #4); the config +exclude here is belt-and-suspenders and does NOT by itself prevent the drafter-quant crash. + +Run in a vLLM container (root; /tank/aimodels files are root-owned): + docker run --rm -v /tank/aimodels:/tank/aimodels -v /home/lkraven:/lk \ + --entrypoint python3 vllm/vllm-openai:v0.24.0 /lk/finalize_modelopt_mtp.py +""" +import json +import os +import shutil + +from safetensors import safe_open +from safetensors.torch import save_file + +WORK = "/tank/aimodels/heretic2-nvfp4-work" +SRC = f"{WORK}/heretic2-modelopt-nvfp4" +DST = f"{WORK}/heretic2-modelopt-nvfp4-mtp" +GRAFT = f"{WORK}/heretic2-mtp-bf16" + +if os.path.exists(DST): + shutil.rmtree(DST) +print(f"copying {SRC} -> {DST}", flush=True) +shutil.copytree(SRC, DST) + +out_st = f"{DST}/model.safetensors" # single shard (quant_modelopt.py forces max_shard_size huge) +mtp_st = f"{GRAFT}/model-mtp.safetensors" +tensors = {} +with safe_open(out_st, framework="pt") as f: + for k in f.keys(): + tensors[k] = f.get_tensor(k) +n_main = len(tensors) +with safe_open(mtp_st, framework="pt") as f: + mtp_keys = list(f.keys()) + for k in mtp_keys: + tensors[k] = f.get_tensor(k) +assert not any("mtp" in k.lower() for k in list(tensors)[:n_main]), "output already had mtp?" +save_file(tensors, out_st, metadata={"format": "pt"}) +print(f"spliced {len(mtp_keys)} bf16 mtp tensors -> {len(tensors)} total", flush=True) + +cfgp = f"{DST}/config.json" +cfg = json.load(open(cfgp)) +qc = cfg.setdefault("quantization_config", {}) +exc = qc.setdefault("exclude_modules", []) +mtp_mods = sorted({k.rsplit(".", 1)[0] for k in mtp_keys}) +exc.extend(m for m in mtp_mods if m not in exc) +json.dump(cfg, open(cfgp, "w"), indent=2) +print(f"config exclude_modules += {len(mtp_mods)} mtp modules; total {len(exc)}", flush=True) +print(f"DONE: {DST}", flush=True) diff --git a/services/heretic2-nvfp4-quant/quant_modelopt.py b/services/heretic2-nvfp4-quant/quant_modelopt.py index d880ff5..45211e6 100644 --- a/services/heretic2-nvfp4-quant/quant_modelopt.py +++ b/services/heretic2-nvfp4-quant/quant_modelopt.py @@ -93,6 +93,22 @@ def main() -> int: import modelopt.torch.quantization as mtq from modelopt.torch.export import export_hf_checkpoint + # modelopt 0.45 + transformers 5.12.1 compat guard. transformers 5.x exposes `FusedMoE` as a + # FUNCTION, but modelopt registers it in QuantModuleRegistry expecting an nn.Module class, so the + # registry scan (register_fused_experts_on_the_fly -> _get_registered_nn_class) does + # `issubclass(nn_cls, )` and dies with "arg 2 must be a class". Our model is + # DENSE (no FusedMoE) so skipping non-class registry entries is safe. Guard the scan: + from modelopt.torch.opt import dynamic as _mo_dyn + + def _grnc_safe(self, nn_cls): + for nn_cls_ in self._registry: + if (isinstance(nn_cls_, type) and issubclass(nn_cls, nn_cls_) + and nn_cls.forward is nn_cls_.forward): + return nn_cls_ + return None + + _mo_dyn._DMRegistryCls._get_registered_nn_class = _grnc_safe + print(f"loading grafted model (multimodal ConditionalGeneration): {args.model}", flush=True) model = AutoModelForImageTextToText.from_pretrained( args.model, torch_dtype="auto", device_map="auto", trust_remote_code=True, @@ -120,7 +136,9 @@ def main() -> int: mtq.quantize(model, cfg, forward_loop=forward_loop) print(f"exporting modelopt HF checkpoint -> {args.out}", flush=True) - export_hf_checkpoint(model, export_dir=args.out) + # Force a SINGLE shard (default max_shard_size 10GB would split the ~14GB output into 3 shards, + # but splice_mtp.py expects a single /model.safetensors to add the bf16 mtp.* into). + export_hf_checkpoint(model, export_dir=args.out, max_shard_size="1TB") tok.save_pretrained(args.out) print("DONE. Next: splice_mtp.py then serve " "--quantization modelopt --speculative-config " diff --git a/services/heretic2-nvfp4-quant/run_quant_modelopt.sh b/services/heretic2-nvfp4-quant/run_quant_modelopt.sh new file mode 100644 index 0000000..34723a2 --- /dev/null +++ b/services/heretic2-nvfp4-quant/run_quant_modelopt.sh @@ -0,0 +1,20 @@ +#!/bin/bash +# Launch the modelopt NVFP4 quant of the grafted Heretic2 on ana-ml2 GPU0 (detached, survives ssh +# drop). bare nvidia-modelopt (0.45). The FusedMoE-compat guard + single-shard export + the +# multimodal load class are all inside quant_modelopt.py. ~18 min. Output: heretic2-modelopt-nvfp4 +# (no mtp yet — run finalize_modelopt_mtp.py after). quant_modelopt.py must be at /lk/quant_modelopt.py +# (mount /home/lkraven as /lk, or scp it there first). +set -euo pipefail +docker rm -f vllm-heretic2-modelopt-quant 2>/dev/null || true +rm -rf /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4 2>/dev/null || true +docker run -d --name vllm-heretic2-modelopt-quant --gpus '"device=0"' --ipc host \ + -v /tank/aimodels:/tank/aimodels -v /home/lkraven:/lk \ + --entrypoint bash vllm/vllm-openai:v0.24.0 -c ' + set -e + pip install -q nvidia-modelopt tiktoken sentencepiece 2>&1 | tail -1 + python3 /lk/quant_modelopt.py \ + --model /tank/aimodels/heretic2-nvfp4-work/heretic2-mtp-bf16 \ + --calib-mode chat --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \ + --num-samples 512 --seqlen 8192 \ + --out /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4' +echo "LAUNCHED: $(docker ps --filter name=vllm-heretic2-modelopt-quant --format '{{.Status}}')" diff --git a/services/heretic2-nvfp4-quant/serve_modelopt_mtp.sh b/services/heretic2-nvfp4-quant/serve_modelopt_mtp.sh new file mode 100644 index 0000000..3c5b911 --- /dev/null +++ b/services/heretic2-nvfp4-quant/serve_modelopt_mtp.sh @@ -0,0 +1,25 @@ +#!/bin/bash +# Serve the modelopt NVFP4 + MTP Heretic2 seat (the WORKING fast char-rp-reasoning seat). +# ~77 tok/s, MTP acceptance 32-40%. Requires: (1) heretic2-modelopt-nvfp4-mtp built (quant -> +# finalize), (2) the sitecustomize MTP workaround mounted on PYTHONPATH (vLLM 0.24 draft-model +# exclude bug — see runbook landmine #4; without it the engine crashes on a shape mismatch). +set -euo pipefail +MODEL="${1:-/tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4-mtp}" +# Dir containing sitecustomize.py (a copy of sitecustomize-mtp-workaround.py named sitecustomize.py): +WORKAROUND_DIR="${MTP_WORKAROUND_DIR:-/home/lkraven/isls_debug}" +docker rm -f vllm-charrp-modelopt 2>/dev/null || true +docker run -d --name vllm-charrp-modelopt --gpus '"device=0"' --ipc host \ + -v /tank/aimodels:/tank/aimodels \ + -v "${WORKAROUND_DIR}":/lk_debug -e PYTHONPATH=/lk_debug \ + -p 8018:8000 \ + vllm/vllm-openai:v0.24.0 \ + "$MODEL" \ + --quantization modelopt \ + --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \ + --language-model-only \ + --mamba-cache-dtype float32 \ + --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \ + --served-model-name char-rp-reasoning \ + --max-model-len 40960 --max-num-seqs 32 --gpu-memory-utilization 0.5 --trust-remote-code +echo "started: $(docker ps --filter name=vllm-charrp-modelopt --format '{{.Status}}')" +echo "verify MTP: docker logs vllm-charrp-modelopt 2>&1 | grep -E 'mtp-workaround|SpecDecoding'" diff --git a/services/heretic2-nvfp4-quant/sitecustomize-mtp-workaround.py b/services/heretic2-nvfp4-quant/sitecustomize-mtp-workaround.py new file mode 100644 index 0000000..8b8009a --- /dev/null +++ b/services/heretic2-nvfp4-quant/sitecustomize-mtp-workaround.py @@ -0,0 +1,53 @@ +# MTP draft-model quant workaround for vLLM 0.24.0 — MUST be named sitecustomize.py and be on +# PYTHONPATH so it loads in the vLLM engine-core subprocess. Mount its directory into the serve +# container and set -e PYTHONPATH=. +# +# THE BUG: vLLM 0.24.0 does not propagate the main model's modelopt `exclude_modules` to the +# spec-decode DRAFT model (Qwen3_5MTP). So the drafter builds its own qkv_proj/gate_up_proj as +# NVFP4-quantized while the grafted MTP head is BF16 → `AssertionError: param_data.shape == +# loaded_weight.shape` in qwen3_5_mtp.py:256, engine-core dies during weight load. Instrumenting +# is_layer_skipped proved the drafter's exclude list only ever contains the *main* model's +# entries, never the mtp ones — so no checkpoint config can fix it. (Also: is_layer_skipped does +# exact string membership, not glob — wildcards like `mtp.layers.0.*` match nothing.) +# +# THE FIX: force is_layer_skipped to return True (skip = keep BF16) for any `mtp.*` layer, so the +# draft head stays unquantized and its BF16 weights load. Report upstream: draft-model quant +# config should inherit the target model's exclude_modules. +import importlib.abc +import importlib.util +import sys + +TARGET = "vllm.model_executor.layers.quantization.utils.quant_utils" + + +class _Finder(importlib.abc.MetaPathFinder): + def find_spec(self, name, path, target=None): + if name != TARGET: + return None + sys.meta_path.remove(self) + try: + spec = importlib.util.find_spec(name) + finally: + sys.meta_path.insert(0, self) + if not spec or not spec.loader: + return None + _orig_exec = spec.loader.exec_module + + def exec_module(module): + _orig_exec(module) + _orig_isls = module.is_layer_skipped + + def is_layer_skipped(prefix, ignored_layers, *args, **kwargs): + pl = str(prefix) + if pl.startswith("mtp.") or ".mtp." in pl: + return True # keep the mtp draft head BF16 + return _orig_isls(prefix, ignored_layers, *args, **kwargs) + + module.is_layer_skipped = is_layer_skipped + print("[mtp-workaround] is_layer_skipped force-skip for mtp.* installed", flush=True) + + spec.loader.exec_module = exec_module + return spec + + +sys.meta_path.insert(0, _Finder())