feat(heretic2-nvfp4): WORKING modelopt NVFP4+MTP seat + full recipe runbook
The fast char-rp-reasoning seat works: ~77 tok/s (vs GGUF ~59.5, base NVFP4 ~53), MTP draft-acceptance 32-40%, mean acceptance length 2.19. Same Heretic2/NEO-CODE model, NVFP4 + native qwen3_5_mtp spec-decode. Full end-to-end recipe + the four landmines in docs/runbooks/heretic2-nvfp4-mtp-seat.md: (1) load as AutoModelForImageTextToText not AutoModelForCausalLM (namespace/gibberish); (2) modelopt format not compressed-tensors (compressed-tensors MTP = 0% accept); (3) modelopt 0.45 <-> transformers 5.12.1 FusedMoE crash (guarded in quant_modelopt.py); (4) vLLM 0.24.0 does NOT propagate modelopt exclude_modules to the spec-decode draft model -> BF16 mtp head gets quantized -> shape crash; no checkpoint config fixes it (is_layer_skipped is exact-membership not glob) -> fix is a mounted sitecustomize that force-skips mtp.* in is_layer_skipped (upstream vLLM bug to report). Scripts: quant_modelopt.py (FusedMoE guard + single-shard export + multimodal load), finalize_modelopt_mtp.py (splice bf16 mtp), serve_modelopt_mtp.sh, run_quant_modelopt.sh, sitecustomize-mtp-workaround.py.
This commit is contained in:
@@ -0,0 +1,144 @@
|
||||
# Heretic2 NVFP4 + MTP fast char-rp-reasoning seat — the working recipe
|
||||
|
||||
**Status: WORKING (2026-07-14).** ~77 tok/s single-stream (vs GGUF NEO-CODE ~59.5, base
|
||||
NVFP4 ~53) — **~1.3× over GGUF**, MTP draft-acceptance **32–40%**, mean acceptance length
|
||||
**2.19**. This is a drop-in faster replacement for the GGUF NEO-CODE `char-rp-reasoning`
|
||||
seat (same Heretic2/NEO-CODE model, NVFP4 + native MTP spec-decode).
|
||||
|
||||
This runbook exists because getting here was a multi-hour fire drill. **Every gotcha below
|
||||
cost real time — read them before touching this.** The TL;DR: three things all had to be
|
||||
right at once — (1) quant as the *multimodal* class, (2) use the *modelopt* format not
|
||||
compressed-tensors, (3) work around a vLLM bug that quantizes the MTP draft head.
|
||||
|
||||
---
|
||||
|
||||
## What / where
|
||||
|
||||
- **Model:** NEO-CODE = `DavidAU/Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking` (dense
|
||||
27B, `Qwen3_5` GDN-hybrid arch, multimodal `Qwen3_5ForConditionalGeneration`).
|
||||
- **Runs only on ana-ml2 GPU0** (NVFP4 is Blackwell-only; irv-ml1 is Ampere).
|
||||
- **Artifacts** (ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/`, root-owned):
|
||||
- `heretic2-mtp-bf16/` — BF16 graft (Heretic2 + 15 base-Qwen3.6 MTP tensors). [graft input]
|
||||
- `heretic2-modelopt-nvfp4/` — modelopt NVFP4 quant, single shard, **no mtp**. [quant output]
|
||||
- `heretic2-modelopt-nvfp4-mtp/` — the above + spliced 15 BF16 mtp → **the seat**. [SERVE THIS]
|
||||
- (superseded: `heretic2-nvfp4-cg*` = compressed-tensors path, coherent but MTP-inert;
|
||||
`heretic2-mtp-nvfp4-prod` = original gibberish. Keep for diff, do not serve.)
|
||||
- **Scripts** (eshpfi `services/heretic2-nvfp4-quant/`): `graft_mtp.py`, `quant_modelopt.py`,
|
||||
`finalize_modelopt_mtp.py`, `serve_modelopt_mtp.sh`, `sitecustomize-mtp-workaround.py`.
|
||||
- **Reference:** the MoE `gen` (`qwen36-35b-a3b-heretic-nvfp4`, `quant_method: modelopt`) and
|
||||
the qwopus-122B `gen` both ran MTP before (qwopus +12% single-stream, archival-memory
|
||||
2026-07-01) — dropped for `gen` because MTP *hurts concurrency*, which is why it belongs on
|
||||
the single-stream RP seats, not `gen`.
|
||||
|
||||
## GPU window ritual
|
||||
|
||||
Base NVFP4 quant needs ~55 GB free on GPU0. `docker stop llama-charrp
|
||||
llama-charrp-reasoning vllm-aeon-gen` (→ ~97 GB free); restore with `docker start …`
|
||||
(~90–230 s to healthy). The GGUF NEO-CODE seat is the always-restorable fallback. Heads-up
|
||||
wt-dev (their character / thoughtful-character / gen route through these) — unless told
|
||||
otherwise. `ssh ana-ml2` = lkraven, in the docker group (no sudo needed for docker).
|
||||
|
||||
---
|
||||
|
||||
## The pipeline (4 steps)
|
||||
|
||||
### 1. GRAFT (CPU, seats up) — `graft_mtp.py`
|
||||
Heretic2's finetune dropped the MTP head; graft the 15 BF16 `mtp.*` tensors from base
|
||||
`Qwen/Qwen3.6-27B` (shards 13+15). Symlinks Heretic2 shards + one `model-mtp.safetensors`.
|
||||
Idempotent, refuses to clobber. Output: `heretic2-mtp-bf16/`.
|
||||
|
||||
### 2. QUANT (GPU0 window, ~18 min) — `quant_modelopt.py` via `run_quant_modelopt.sh`
|
||||
`nvidia-modelopt` PTQ → **modelopt** NVFP4 format. Three things this script gets right (each a
|
||||
gotcha — see below): loads as **`AutoModelForImageTextToText`**, patches the modelopt↔transformers
|
||||
**FusedMoE** bug, and forces **single-shard** export. Excludes `lm_head` + `visual` + all
|
||||
`linear_attn` (GDN) → BF16, matching AEON. Calib = the 512-row workload-matched chat mix.
|
||||
```bash
|
||||
docker run -d --name vllm-heretic2-modelopt-quant --gpus '"device=0"' --ipc host \
|
||||
-v /tank/aimodels:/tank/aimodels -v /home/lkraven:/lk \
|
||||
--entrypoint bash vllm/vllm-openai:v0.24.0 -c '
|
||||
set -e
|
||||
pip install -q nvidia-modelopt tiktoken sentencepiece 2>&1 | tail -1
|
||||
python3 /lk/quant_modelopt.py \
|
||||
--model /tank/aimodels/heretic2-nvfp4-work/heretic2-mtp-bf16 \
|
||||
--calib-mode chat --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
|
||||
--num-samples 512 --seqlen 8192 \
|
||||
--out /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4'
|
||||
```
|
||||
CPU dry-run (no GPU, tiny calib) to validate the pipeline without an outage: same command
|
||||
minus `--gpus`, add `-e CUDA_VISIBLE_DEVICES=""`, `--num-samples 2 --seqlen 512`.
|
||||
|
||||
### 3. SPLICE (CPU) — `finalize_modelopt_mtp.py`
|
||||
Copy `heretic2-modelopt-nvfp4` → `heretic2-modelopt-nvfp4-mtp`, splice the 15 BF16 `mtp.*`
|
||||
tensors into the single shard (→ 1967 tensors). (The transformers load never builds an mtp
|
||||
module, so mtp must be spliced post-quant — same as AEON/pantheon.)
|
||||
|
||||
### 4. SERVE (GPU0) — `serve_modelopt_mtp.sh` + the MTP workaround
|
||||
```bash
|
||||
docker run -d --name vllm-charrp-modelopt --gpus '"device=0"' --ipc host \
|
||||
-v /tank/aimodels:/tank/aimodels \
|
||||
-v <sitecustomize dir>:/lk_debug -e PYTHONPATH=/lk_debug \ # ← the MTP workaround, see below
|
||||
-p 8018:8000 vllm/vllm-openai:v0.24.0 \
|
||||
/tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4-mtp \
|
||||
--quantization modelopt \
|
||||
--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \
|
||||
--language-model-only --mamba-cache-dtype float32 \
|
||||
--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \
|
||||
--served-model-name char-rp-reasoning --max-model-len 40960 --max-num-seqs 32 \
|
||||
--gpu-memory-utilization 0.5 --trust-remote-code
|
||||
```
|
||||
`--language-model-only` skips the vision tower (RP seat doesn't need it; saves ~1–2 GB — the
|
||||
tower is preserved BF16 in the weights, so multimodal is recoverable by dropping the flag).
|
||||
|
||||
---
|
||||
|
||||
## The four landmines (each cost hours)
|
||||
|
||||
1. **Load as `AutoModelForImageTextToText`, NEVER `AutoModelForCausalLM`.** The latter resolves
|
||||
`qwen3_5` → text-only `Qwen3_5ForCausalLM` → flat `model.layers.*` keys. vLLM only serves
|
||||
`Qwen3_5ForConditionalGeneration`, whose weight mapper needs `model.language_model.*` (+
|
||||
`model.visual.*`). Wrong class → every layer weight silently fails to load → **`!!!!` gibberish**.
|
||||
|
||||
2. **Use the MODELOPT format (nvidia-modelopt), not compressed-tensors (llm-compressor).** On
|
||||
compressed-tensors the MTP drafter can't load the BF16 mtp head at all (`not found in
|
||||
params_dict`, **0% acceptance** — loads but never accelerates; this is what pantheon and the
|
||||
"AEON RP seat" actually were). Base NVFP4 *alone* ≈ GGUF at batch-1 (no single-stream win) —
|
||||
**the MTP multiplier is the entire point**, and it needs modelopt.
|
||||
|
||||
3. **modelopt 0.45 ↔ transformers 5.12.1 FusedMoE crash.** `mtq.quantize` dies with
|
||||
`TypeError: issubclass() arg 2 must be a class` — modelopt registered transformers' `FusedMoE`
|
||||
(a *function* in 5.x) as an nn class. `quant_modelopt.py` guards it (patches
|
||||
`_DMRegistryCls._get_registered_nn_class` to skip non-class registry entries). Do **not**
|
||||
pin `nvidia-modelopt[hf]==0.43` to dodge it — that drags transformers back to 4.57 which can't
|
||||
load `qwen3_5` at all.
|
||||
|
||||
4. **⭐ THE BIG ONE — vLLM 0.24.0 does not propagate modelopt `exclude_modules` to the
|
||||
spec-decode DRAFT model.** The MTP drafter builds its own `qkv_proj`/`gate_up_proj` as
|
||||
*quantized* (NVFP4-packed) while the mtp head is BF16 → `AssertionError: param_data.shape ==
|
||||
loaded_weight.shape` in `qwen3_5_mtp.py:256`. **No checkpoint config fixes this** — instrumenting
|
||||
`is_layer_skipped` proved the drafter's exclude list contains only the *main* model's
|
||||
`linear_attn` entries, never the mtp ones. Also note `is_layer_skipped` does **exact string
|
||||
membership, not glob** — so wildcards like `mtp.layers.0.*` never match anything. **Fix = a
|
||||
runtime patch** (`sitecustomize-mtp-workaround.py`, mounted on `PYTHONPATH`) that force-skips
|
||||
any `mtp.*` prefix in `is_layer_skipped`, keeping the drafter BF16. This is a genuine vLLM bug —
|
||||
**report upstream** (draft-model quant-config should inherit the target's exclude_modules).
|
||||
|
||||
## Verify it's actually accelerating
|
||||
|
||||
```bash
|
||||
# coherence
|
||||
curl -s :8018/v1/completions -d '{"model":"char-rp-reasoning","prompt":"The old tavern","max_tokens":40,"temperature":0}'
|
||||
# drive tokens, then read acceptance from the seat log:
|
||||
docker logs vllm-charrp-modelopt 2>&1 | grep SpecDecoding | tail -2
|
||||
# -> "Mean acceptance length: 2.19 ... Avg Draft acceptance rate: 39.7%" [GOOD: >0%, ~2 length]
|
||||
# -> "Avg Draft acceptance rate: 0.0%" [BAD: compressed-tensors, or mtp quantized]
|
||||
```
|
||||
`SpecDecoding` line only appears during active generation. 0% acceptance = you're on
|
||||
compressed-tensors, or the workaround didn't load (check for `[ISLS] ... workaround installed`).
|
||||
|
||||
## Productionization TODO (not yet done)
|
||||
- Bake the sitecustomize workaround into a compose stack (mount + `PYTHONPATH`), served-name
|
||||
`char-rp-reasoning`, alongside/replacing the GGUF seat.
|
||||
- brokkr P00 (soong 9-tool k5) — same base model as GGUF NEO-CODE so R36 should carry, but the
|
||||
NVFP4-vs-Q5 quality + tool-path must be confirmed before cutover.
|
||||
- Repoint gateway `char-rp-reasoning` alias + heads-up wt-dev.
|
||||
- File the vLLM upstream bug (draft-model exclude non-inheritance).
|
||||
+28
-1
@@ -104,7 +104,34 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
_As of 2026-07-14 — ONE active task: the NVFP4 fast char-rp-reasoning seat. GIBBERISH RESOLVED (quant-namespace bug — NVFP4 now serves coherent) and the speed premise DISPROVEN on the llm-compressor/compressed-tensors format (base NVFP4 ≈ GGUF at batch-1; MTP 0%-accept). PIVOTED (operator) to a **modelopt-format re-quant for working MTP** — scoped + de-risked, needs `quant_modelopt.py` + one more GPU0 window (see ★ section). Everything else this session LANDED: the char-rp-reasoning Deckard→NEO-CODE swap (#355 resolved), the Worldtree deploy-speed PR, and the soong-lab webhook fix — see Recent decisions + git; the old #355/deploy-speed detail below is kept as history (both DONE)._
|
||||
|
||||
### ✅ RESOLVED (gibberish) → ⏭ PIVOT: modelopt-format re-quant for working MTP
|
||||
### ✅✅ DONE — modelopt NVFP4 + MTP fast char-rp-reasoning seat WORKS (2026-07-14)
|
||||
|
||||
**WORKING at ~77 tok/s** (vs GGUF NEO-CODE ~59.5, base NVFP4 ~53 → **~1.3× over GGUF**), MTP
|
||||
draft-acceptance **32–40%**, mean acceptance length **2.19**. Same Heretic2/NEO-CODE model, NVFP4
|
||||
+ native MTP. **★ FULL RECIPE + all gotchas: `docs/runbooks/heretic2-nvfp4-mtp-seat.md`** (the
|
||||
fire-drill-killer the operator demanded). Scripts: eshpfi `services/heretic2-nvfp4-quant/`
|
||||
(`quant_modelopt.py`, `finalize_modelopt_mtp.py`, `serve_modelopt_mtp.sh`,
|
||||
`sitecustomize-mtp-workaround.py`). Seat artifact: ana-ml2
|
||||
`/tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4-mtp`.
|
||||
|
||||
**The four landmines (each cost hours — full detail in the runbook):** (1) load as
|
||||
`AutoModelForImageTextToText` not `AutoModelForCausalLM` (namespace → else `!!!!`); (2) MODELOPT
|
||||
format not compressed-tensors (compressed-tensors MTP = 0% accept; base NVFP4 alone ≈ GGUF, MTP is
|
||||
the whole win); (3) modelopt 0.45↔transformers 5.12.1 FusedMoE crash — guarded in `quant_modelopt.py`;
|
||||
(4) **⭐ vLLM 0.24.0 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** →
|
||||
the BF16 mtp head gets quantized → shape crash; **no checkpoint config fixes it** (`is_layer_skipped`
|
||||
uses exact membership not glob, and the drafter never sees the mtp excludes) → **fix = a mounted
|
||||
`sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (report upstream as a vLLM bug).
|
||||
|
||||
**Operator's memory was right:** MTP ran before on the MoE `gen` (qwopus-122B +12% single-stream,
|
||||
archival 2026-07-01) — dropped for `gen` because MTP HURTS concurrency; it belongs on the
|
||||
single-stream RP seats. **NOT-YET-DONE (productionization):** compose stack with the workaround baked
|
||||
in, brokkr P00 (soong 9-tool k5 — same base as GGUF so R36 should carry, confirm NVFP4-vs-Q5 quality),
|
||||
gateway `char-rp-reasoning` repoint, wt-dev heads-up, file the vLLM upstream bug.
|
||||
|
||||
---
|
||||
|
||||
### (historical) RESOLVED (gibberish) → PIVOT: modelopt-format re-quant for working MTP
|
||||
|
||||
**2026-07-14 WINDOW OUTCOME (this session).** Ran the diagnostic ladder in one clean ~40-min GPU0 window.
|
||||
- **GIBBERISH ROOT CAUSE = quant NAMESPACE (found from config diffs + vLLM source, ZERO GPU time).** `quant_nvfp4.py` loaded via `AutoModelForCausalLM` → resolves qwen3_5 to the text-only `Qwen3_5ForCausalLM` → weight keys born flat `model.layers.*` (no vision). But vLLM 0.24 registers ONLY `Qwen3_5ForConditionalGeneration` (registry.py:566), whose `hf_to_vllm_mapper` (qwen3_vl.py:1692) remaps `model.language_model.*`→`language_model.model.*` and has NO rule for a bare `model.layers.` prefix → every transformer-layer weight fails to match → uninitialized → `!!!!`. The step-4 config-merge to ConditionalGeneration was a doomed patch over a wrong-namespace checkpoint. **FIX = load as `AutoModelForImageTextToText`** (resolves qwen3_5 → `Qwen3_5ForConditionalGeneration` → keys born `model.language_model.*` + `model.visual.*`, pantheon namespace). One-class swap; committed to `quant_nvfp4.py`.
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Splice the 15 BF16 mtp.* tensors into the modelopt NVFP4 quant output → the servable seat.
|
||||
|
||||
Runs AFTER quant_modelopt.py. Copies heretic2-modelopt-nvfp4 → heretic2-modelopt-nvfp4-mtp,
|
||||
splices the grafted BF16 mtp head into the single shard (transformers never builds an mtp module
|
||||
at load, so mtp is always post-quant-spliced — same as AEON/pantheon), and adds the mtp module
|
||||
names to config.json exclude_modules for tidiness. NOTE: the actual thing that keeps the mtp head
|
||||
BF16 at serve time is the sitecustomize MTP workaround (see runbook landmine #4); the config
|
||||
exclude here is belt-and-suspenders and does NOT by itself prevent the drafter-quant crash.
|
||||
|
||||
Run in a vLLM container (root; /tank/aimodels files are root-owned):
|
||||
docker run --rm -v /tank/aimodels:/tank/aimodels -v /home/lkraven:/lk \
|
||||
--entrypoint python3 vllm/vllm-openai:v0.24.0 /lk/finalize_modelopt_mtp.py
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
|
||||
from safetensors import safe_open
|
||||
from safetensors.torch import save_file
|
||||
|
||||
WORK = "/tank/aimodels/heretic2-nvfp4-work"
|
||||
SRC = f"{WORK}/heretic2-modelopt-nvfp4"
|
||||
DST = f"{WORK}/heretic2-modelopt-nvfp4-mtp"
|
||||
GRAFT = f"{WORK}/heretic2-mtp-bf16"
|
||||
|
||||
if os.path.exists(DST):
|
||||
shutil.rmtree(DST)
|
||||
print(f"copying {SRC} -> {DST}", flush=True)
|
||||
shutil.copytree(SRC, DST)
|
||||
|
||||
out_st = f"{DST}/model.safetensors" # single shard (quant_modelopt.py forces max_shard_size huge)
|
||||
mtp_st = f"{GRAFT}/model-mtp.safetensors"
|
||||
tensors = {}
|
||||
with safe_open(out_st, framework="pt") as f:
|
||||
for k in f.keys():
|
||||
tensors[k] = f.get_tensor(k)
|
||||
n_main = len(tensors)
|
||||
with safe_open(mtp_st, framework="pt") as f:
|
||||
mtp_keys = list(f.keys())
|
||||
for k in mtp_keys:
|
||||
tensors[k] = f.get_tensor(k)
|
||||
assert not any("mtp" in k.lower() for k in list(tensors)[:n_main]), "output already had mtp?"
|
||||
save_file(tensors, out_st, metadata={"format": "pt"})
|
||||
print(f"spliced {len(mtp_keys)} bf16 mtp tensors -> {len(tensors)} total", flush=True)
|
||||
|
||||
cfgp = f"{DST}/config.json"
|
||||
cfg = json.load(open(cfgp))
|
||||
qc = cfg.setdefault("quantization_config", {})
|
||||
exc = qc.setdefault("exclude_modules", [])
|
||||
mtp_mods = sorted({k.rsplit(".", 1)[0] for k in mtp_keys})
|
||||
exc.extend(m for m in mtp_mods if m not in exc)
|
||||
json.dump(cfg, open(cfgp, "w"), indent=2)
|
||||
print(f"config exclude_modules += {len(mtp_mods)} mtp modules; total {len(exc)}", flush=True)
|
||||
print(f"DONE: {DST}", flush=True)
|
||||
@@ -93,6 +93,22 @@ def main() -> int:
|
||||
import modelopt.torch.quantization as mtq
|
||||
from modelopt.torch.export import export_hf_checkpoint
|
||||
|
||||
# modelopt 0.45 + transformers 5.12.1 compat guard. transformers 5.x exposes `FusedMoE` as a
|
||||
# FUNCTION, but modelopt registers it in QuantModuleRegistry expecting an nn.Module class, so the
|
||||
# registry scan (register_fused_experts_on_the_fly -> _get_registered_nn_class) does
|
||||
# `issubclass(nn_cls, <function FusedMoE>)` and dies with "arg 2 must be a class". Our model is
|
||||
# DENSE (no FusedMoE) so skipping non-class registry entries is safe. Guard the scan:
|
||||
from modelopt.torch.opt import dynamic as _mo_dyn
|
||||
|
||||
def _grnc_safe(self, nn_cls):
|
||||
for nn_cls_ in self._registry:
|
||||
if (isinstance(nn_cls_, type) and issubclass(nn_cls, nn_cls_)
|
||||
and nn_cls.forward is nn_cls_.forward):
|
||||
return nn_cls_
|
||||
return None
|
||||
|
||||
_mo_dyn._DMRegistryCls._get_registered_nn_class = _grnc_safe
|
||||
|
||||
print(f"loading grafted model (multimodal ConditionalGeneration): {args.model}", flush=True)
|
||||
model = AutoModelForImageTextToText.from_pretrained(
|
||||
args.model, torch_dtype="auto", device_map="auto", trust_remote_code=True,
|
||||
@@ -120,7 +136,9 @@ def main() -> int:
|
||||
mtq.quantize(model, cfg, forward_loop=forward_loop)
|
||||
|
||||
print(f"exporting modelopt HF checkpoint -> {args.out}", flush=True)
|
||||
export_hf_checkpoint(model, export_dir=args.out)
|
||||
# Force a SINGLE shard (default max_shard_size 10GB would split the ~14GB output into 3 shards,
|
||||
# but splice_mtp.py expects a single <out>/model.safetensors to add the bf16 mtp.* into).
|
||||
export_hf_checkpoint(model, export_dir=args.out, max_shard_size="1TB")
|
||||
tok.save_pretrained(args.out)
|
||||
print("DONE. Next: splice_mtp.py <out> <graft> then serve "
|
||||
"--quantization modelopt --speculative-config "
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
#!/bin/bash
|
||||
# Launch the modelopt NVFP4 quant of the grafted Heretic2 on ana-ml2 GPU0 (detached, survives ssh
|
||||
# drop). bare nvidia-modelopt (0.45). The FusedMoE-compat guard + single-shard export + the
|
||||
# multimodal load class are all inside quant_modelopt.py. ~18 min. Output: heretic2-modelopt-nvfp4
|
||||
# (no mtp yet — run finalize_modelopt_mtp.py after). quant_modelopt.py must be at /lk/quant_modelopt.py
|
||||
# (mount /home/lkraven as /lk, or scp it there first).
|
||||
set -euo pipefail
|
||||
docker rm -f vllm-heretic2-modelopt-quant 2>/dev/null || true
|
||||
rm -rf /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4 2>/dev/null || true
|
||||
docker run -d --name vllm-heretic2-modelopt-quant --gpus '"device=0"' --ipc host \
|
||||
-v /tank/aimodels:/tank/aimodels -v /home/lkraven:/lk \
|
||||
--entrypoint bash vllm/vllm-openai:v0.24.0 -c '
|
||||
set -e
|
||||
pip install -q nvidia-modelopt tiktoken sentencepiece 2>&1 | tail -1
|
||||
python3 /lk/quant_modelopt.py \
|
||||
--model /tank/aimodels/heretic2-nvfp4-work/heretic2-mtp-bf16 \
|
||||
--calib-mode chat --calib /tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl \
|
||||
--num-samples 512 --seqlen 8192 \
|
||||
--out /tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4'
|
||||
echo "LAUNCHED: $(docker ps --filter name=vllm-heretic2-modelopt-quant --format '{{.Status}}')"
|
||||
@@ -0,0 +1,25 @@
|
||||
#!/bin/bash
|
||||
# Serve the modelopt NVFP4 + MTP Heretic2 seat (the WORKING fast char-rp-reasoning seat).
|
||||
# ~77 tok/s, MTP acceptance 32-40%. Requires: (1) heretic2-modelopt-nvfp4-mtp built (quant ->
|
||||
# finalize), (2) the sitecustomize MTP workaround mounted on PYTHONPATH (vLLM 0.24 draft-model
|
||||
# exclude bug — see runbook landmine #4; without it the engine crashes on a shape mismatch).
|
||||
set -euo pipefail
|
||||
MODEL="${1:-/tank/aimodels/heretic2-nvfp4-work/heretic2-modelopt-nvfp4-mtp}"
|
||||
# Dir containing sitecustomize.py (a copy of sitecustomize-mtp-workaround.py named sitecustomize.py):
|
||||
WORKAROUND_DIR="${MTP_WORKAROUND_DIR:-/home/lkraven/isls_debug}"
|
||||
docker rm -f vllm-charrp-modelopt 2>/dev/null || true
|
||||
docker run -d --name vllm-charrp-modelopt --gpus '"device=0"' --ipc host \
|
||||
-v /tank/aimodels:/tank/aimodels \
|
||||
-v "${WORKAROUND_DIR}":/lk_debug -e PYTHONPATH=/lk_debug \
|
||||
-p 8018:8000 \
|
||||
vllm/vllm-openai:v0.24.0 \
|
||||
"$MODEL" \
|
||||
--quantization modelopt \
|
||||
--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \
|
||||
--language-model-only \
|
||||
--mamba-cache-dtype float32 \
|
||||
--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \
|
||||
--served-model-name char-rp-reasoning \
|
||||
--max-model-len 40960 --max-num-seqs 32 --gpu-memory-utilization 0.5 --trust-remote-code
|
||||
echo "started: $(docker ps --filter name=vllm-charrp-modelopt --format '{{.Status}}')"
|
||||
echo "verify MTP: docker logs vllm-charrp-modelopt 2>&1 | grep -E 'mtp-workaround|SpecDecoding'"
|
||||
@@ -0,0 +1,53 @@
|
||||
# MTP draft-model quant workaround for vLLM 0.24.0 — MUST be named sitecustomize.py and be on
|
||||
# PYTHONPATH so it loads in the vLLM engine-core subprocess. Mount its directory into the serve
|
||||
# container and set -e PYTHONPATH=<mount>.
|
||||
#
|
||||
# THE BUG: vLLM 0.24.0 does not propagate the main model's modelopt `exclude_modules` to the
|
||||
# spec-decode DRAFT model (Qwen3_5MTP). So the drafter builds its own qkv_proj/gate_up_proj as
|
||||
# NVFP4-quantized while the grafted MTP head is BF16 → `AssertionError: param_data.shape ==
|
||||
# loaded_weight.shape` in qwen3_5_mtp.py:256, engine-core dies during weight load. Instrumenting
|
||||
# is_layer_skipped proved the drafter's exclude list only ever contains the *main* model's
|
||||
# entries, never the mtp ones — so no checkpoint config can fix it. (Also: is_layer_skipped does
|
||||
# exact string membership, not glob — wildcards like `mtp.layers.0.*` match nothing.)
|
||||
#
|
||||
# THE FIX: force is_layer_skipped to return True (skip = keep BF16) for any `mtp.*` layer, so the
|
||||
# draft head stays unquantized and its BF16 weights load. Report upstream: draft-model quant
|
||||
# config should inherit the target model's exclude_modules.
|
||||
import importlib.abc
|
||||
import importlib.util
|
||||
import sys
|
||||
|
||||
TARGET = "vllm.model_executor.layers.quantization.utils.quant_utils"
|
||||
|
||||
|
||||
class _Finder(importlib.abc.MetaPathFinder):
|
||||
def find_spec(self, name, path, target=None):
|
||||
if name != TARGET:
|
||||
return None
|
||||
sys.meta_path.remove(self)
|
||||
try:
|
||||
spec = importlib.util.find_spec(name)
|
||||
finally:
|
||||
sys.meta_path.insert(0, self)
|
||||
if not spec or not spec.loader:
|
||||
return None
|
||||
_orig_exec = spec.loader.exec_module
|
||||
|
||||
def exec_module(module):
|
||||
_orig_exec(module)
|
||||
_orig_isls = module.is_layer_skipped
|
||||
|
||||
def is_layer_skipped(prefix, ignored_layers, *args, **kwargs):
|
||||
pl = str(prefix)
|
||||
if pl.startswith("mtp.") or ".mtp." in pl:
|
||||
return True # keep the mtp draft head BF16
|
||||
return _orig_isls(prefix, ignored_layers, *args, **kwargs)
|
||||
|
||||
module.is_layer_skipped = is_layer_skipped
|
||||
print("[mtp-workaround] is_layer_skipped force-skip for mtp.* installed", flush=True)
|
||||
|
||||
spec.loader.exec_module = exec_module
|
||||
return spec
|
||||
|
||||
|
||||
sys.meta_path.insert(0, _Finder())
|
||||
Reference in New Issue
Block a user