vh
982c319d9f
The fast char-rp-reasoning seat works: ~77 tok/s (vs GGUF ~59.5, base NVFP4 ~53), MTP draft-acceptance 32-40%, mean acceptance length 2.19. Same Heretic2/NEO-CODE model, NVFP4 + native qwen3_5_mtp spec-decode. Full end-to-end recipe + the four landmines in docs/runbooks/heretic2-nvfp4-mtp-seat.md: (1) load as AutoModelForImageTextToText not AutoModelForCausalLM (namespace/gibberish); (2) modelopt format not compressed-tensors (compressed-tensors MTP = 0% accept); (3) modelopt 0.45 <-> transformers 5.12.1 FusedMoE crash (guarded in quant_modelopt.py); (4) vLLM 0.24.0 does NOT propagate modelopt exclude_modules to the spec-decode draft model -> BF16 mtp head gets quantized -> shape crash; no checkpoint config fixes it (is_layer_skipped is exact-membership not glob) -> fix is a mounted sitecustomize that force-skips mtp.* in is_layer_skipped (upstream vLLM bug to report). Scripts: quant_modelopt.py (FusedMoE guard + single-shard export + multimodal load), finalize_modelopt_mtp.py (splice bf16 mtp), serve_modelopt_mtp.sh, run_quant_modelopt.sh, sitecustomize-mtp-workaround.py.
54 lines
2.3 KiB
Python
54 lines
2.3 KiB
Python
# MTP draft-model quant workaround for vLLM 0.24.0 — MUST be named sitecustomize.py and be on
|
|
# PYTHONPATH so it loads in the vLLM engine-core subprocess. Mount its directory into the serve
|
|
# container and set -e PYTHONPATH=<mount>.
|
|
#
|
|
# THE BUG: vLLM 0.24.0 does not propagate the main model's modelopt `exclude_modules` to the
|
|
# spec-decode DRAFT model (Qwen3_5MTP). So the drafter builds its own qkv_proj/gate_up_proj as
|
|
# NVFP4-quantized while the grafted MTP head is BF16 → `AssertionError: param_data.shape ==
|
|
# loaded_weight.shape` in qwen3_5_mtp.py:256, engine-core dies during weight load. Instrumenting
|
|
# is_layer_skipped proved the drafter's exclude list only ever contains the *main* model's
|
|
# entries, never the mtp ones — so no checkpoint config can fix it. (Also: is_layer_skipped does
|
|
# exact string membership, not glob — wildcards like `mtp.layers.0.*` match nothing.)
|
|
#
|
|
# THE FIX: force is_layer_skipped to return True (skip = keep BF16) for any `mtp.*` layer, so the
|
|
# draft head stays unquantized and its BF16 weights load. Report upstream: draft-model quant
|
|
# config should inherit the target model's exclude_modules.
|
|
import importlib.abc
|
|
import importlib.util
|
|
import sys
|
|
|
|
TARGET = "vllm.model_executor.layers.quantization.utils.quant_utils"
|
|
|
|
|
|
class _Finder(importlib.abc.MetaPathFinder):
|
|
def find_spec(self, name, path, target=None):
|
|
if name != TARGET:
|
|
return None
|
|
sys.meta_path.remove(self)
|
|
try:
|
|
spec = importlib.util.find_spec(name)
|
|
finally:
|
|
sys.meta_path.insert(0, self)
|
|
if not spec or not spec.loader:
|
|
return None
|
|
_orig_exec = spec.loader.exec_module
|
|
|
|
def exec_module(module):
|
|
_orig_exec(module)
|
|
_orig_isls = module.is_layer_skipped
|
|
|
|
def is_layer_skipped(prefix, ignored_layers, *args, **kwargs):
|
|
pl = str(prefix)
|
|
if pl.startswith("mtp.") or ".mtp." in pl:
|
|
return True # keep the mtp draft head BF16
|
|
return _orig_isls(prefix, ignored_layers, *args, **kwargs)
|
|
|
|
module.is_layer_skipped = is_layer_skipped
|
|
print("[mtp-workaround] is_layer_skipped force-skip for mtp.* installed", flush=True)
|
|
|
|
spec.loader.exec_module = exec_module
|
|
return spec
|
|
|
|
|
|
sys.meta_path.insert(0, _Finder())
|