fix(gen-seat): MTP mitigation — disable prefix caching, keep MTP (speed restored, multi-turn clean)

The qwen3_5_mtp corruption (playbook 3.7) is gated on MTP x prefix-caching
TOGETHER (vllm#43559 / #47194), per both cross-frontier peers. Disabling
prefix caching (--no-enable-prefix-caching; vLLM V1 defaults it ON, so the
explicit --no- form is required) forces the GDN cache into a mode where the
partial-accept align-path bug is inert, so MTP can stay on.

Verified on our stack (AEON W4A4): MTP on + prefix-caching off -> the 7-turn
varied series stays coherent through 3.9k tokens, zero cross-turn bleed, at
104.6 tok/s / 53.6% acceptance -- the FULL MTP speedup restored (vs ~half
with MTP off), losing only prefix-cache reuse. All 7 aliases route.

Ruled out on the way: num_speculative_tokens=1 (corruption is
depth-independent, n=1 and n=2 both corrupt); switching to SGLang (vLLM /
SGLang / llama.cpp mainline all share the architectural GDN-rollback bug).
Proper upstream fix (#51113) is in main / v0.27.2rc0 only, not stable, so we
hold at APC-off rather than jump the fleet gateway to an RC.

Supersedes the MTP-off config from 7bd38b3.
This commit is contained in:
vh
2026-08-16 22:44:14 -07:00
parent a8ed6e7428
commit 63a3cb2d86
2 changed files with 49 additions and 27 deletions
+31 -24
View File
@@ -56,7 +56,14 @@ services:
- float32
- --kv-cache-dtype
- ${GEN_KV_CACHE_DTYPE:-fp8}
- --enable-prefix-caching
# ⚠️ PREFIX CACHING DISABLED 2026-08-17 as the MTP mitigation. vLLM V1
# defaults prefix caching ON, so the explicit --no- form is required to
# turn it off. With MTP re-enabled below, APC-off forces the GDN cache
# into a mode where the partial-accept align-path bug is inert
# (vllm#43559 / #47194): the corruption is gated on MTP × prefix-caching
# together. Trade vs the alternative (MTP off): this keeps MTP's ~2x decode
# and loses only prefix-cache reuse. GATED ON the multi-turn probe passing.
- --no-enable-prefix-caching
- --enable-chunked-prefill
- --limit-mm-per-prompt
- '{"image": 4}'
@@ -95,34 +102,34 @@ services:
- --enable-auto-tool-choice
- --tool-call-parser
- qwen3_coder
# ⚠️ MTP / SPECULATIVE DECODING DISABLED 2026-08-17 — DO NOT re-enable
# without re-testing multi-turn coherence past ~2k tokens.
# MTP RE-ENABLED 2026-08-17, PAIRED WITH --no-enable-prefix-caching above.
#
# ROOT CAUSE (isolated 2026-08-16, operator-confirmed): `qwen3_5_mtp`
# speculative decoding CORRUPTS Qwen3.8-27B output in multi-turn
# conversations once cumulative context passes ~2,000 tokens. The draft
# head emits bad tokens that get accepted, and the response degenerates
# into CONTEXT-BLEEDING — stitching prior turns' content into the current
# answer (a "describe durian" reply that contained the Krebs-cycle and
# winter answers from earlier turns), then collapsing to a few words.
# conversations once cumulative context passes ~2,000 tokens — CONTEXT-
# BLEEDING (a "describe durian" reply that contained the Krebs-cycle and
# winter answers from earlier turns), then collapse. Confirmed by two
# cross-frontier research peers as a KNOWN upstream bug: the Gated-DeltaNet
# recurrent state cannot roll back on a partial draft-accept, so the SSM
# state corrupts (vllm#47087 symptom-twin, #43559 fix lineage). It is
# architectural — vLLM, SGLang, and llama.cpp mainline all share it — and
# NOT depth-dependent (n=1 and n=2 both corrupt, so num_speculative_tokens=1
# is NOT a fix).
#
# Proven model-INDEPENDENT across every Qwen3.8-27B quant we serve (AEON
# W4A4, unsloth FP8-attn, our in-house mixed) and FIXED by disabling MTP:
# the identical model stays coherent through 4k+ tokens with this off.
# It is MTP-on-Qwen3.8 specifically — Qwen3.6 (char-rp-reasoning) runs the
# same qwen3_5_mtp method and is CLEAN, so the 3.6 MTP head/graft is fine
# and the 3.8 one is not (suspects: the bf16 MTP graft, or spec depth 3).
# THE MITIGATION (this config): the corruption is gated on MTP × prefix-
# caching TOGETHER (#43559/#47194). Disabling prefix caching forces the GDN
# cache into a mode where the buggy align-path is inert, so MTP can stay on.
# Keeps MTP's ~2x decode; costs only prefix-cache reuse. The proper upstream
# fix (#51113) is in vLLM main / v0.27.2rc0 only — not in a stable release
# as of this date, so we do not jump the fleet gateway to an RC.
#
# COST: no speculative decoding = ~half decode tok/s. Accepted as the
# known-good until the 3.8 MTP is fixed.
#
# BEFORE RE-ENABLING: run the multi-turn probe in
# services/gen-seat-mixed-quant/bench/ (varied 7-turn series, watch t5-t7
# past 2k tokens for the collapse). First thing worth trying is
# num_speculative_tokens=1 (shallowest draft, least divergence). Restore by
# uncommenting the two lines below.
# - --speculative-config
# - '{"method": "${GEN_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${GEN_SPEC_TOKENS:-3}}'
# ⚠️ GATED ON THE MULTI-TURN PROBE. If the 7-turn varied series
# (services/gen-seat-mixed-quant/bench/) still collapses past ~2k tokens
# with this config, APC-off did not hold on our stack — revert to MTP OFF
# (comment these two lines, restore --enable-prefix-caching) which is the
# verified known-good.
- --speculative-config
- '{"method": "${GEN_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${GEN_SPEC_TOKENS:-3}}'
deploy:
resources:
reservations: