From 63a3cb2d86d5ca18374b70156f2b80c75a484c62 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 16 Aug 2026 22:44:14 -0700 Subject: [PATCH] =?UTF-8?q?fix(gen-seat):=20MTP=20mitigation=20=E2=80=94?= =?UTF-8?q?=20disable=20prefix=20caching,=20keep=20MTP=20(speed=20restored?= =?UTF-8?q?,=20multi-turn=20clean)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The qwen3_5_mtp corruption (playbook 3.7) is gated on MTP x prefix-caching TOGETHER (vllm#43559 / #47194), per both cross-frontier peers. Disabling prefix caching (--no-enable-prefix-caching; vLLM V1 defaults it ON, so the explicit --no- form is required) forces the GDN cache into a mode where the partial-accept align-path bug is inert, so MTP can stay on. Verified on our stack (AEON W4A4): MTP on + prefix-caching off -> the 7-turn varied series stays coherent through 3.9k tokens, zero cross-turn bleed, at 104.6 tok/s / 53.6% acceptance -- the FULL MTP speedup restored (vs ~half with MTP off), losing only prefix-cache reuse. All 7 aliases route. Ruled out on the way: num_speculative_tokens=1 (corruption is depth-independent, n=1 and n=2 both corrupt); switching to SGLang (vLLM / SGLang / llama.cpp mainline all share the architectural GDN-rollback bug). Proper upstream fix (#51113) is in main / v0.27.2rc0 only, not stable, so we hold at APC-off rather than jump the fleet gateway to an RC. Supersedes the MTP-off config from 7bd38b3. --- docs/pfi/model-quantization-playbook.md | 21 ++++++++-- stacks/gen-seat/compose.yaml | 55 ++++++++++++++----------- 2 files changed, 49 insertions(+), 27 deletions(-) diff --git a/docs/pfi/model-quantization-playbook.md b/docs/pfi/model-quantization-playbook.md index 53b052a..733f380 100644 --- a/docs/pfi/model-quantization-playbook.md +++ b/docs/pfi/model-quantization-playbook.md @@ -160,9 +160,24 @@ Open upstream question (queried dvalin/bil-smithy 2026-08-17). **Rule: gate MTP on a MULTI-TURN coherence probe, not just single-shot acceptance.** Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens -for length-collapse and cross-turn bleed. If it corrupts: disable MTP (costs ~half -decode tok/s) or try `num_speculative_tokens=1` (shallowest draft, least divergence -— untested as of this writing). The gen seat runs MTP **off** as of 2026-08-17. +for length-collapse and cross-turn bleed. + +**THE MITIGATION (resolved 2026-08-17): disable prefix caching, keep MTP.** The +corruption is gated on MTP × prefix-caching *together* (vllm#43559 / #47194) — with +`--no-enable-prefix-caching` the GDN cache runs in a mode where the buggy +partial-accept align-path is inert. Confirmed on our stack: AEON W4A4, MTP on + +prefix-caching off → the 7-turn varied series stays coherent through 3.9k tokens, +zero bleed, at **104.6 tok/s / 53.6% acceptance** — i.e. the FULL MTP speedup back +(vs ~half with MTP off), losing only prefix-cache reuse. The gen seat runs this +config as of 2026-08-17. + +Things that do **not** work, ruled out: `num_speculative_tokens=1` (corruption is +depth-independent — reproduces at n=1 and n=2, deterministically probed upstream); +switching engine (vLLM / SGLang / llama.cpp mainline all share the GDN-rollback +bug — it is architectural). The proper upstream fix (vllm#51113) is in `main` / +`v0.27.2rc0` only — not in a stable release, so we hold at APC-off until it lands. +Two cross-frontier peers (dvalin/bil-smithy) confirmed the bug class and pointed +at the open symptom-twin issue #47087. ### 3.4 Toolchain version deadlocks diff --git a/stacks/gen-seat/compose.yaml b/stacks/gen-seat/compose.yaml index fa060e0..e276322 100644 --- a/stacks/gen-seat/compose.yaml +++ b/stacks/gen-seat/compose.yaml @@ -56,7 +56,14 @@ services: - float32 - --kv-cache-dtype - ${GEN_KV_CACHE_DTYPE:-fp8} - - --enable-prefix-caching + # ⚠️ PREFIX CACHING DISABLED 2026-08-17 as the MTP mitigation. vLLM V1 + # defaults prefix caching ON, so the explicit --no- form is required to + # turn it off. With MTP re-enabled below, APC-off forces the GDN cache + # into a mode where the partial-accept align-path bug is inert + # (vllm#43559 / #47194): the corruption is gated on MTP × prefix-caching + # together. Trade vs the alternative (MTP off): this keeps MTP's ~2x decode + # and loses only prefix-cache reuse. GATED ON the multi-turn probe passing. + - --no-enable-prefix-caching - --enable-chunked-prefill - --limit-mm-per-prompt - '{"image": 4}' @@ -95,34 +102,34 @@ services: - --enable-auto-tool-choice - --tool-call-parser - qwen3_coder - # ⚠️ MTP / SPECULATIVE DECODING DISABLED 2026-08-17 — DO NOT re-enable - # without re-testing multi-turn coherence past ~2k tokens. + # MTP RE-ENABLED 2026-08-17, PAIRED WITH --no-enable-prefix-caching above. # # ROOT CAUSE (isolated 2026-08-16, operator-confirmed): `qwen3_5_mtp` # speculative decoding CORRUPTS Qwen3.8-27B output in multi-turn - # conversations once cumulative context passes ~2,000 tokens. The draft - # head emits bad tokens that get accepted, and the response degenerates - # into CONTEXT-BLEEDING — stitching prior turns' content into the current - # answer (a "describe durian" reply that contained the Krebs-cycle and - # winter answers from earlier turns), then collapsing to a few words. + # conversations once cumulative context passes ~2,000 tokens — CONTEXT- + # BLEEDING (a "describe durian" reply that contained the Krebs-cycle and + # winter answers from earlier turns), then collapse. Confirmed by two + # cross-frontier research peers as a KNOWN upstream bug: the Gated-DeltaNet + # recurrent state cannot roll back on a partial draft-accept, so the SSM + # state corrupts (vllm#47087 symptom-twin, #43559 fix lineage). It is + # architectural — vLLM, SGLang, and llama.cpp mainline all share it — and + # NOT depth-dependent (n=1 and n=2 both corrupt, so num_speculative_tokens=1 + # is NOT a fix). # - # Proven model-INDEPENDENT across every Qwen3.8-27B quant we serve (AEON - # W4A4, unsloth FP8-attn, our in-house mixed) and FIXED by disabling MTP: - # the identical model stays coherent through 4k+ tokens with this off. - # It is MTP-on-Qwen3.8 specifically — Qwen3.6 (char-rp-reasoning) runs the - # same qwen3_5_mtp method and is CLEAN, so the 3.6 MTP head/graft is fine - # and the 3.8 one is not (suspects: the bf16 MTP graft, or spec depth 3). + # THE MITIGATION (this config): the corruption is gated on MTP × prefix- + # caching TOGETHER (#43559/#47194). Disabling prefix caching forces the GDN + # cache into a mode where the buggy align-path is inert, so MTP can stay on. + # Keeps MTP's ~2x decode; costs only prefix-cache reuse. The proper upstream + # fix (#51113) is in vLLM main / v0.27.2rc0 only — not in a stable release + # as of this date, so we do not jump the fleet gateway to an RC. # - # COST: no speculative decoding = ~half decode tok/s. Accepted as the - # known-good until the 3.8 MTP is fixed. - # - # BEFORE RE-ENABLING: run the multi-turn probe in - # services/gen-seat-mixed-quant/bench/ (varied 7-turn series, watch t5-t7 - # past 2k tokens for the collapse). First thing worth trying is - # num_speculative_tokens=1 (shallowest draft, least divergence). Restore by - # uncommenting the two lines below. - # - --speculative-config - # - '{"method": "${GEN_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${GEN_SPEC_TOKENS:-3}}' + # ⚠️ GATED ON THE MULTI-TURN PROBE. If the 7-turn varied series + # (services/gen-seat-mixed-quant/bench/) still collapses past ~2k tokens + # with this config, APC-off did not hold on our stack — revert to MTP OFF + # (comment these two lines, restore --enable-prefix-caching) which is the + # verified known-good. + - --speculative-config + - '{"method": "${GEN_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${GEN_SPEC_TOKENS:-3}}' deploy: resources: reservations: