fix(gen-seat): MTP mitigation — disable prefix caching, keep MTP (speed restored, multi-turn clean)

The qwen3_5_mtp corruption (playbook 3.7) is gated on MTP x prefix-caching TOGETHER (vllm#43559 / #47194), per both cross-frontier peers. Disabling prefix caching (--no-enable-prefix-caching; vLLM V1 defaults it ON, so the explicit --no- form is required) forces the GDN cache into a mode where the partial-accept align-path bug is inert, so MTP can stay on. Verified on our stack (AEON W4A4): MTP on + prefix-caching off -> the 7-turn varied series stays coherent through 3.9k tokens, zero cross-turn bleed, at 104.6 tok/s / 53.6% acceptance -- the FULL MTP speedup restored (vs ~half with MTP off), losing only prefix-cache reuse. All 7 aliases route. Ruled out on the way: num_speculative_tokens=1 (corruption is depth-independent, n=1 and n=2 both corrupt); switching to SGLang (vLLM / SGLang / llama.cpp mainline all share the architectural GDN-rollback bug). Proper upstream fix (#51113) is in main / v0.27.2rc0 only, not stable, so we hold at APC-off rather than jump the fleet gateway to an RC. Supersedes the MTP-off config from 7bd38b3.
2026-08-16 22:44:14 -07:00
parent a8ed6e7428
commit 63a3cb2d86
2 changed files with 49 additions and 27 deletions
@@ -160,9 +160,24 @@ Open upstream question (queried dvalin/bil-smithy 2026-08-17).

 **Rule: gate MTP on a MULTI-TURN coherence probe, not just single-shot acceptance.**
 Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens
-for length-collapse and cross-turn bleed. If it corrupts: disable MTP (costs ~half
-decode tok/s) or try `num_speculative_tokens=1` (shallowest draft, least divergence
-— untested as of this writing). The gen seat runs MTP **off** as of 2026-08-17.
+for length-collapse and cross-turn bleed.
+
+**THE MITIGATION (resolved 2026-08-17): disable prefix caching, keep MTP.** The
+corruption is gated on MTP × prefix-caching *together* (vllm#43559 / #47194) — with
+`--no-enable-prefix-caching` the GDN cache runs in a mode where the buggy
+partial-accept align-path is inert. Confirmed on our stack: AEON W4A4, MTP on +
+prefix-caching off → the 7-turn varied series stays coherent through 3.9k tokens,
+zero bleed, at **104.6 tok/s / 53.6% acceptance** — i.e. the FULL MTP speedup back
+(vs ~half with MTP off), losing only prefix-cache reuse. The gen seat runs this
+config as of 2026-08-17.
+
+Things that do **not** work, ruled out: `num_speculative_tokens=1` (corruption is
+depth-independent — reproduces at n=1 and n=2, deterministically probed upstream);
+switching engine (vLLM / SGLang / llama.cpp mainline all share the GDN-rollback
+bug — it is architectural). The proper upstream fix (vllm#51113) is in `main` /
+`v0.27.2rc0` only — not in a stable release, so we hold at APC-off until it lands.
+Two cross-frontier peers (dvalin/bil-smithy) confirmed the bug class and pointed
+at the open symptom-twin issue #47087.

 ### 3.4 Toolchain version deadlocks

@@ -56,7 +56,14 @@ services:
      - float32
      - --kv-cache-dtype
      - ${GEN_KV_CACHE_DTYPE:-fp8}
-      - --enable-prefix-caching
+      # ⚠️ PREFIX CACHING DISABLED 2026-08-17 as the MTP mitigation. vLLM V1
+      # defaults prefix caching ON, so the explicit --no- form is required to
+      # turn it off. With MTP re-enabled below, APC-off forces the GDN cache
+      # into a mode where the partial-accept align-path bug is inert
+      # (vllm#43559 / #47194): the corruption is gated on MTP × prefix-caching
+      # together. Trade vs the alternative (MTP off): this keeps MTP's ~2x decode
+      # and loses only prefix-cache reuse. GATED ON the multi-turn probe passing.
+      - --no-enable-prefix-caching
      - --enable-chunked-prefill
      - --limit-mm-per-prompt
      - '{"image": 4}'
@@ -95,34 +102,34 @@ services:
      - --enable-auto-tool-choice
      - --tool-call-parser
      - qwen3_coder
-      # ⚠️ MTP / SPECULATIVE DECODING DISABLED 2026-08-17 — DO NOT re-enable
-      # without re-testing multi-turn coherence past ~2k tokens.
+      # MTP RE-ENABLED 2026-08-17, PAIRED WITH --no-enable-prefix-caching above.
      #
      # ROOT CAUSE (isolated 2026-08-16, operator-confirmed): `qwen3_5_mtp`
      # speculative decoding CORRUPTS Qwen3.8-27B output in multi-turn
-      # conversations once cumulative context passes ~2,000 tokens. The draft
-      # head emits bad tokens that get accepted, and the response degenerates
-      # into CONTEXT-BLEEDING — stitching prior turns' content into the current
-      # answer (a "describe durian" reply that contained the Krebs-cycle and
-      # winter answers from earlier turns), then collapsing to a few words.
+      # conversations once cumulative context passes ~2,000 tokens — CONTEXT-
+      # BLEEDING (a "describe durian" reply that contained the Krebs-cycle and
+      # winter answers from earlier turns), then collapse. Confirmed by two
+      # cross-frontier research peers as a KNOWN upstream bug: the Gated-DeltaNet
+      # recurrent state cannot roll back on a partial draft-accept, so the SSM
+      # state corrupts (vllm#47087 symptom-twin, #43559 fix lineage). It is
+      # architectural — vLLM, SGLang, and llama.cpp mainline all share it — and
+      # NOT depth-dependent (n=1 and n=2 both corrupt, so num_speculative_tokens=1
+      # is NOT a fix).
      #
-      # Proven model-INDEPENDENT across every Qwen3.8-27B quant we serve (AEON
-      # W4A4, unsloth FP8-attn, our in-house mixed) and FIXED by disabling MTP:
-      # the identical model stays coherent through 4k+ tokens with this off.
-      # It is MTP-on-Qwen3.8 specifically — Qwen3.6 (char-rp-reasoning) runs the
-      # same qwen3_5_mtp method and is CLEAN, so the 3.6 MTP head/graft is fine
-      # and the 3.8 one is not (suspects: the bf16 MTP graft, or spec depth 3).
+      # THE MITIGATION (this config): the corruption is gated on MTP × prefix-
+      # caching TOGETHER (#43559/#47194). Disabling prefix caching forces the GDN
+      # cache into a mode where the buggy align-path is inert, so MTP can stay on.
+      # Keeps MTP's ~2x decode; costs only prefix-cache reuse. The proper upstream
+      # fix (#51113) is in vLLM main / v0.27.2rc0 only — not in a stable release
+      # as of this date, so we do not jump the fleet gateway to an RC.
      #
-      # COST: no speculative decoding = ~half decode tok/s. Accepted as the
-      # known-good until the 3.8 MTP is fixed.
-      #
-      # BEFORE RE-ENABLING: run the multi-turn probe in
-      # services/gen-seat-mixed-quant/bench/ (varied 7-turn series, watch t5-t7
-      # past 2k tokens for the collapse). First thing worth trying is
-      # num_speculative_tokens=1 (shallowest draft, least divergence). Restore by
-      # uncommenting the two lines below.
-      # - --speculative-config
-      # - '{"method": "${GEN_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${GEN_SPEC_TOKENS:-3}}'
+      # ⚠️ GATED ON THE MULTI-TURN PROBE. If the 7-turn varied series
+      # (services/gen-seat-mixed-quant/bench/) still collapses past ~2k tokens
+      # with this config, APC-off did not hold on our stack — revert to MTP OFF
+      # (comment these two lines, restore --enable-prefix-caching) which is the
+      # verified known-good.
+      - --speculative-config
+      - '{"method": "${GEN_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${GEN_SPEC_TOKENS:-3}}'
    deploy:
      resources:
        reservations: