fix(gen-seat): disable MTP — qwen3_5_mtp corrupts Qwen3.8-27B multi-turn past ~2k tokens

Root cause of the long-hunted 'gen goes degenerate in conversation',
isolated 2026-08-16 and operator-confirmed. qwen3_5_mtp speculative
decoding corrupts Qwen3.8-27B output once cumulative multi-turn context
passes ~2,000 tokens: the draft head's bad tokens get accepted and the
reply degenerates into CONTEXT-BLEEDING (a 'describe durian' answer that
contained the Krebs-cycle and winter replies from earlier turns), then
collapses to a few words.

Isolation, each step measured on the varied 7-turn probe:
  - not the gateway (identical input -> gateway == direct; echo intact)
  - not presence_penalty (1.5/0.5/0.0 all collapse), not temperature
    (1.0 collapses harder), not repetition (varied unrelated topics
    collapse identically -> it is context length, not template-lock)
  - model-INDEPENDENT across all three Qwen3.8-27B quants we serve
    (AEON W4A4, unsloth FP8-attn, in-house mixed)
  - Qwen3.6 (char-rp-reasoning) and Gemma-4 (char-rp) are CLEAN
  - DECISIVE: same Qwen3.8 model + same conversation, MTP OFF -> coherent
    through 4k+ tokens, no bleed. MTP is the cause.

Qwen3.6 runs the same qwen3_5_mtp method and is clean, so the 3.6 MTP
head/graft is fine and the 3.8 one is not (suspects: the bf16 MTP graft,
or spec depth 3). COST: ~half decode tok/s without spec decoding.
Accepted as known-good until the 3.8 MTP is fixed; first thing to try on
re-enable is num_speculative_tokens=1. Seat restored to AEON W4A4 (the
production choice); verified clean on the varied series after this change.
This commit is contained in:
2026-08-16 22:22:58 -07:00
parent 01b5ad93ed
commit 7bd38b33b5
+28 -2
View File
@@ -95,8 +95,34 @@ services:
- --enable-auto-tool-choice
- --tool-call-parser
- qwen3_coder
- --speculative-config
- '{"method": "${GEN_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${GEN_SPEC_TOKENS:-3}}'
# ⚠️ MTP / SPECULATIVE DECODING DISABLED 2026-08-17 — DO NOT re-enable
# without re-testing multi-turn coherence past ~2k tokens.
#
# ROOT CAUSE (isolated 2026-08-16, operator-confirmed): `qwen3_5_mtp`
# speculative decoding CORRUPTS Qwen3.8-27B output in multi-turn
# conversations once cumulative context passes ~2,000 tokens. The draft
# head emits bad tokens that get accepted, and the response degenerates
# into CONTEXT-BLEEDING — stitching prior turns' content into the current
# answer (a "describe durian" reply that contained the Krebs-cycle and
# winter answers from earlier turns), then collapsing to a few words.
#
# Proven model-INDEPENDENT across every Qwen3.8-27B quant we serve (AEON
# W4A4, unsloth FP8-attn, our in-house mixed) and FIXED by disabling MTP:
# the identical model stays coherent through 4k+ tokens with this off.
# It is MTP-on-Qwen3.8 specifically — Qwen3.6 (char-rp-reasoning) runs the
# same qwen3_5_mtp method and is CLEAN, so the 3.6 MTP head/graft is fine
# and the 3.8 one is not (suspects: the bf16 MTP graft, or spec depth 3).
#
# COST: no speculative decoding = ~half decode tok/s. Accepted as the
# known-good until the 3.8 MTP is fixed.
#
# BEFORE RE-ENABLING: run the multi-turn probe in
# services/gen-seat-mixed-quant/bench/ (varied 7-turn series, watch t5-t7
# past 2k tokens for the collapse). First thing worth trying is
# num_speculative_tokens=1 (shallowest draft, least divergence). Restore by
# uncommenting the two lines below.
# - --speculative-config
# - '{"method": "${GEN_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${GEN_SPEC_TOKENS:-3}}'
deploy:
resources:
reservations: