fix(gen-seat): disable MTP — qwen3_5_mtp corrupts Qwen3.8-27B multi-turn past ~2k tokens
Root cause of the long-hunted 'gen goes degenerate in conversation',
isolated 2026-08-16 and operator-confirmed. qwen3_5_mtp speculative
decoding corrupts Qwen3.8-27B output once cumulative multi-turn context
passes ~2,000 tokens: the draft head's bad tokens get accepted and the
reply degenerates into CONTEXT-BLEEDING (a 'describe durian' answer that
contained the Krebs-cycle and winter replies from earlier turns), then
collapses to a few words.
Isolation, each step measured on the varied 7-turn probe:
- not the gateway (identical input -> gateway == direct; echo intact)
- not presence_penalty (1.5/0.5/0.0 all collapse), not temperature
(1.0 collapses harder), not repetition (varied unrelated topics
collapse identically -> it is context length, not template-lock)
- model-INDEPENDENT across all three Qwen3.8-27B quants we serve
(AEON W4A4, unsloth FP8-attn, in-house mixed)
- Qwen3.6 (char-rp-reasoning) and Gemma-4 (char-rp) are CLEAN
- DECISIVE: same Qwen3.8 model + same conversation, MTP OFF -> coherent
through 4k+ tokens, no bleed. MTP is the cause.
Qwen3.6 runs the same qwen3_5_mtp method and is clean, so the 3.6 MTP
head/graft is fine and the 3.8 one is not (suspects: the bf16 MTP graft,
or spec depth 3). COST: ~half decode tok/s without spec decoding.
Accepted as known-good until the 3.8 MTP is fixed; first thing to try on
re-enable is num_speculative_tokens=1. Seat restored to AEON W4A4 (the
production choice); verified clean on the varied series after this change.
This commit is contained in:
@@ -95,8 +95,34 @@ services:
|
||||
- --enable-auto-tool-choice
|
||||
- --tool-call-parser
|
||||
- qwen3_coder
|
||||
- --speculative-config
|
||||
- '{"method": "${GEN_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${GEN_SPEC_TOKENS:-3}}'
|
||||
# ⚠️ MTP / SPECULATIVE DECODING DISABLED 2026-08-17 — DO NOT re-enable
|
||||
# without re-testing multi-turn coherence past ~2k tokens.
|
||||
#
|
||||
# ROOT CAUSE (isolated 2026-08-16, operator-confirmed): `qwen3_5_mtp`
|
||||
# speculative decoding CORRUPTS Qwen3.8-27B output in multi-turn
|
||||
# conversations once cumulative context passes ~2,000 tokens. The draft
|
||||
# head emits bad tokens that get accepted, and the response degenerates
|
||||
# into CONTEXT-BLEEDING — stitching prior turns' content into the current
|
||||
# answer (a "describe durian" reply that contained the Krebs-cycle and
|
||||
# winter answers from earlier turns), then collapsing to a few words.
|
||||
#
|
||||
# Proven model-INDEPENDENT across every Qwen3.8-27B quant we serve (AEON
|
||||
# W4A4, unsloth FP8-attn, our in-house mixed) and FIXED by disabling MTP:
|
||||
# the identical model stays coherent through 4k+ tokens with this off.
|
||||
# It is MTP-on-Qwen3.8 specifically — Qwen3.6 (char-rp-reasoning) runs the
|
||||
# same qwen3_5_mtp method and is CLEAN, so the 3.6 MTP head/graft is fine
|
||||
# and the 3.8 one is not (suspects: the bf16 MTP graft, or spec depth 3).
|
||||
#
|
||||
# COST: no speculative decoding = ~half decode tok/s. Accepted as the
|
||||
# known-good until the 3.8 MTP is fixed.
|
||||
#
|
||||
# BEFORE RE-ENABLING: run the multi-turn probe in
|
||||
# services/gen-seat-mixed-quant/bench/ (varied 7-turn series, watch t5-t7
|
||||
# past 2k tokens for the collapse). First thing worth trying is
|
||||
# num_speculative_tokens=1 (shallowest draft, least divergence). Restore by
|
||||
# uncommenting the two lines below.
|
||||
# - --speculative-config
|
||||
# - '{"method": "${GEN_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${GEN_SPEC_TOKENS:-3}}'
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
|
||||
Reference in New Issue
Block a user