docs(quant): record the MTP-corrupts-Qwen3.8-multi-turn lesson (playbook 3.7)

The single hardest bug of the night, and invisible to the existing
acceptance gate: a LOADED, healthy-accepting MTP head still corrupts
Qwen3.8-27B multi-turn output past ~2k cumulative tokens (length collapse +
cross-turn content bleed), while single-turn is perfect. Model-independent
across all three of our Qwen3.8 quants; Qwen3.6 on the same qwen3_5_mtp
method is clean; disabling MTP fixes it. New rule: gate MTP on a multi-turn
coherence probe, not just single-shot acceptance.
This commit is contained in:
vh
2026-08-16 22:24:19 -07:00
parent 7bd38b33b5
commit a8ed6e7428
+33
View File
@@ -131,6 +131,39 @@ There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not
modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix
(needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4.
### 3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn
§3.3 is about *losing* the head (0% acceptance, silent). This is the opposite and
worse failure: the head loads, acceptance looks healthy, single-turn output is
perfect — and then it **corrupts multi-turn conversations** once cumulative context
passes **~2,000 tokens**. The reply collapses in length *and* bleeds earlier turns
into the current answer (a "describe durian" reply that contained the Krebs-cycle
and winter answers from three turns back). Single-turn probes and the acceptance
gate (§5) **do not catch it** — it only appears as accumulated context grows.
Isolated 2026-08-16 (operator-confirmed), each step measured on a fixed 7-turn probe:
- **Not the serving gateway, not sampling, not repetition/template.** Identical
input gateway-vs-direct behaves the same; presence_penalty 1.5/0.5/0.0 all
collapse; higher temperature collapses harder; a conversation of *unrelated*
topics collapses at the same ~2k tokens as a repetitive one → it is context-
length-driven, not template lock-in.
- **Model-independent across every Qwen3.8-27B quant** (AEON W4A4, unsloth
FP8-attn, our in-house mixed) — so not a quant-brand or scheme artifact.
- **DECISIVE: same model + same conversation, MTP OFF → coherent through 4k+
tokens, zero bleed.** Toggle it back on → collapse returns. MTP is the cause.
**Qwen3.6-27B running the same `qwen3_5_mtp` method is CLEAN.** So the 3.6 MTP
head/graft is fine and the 3.8 one is not — suspects: the bf16 graft being subtly
wrong for the 3.8 head, or the vLLM `qwen3_5_mtp` impl diverging at `num_speculative_tokens=3`.
Open upstream question (queried dvalin/bil-smithy 2026-08-17).
**Rule: gate MTP on a MULTI-TURN coherence probe, not just single-shot acceptance.**
Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens
for length-collapse and cross-turn bleed. If it corrupts: disable MTP (costs ~half
decode tok/s) or try `num_speculative_tokens=1` (shallowest draft, least divergence
— untested as of this writing). The gen seat runs MTP **off** as of 2026-08-17.
### 3.4 Toolchain version deadlocks
Both directions have burned us, so the resolution is: **use llm-compressor / compressed-tensors,