vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(gen-seat): MTP mitigation — disable prefix caching, keep MTP (speed restored, multi-turn clean)

Browse Source 
The qwen3_5_mtp corruption (playbook 3.7) is gated on MTP x prefix-caching
TOGETHER (vllm#43559 / #47194), per both cross-frontier peers. Disabling
prefix caching (--no-enable-prefix-caching; vLLM V1 defaults it ON, so the
explicit --no- form is required) forces the GDN cache into a mode where the
partial-accept align-path bug is inert, so MTP can stay on.

Verified on our stack (AEON W4A4): MTP on + prefix-caching off -> the 7-turn
varied series stays coherent through 3.9k tokens, zero cross-turn bleed, at
104.6 tok/s / 53.6% acceptance -- the FULL MTP speedup restored (vs ~half
with MTP off), losing only prefix-cache reuse. All 7 aliases route.

Ruled out on the way: num_speculative_tokens=1 (corruption is
depth-independent, n=1 and n=2 both corrupt); switching to SGLang (vLLM /
SGLang / llama.cpp mainline all share the architectural GDN-rollback bug).
Proper upstream fix (#51113) is in main / v0.27.2rc0 only, not stable, so we
hold at APC-off rather than jump the fleet gateway to an RC.

Supersedes the MTP-off config from 7bd38b3.
This commit is contained in:
vh
2026-08-16 22:44:14 -07:00
parent a8ed6e7428
commit 63a3cb2d86
2 changed files with 49 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/pfi/model-quantization-playbook.md
+18 -3
View File
@@ -160,9 +160,24 @@ Open upstream question (queried dvalin/bil-smithy 2026-08-17).
**Rule: gate MTP on a MULTI-TURN coherence probe, not just single-shot acceptance.**
Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens
for length-collapse and cross-turn bleed. If it corrupts: disable MTP (costs ~half
decode tok/s) or try `num_speculative_tokens=1` (shallowest draft, least divergence
— untested as of this writing). The gen seat runs MTP **off** as of 2026-08-17.
for length-collapse and cross-turn bleed.
**THE MITIGATION (resolved 2026-08-17): disable prefix caching, keep MTP.** The
corruption is gated on MTP × prefix-caching *together* (vllm#43559 / #47194) — with
`--no-enable-prefix-caching` the GDN cache runs in a mode where the buggy
partial-accept align-path is inert. Confirmed on our stack: AEON W4A4, MTP on +
prefix-caching off → the 7-turn varied series stays coherent through 3.9k tokens,
zero bleed, at **104.6 tok/s / 53.6% acceptance** — i.e. the FULL MTP speedup back
(vs ~half with MTP off), losing only prefix-cache reuse. The gen seat runs this
config as of 2026-08-17.
Things that do **not** work, ruled out: `num_speculative_tokens=1` (corruption is
depth-independent — reproduces at n=1 and n=2, deterministically probed upstream);
switching engine (vLLM / SGLang / llama.cpp mainline all share the GDN-rollback
bug — it is architectural). The proper upstream fix (vllm#51113) is in `main` /
`v0.27.2rc0` only — not in a stable release, so we hold at APC-off until it lands.
Two cross-frontier peers (dvalin/bil-smithy) confirmed the bug class and pointed
at the open symptom-twin issue #47087.
### 3.4 Toolchain version deadlocks