vh
63a3cb2d86
The qwen3_5_mtp corruption (playbook 3.7) is gated on MTP x prefix-caching
TOGETHER (vllm#43559 / #47194), per both cross-frontier peers. Disabling
prefix caching (--no-enable-prefix-caching; vLLM V1 defaults it ON, so the
explicit --no- form is required) forces the GDN cache into a mode where the
partial-accept align-path bug is inert, so MTP can stay on.
Verified on our stack (AEON W4A4): MTP on + prefix-caching off -> the 7-turn
varied series stays coherent through 3.9k tokens, zero cross-turn bleed, at
104.6 tok/s / 53.6% acceptance -- the FULL MTP speedup restored (vs ~half
with MTP off), losing only prefix-cache reuse. All 7 aliases route.
Ruled out on the way: num_speculative_tokens=1 (corruption is
depth-independent, n=1 and n=2 both corrupt); switching to SGLang (vLLM /
SGLang / llama.cpp mainline all share the architectural GDN-rollback bug).
Proper upstream fix (#51113) is in main / v0.27.2rc0 only, not stable, so we
hold at APC-off rather than jump the fleet gateway to an RC.
Supersedes the MTP-off config from 7bd38b3.
gen-seat — the fleet gen seat (ana-ml2 GPU0, :8015)
Serves qwen3.8-27b-uncensored (JonathanColetti/Qwen3.8-27B-Uncensored,
Heretic-abliterated Qwen3.8-27B, vision-intact, 262K context) plus the
-thinking served-name for the reasoning split.
Backs 7 LiteLLM aliases: gen, gen-reasoning, summarizer,
summarizer-large, classifier, image-judge, qwen-image-bench. Treat any
change here as fleet-wide.
- Quant: in-house mixed-precision — NVFP4 W4A4 for layers 0-55 MLPs,
FP8 W8A8 for attention /
linear_attn/lm_head/ layers 56-63 MLPs, FP8 KV. ~94.5 tok/s decode at bs=1, MTP n=3 @ ~48% acceptance. - Speculative decoding:
qwen3_5_mtp,num_speculative_tokens=3(measured optimum — see the quant service README). - Tunables:
.envon the host. ⚠ it is mode 0600 / lkraven-owned, so everydocker composecall against this stack needssudo— without it compose cannot read.env, fails withpermission denied, and leaves the old container running while appearing to have succeeded.
Pipeline, acceptance gate, and rollback
All of it — the recipe, the re:^mtp.* foot-gun, the benchmark harness, the GPU0
budget interaction with meromero-charrp, and the rollback command — lives in
services/gen-seat-mixed-quant/README.md.
Deploy
scripts/deploy-stack.sh ana-ml2 gen-seat # diffs vs live, prompts y/N
# on host:
ssh infra-ops@10.250.50.54
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen