vh
7bd38b33b5
Root cause of the long-hunted 'gen goes degenerate in conversation',
isolated 2026-08-16 and operator-confirmed. qwen3_5_mtp speculative
decoding corrupts Qwen3.8-27B output once cumulative multi-turn context
passes ~2,000 tokens: the draft head's bad tokens get accepted and the
reply degenerates into CONTEXT-BLEEDING (a 'describe durian' answer that
contained the Krebs-cycle and winter replies from earlier turns), then
collapses to a few words.
Isolation, each step measured on the varied 7-turn probe:
- not the gateway (identical input -> gateway == direct; echo intact)
- not presence_penalty (1.5/0.5/0.0 all collapse), not temperature
(1.0 collapses harder), not repetition (varied unrelated topics
collapse identically -> it is context length, not template-lock)
- model-INDEPENDENT across all three Qwen3.8-27B quants we serve
(AEON W4A4, unsloth FP8-attn, in-house mixed)
- Qwen3.6 (char-rp-reasoning) and Gemma-4 (char-rp) are CLEAN
- DECISIVE: same Qwen3.8 model + same conversation, MTP OFF -> coherent
through 4k+ tokens, no bleed. MTP is the cause.
Qwen3.6 runs the same qwen3_5_mtp method and is clean, so the 3.6 MTP
head/graft is fine and the 3.8 one is not (suspects: the bf16 MTP graft,
or spec depth 3). COST: ~half decode tok/s without spec decoding.
Accepted as known-good until the 3.8 MTP is fixed; first thing to try on
re-enable is num_speculative_tokens=1. Seat restored to AEON W4A4 (the
production choice); verified clean on the varied series after this change.
gen-seat — the fleet gen seat (ana-ml2 GPU0, :8015)
Serves qwen3.8-27b-uncensored (JonathanColetti/Qwen3.8-27B-Uncensored,
Heretic-abliterated Qwen3.8-27B, vision-intact, 262K context) plus the
-thinking served-name for the reasoning split.
Backs 7 LiteLLM aliases: gen, gen-reasoning, summarizer,
summarizer-large, classifier, image-judge, qwen-image-bench. Treat any
change here as fleet-wide.
- Quant: in-house mixed-precision — NVFP4 W4A4 for layers 0-55 MLPs,
FP8 W8A8 for attention /
linear_attn/lm_head/ layers 56-63 MLPs, FP8 KV. ~94.5 tok/s decode at bs=1, MTP n=3 @ ~48% acceptance. - Speculative decoding:
qwen3_5_mtp,num_speculative_tokens=3(measured optimum — see the quant service README). - Tunables:
.envon the host. ⚠ it is mode 0600 / lkraven-owned, so everydocker composecall against this stack needssudo— without it compose cannot read.env, fails withpermission denied, and leaves the old container running while appearing to have succeeded.
Pipeline, acceptance gate, and rollback
All of it — the recipe, the re:^mtp.* foot-gun, the benchmark harness, the GPU0
budget interaction with meromero-charrp, and the rollback command — lives in
services/gen-seat-mixed-quant/README.md.
Deploy
scripts/deploy-stack.sh ana-ml2 gen-seat # diffs vs live, prompts y/N
# on host:
ssh infra-ops@10.250.50.54
cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen