Root cause of the multi-day degeneration hunt, operator-confirmed: the AEON
NVFP4 W4A4 quant (sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4,
full W4A4 incl. attention) went degenerate ~15-20% of generations in real
multi-turn use and forced regenerates. MTP, prefix-caching, and the gateway
all merely AMPLIFIED it, which is why MTP-off, APC-off, and the vLLM #51113
fix each 'helped' a synthetic probe without fixing it -- three plausible
false root-causes, each passing one clean run then failing in real use.
The fix was the WEIGHTS: the in-house JonathanColetti/Heretic mixed
NVFP4+FP8 build (qwen38-27b-uncensored-nvfp4-mixed, FP8 attention not W4A4,
same base, same MTP) is coherent through long multi-turn with MTP ON. W4A4
*attention* was the defect; FP8 attention is not.
This commit:
- GEN_MODEL -> the mixed FP8-attn build (primary gen until DavidAU 3.8 lands)
- GEN_IMAGE pinned to vllm/vllm-openai:nightly-311b3513... (v0.27.2rc1.dev150,
carries #51113; pinned by sha so it does not drift on the next pull)
- AEON weights PURGED from /tank (no-good), safety-checked not-in-use first
- playbook 3.8: the stochastic-W4A4-degeneration lesson + isolate-weights-early
+ do-not-declare-a-fix-from-one-probe (it validated three non-fixes)
AEON is re-pullable from HF if ever needed, but the operator ruled it no-good.
The MTP-on + prefix-caching-off mitigation (63a3cb2) passed synthetic
7-turn probes but the operator still saw severe degeneration in real use.
A passing synthetic probe is NOT sufficient evidence -- it under-covers
real workloads (content distribution, conversation depth). Reverted to the
verified known-good: MTP off, prefix caching on (the 7bd38b3 state), ~half
decode speed but coherent. Operator is driving it to re-confirm.
Lesson reinforced (the recurring one this session): do not trust a
synthetic reproduction to VALIDATE a fix for a bug that only manifests in
the operator's real usage -- it validated a non-fix twice tonight.
The qwen3_5_mtp corruption (playbook 3.7) is gated on MTP x prefix-caching
TOGETHER (vllm#43559 / #47194), per both cross-frontier peers. Disabling
prefix caching (--no-enable-prefix-caching; vLLM V1 defaults it ON, so the
explicit --no- form is required) forces the GDN cache into a mode where the
partial-accept align-path bug is inert, so MTP can stay on.
Verified on our stack (AEON W4A4): MTP on + prefix-caching off -> the 7-turn
varied series stays coherent through 3.9k tokens, zero cross-turn bleed, at
104.6 tok/s / 53.6% acceptance -- the FULL MTP speedup restored (vs ~half
with MTP off), losing only prefix-cache reuse. All 7 aliases route.
Ruled out on the way: num_speculative_tokens=1 (corruption is
depth-independent, n=1 and n=2 both corrupt); switching to SGLang (vLLM /
SGLang / llama.cpp mainline all share the architectural GDN-rollback bug).
Proper upstream fix (#51113) is in main / v0.27.2rc0 only, not stable, so we
hold at APC-off rather than jump the fleet gateway to an RC.
Supersedes the MTP-off config from 7bd38b3.
Root cause of the long-hunted 'gen goes degenerate in conversation',
isolated 2026-08-16 and operator-confirmed. qwen3_5_mtp speculative
decoding corrupts Qwen3.8-27B output once cumulative multi-turn context
passes ~2,000 tokens: the draft head's bad tokens get accepted and the
reply degenerates into CONTEXT-BLEEDING (a 'describe durian' answer that
contained the Krebs-cycle and winter replies from earlier turns), then
collapses to a few words.
Isolation, each step measured on the varied 7-turn probe:
- not the gateway (identical input -> gateway == direct; echo intact)
- not presence_penalty (1.5/0.5/0.0 all collapse), not temperature
(1.0 collapses harder), not repetition (varied unrelated topics
collapse identically -> it is context length, not template-lock)
- model-INDEPENDENT across all three Qwen3.8-27B quants we serve
(AEON W4A4, unsloth FP8-attn, in-house mixed)
- Qwen3.6 (char-rp-reasoning) and Gemma-4 (char-rp) are CLEAN
- DECISIVE: same Qwen3.8 model + same conversation, MTP OFF -> coherent
through 4k+ tokens, no bleed. MTP is the cause.
Qwen3.6 runs the same qwen3_5_mtp method and is clean, so the 3.6 MTP
head/graft is fine and the 3.8 one is not (suspects: the bf16 MTP graft,
or spec depth 3). COST: ~half decode tok/s without spec decoding.
Accepted as known-good until the 3.8 MTP is fixed; first thing to try on
re-enable is num_speculative_tokens=1. Seat restored to AEON W4A4 (the
production choice); verified clean on the varied series after this change.
Measured on the restored model: at the template default xhigh, reasoning
runs 4,529-5,532 chars on a 3-turn history and was observed spiking to
9,261; medium holds it to 2,602-3,283 with content length unchanged or
better. Per-request overridable; an invalid value 400s.
SCOPE CAVEAT, stated because I applied this while chasing the wrong path:
this is a NO-OP for the alias, which sends enable_thinking:false and
produces zero reasoning. It affects only. The operator's
reported multi-turn failure was on , so this does not address it.
Operator reports AEON-ULTIMATE degenerate on long MULTI-TURN conversations.
Restored the previous model and removed the reasoning_effort default in
the same change, so the model is the only variable differing from the
pre-trial state and the operator's comparison is clean.
My acceptance gate did not cover this failure mode and should have. Every
probe was SINGLE-TURN -- quickbench, concbench, surface_test, the
long-form smokes -- so a defect that only appears as conversation history
accumulates was structurally invisible to all of it. The gate measures
decode speed, MTP acceptance, abliteration survival, and a 36k needle, and
passes a model that degrades across turns.
Verified restored via docker inspect rather than the compose file:
/model -> qwen38-27b-uncensored-nvfp4-mixed, --default-chat-template-kwargs
absent, MTP drafter loaded, all 7 aliases answering.
AEON weights retained at /tank/aimodels/qwen38-27b-aeon-ultimate-nvfp4 for
diagnosis; bench artifacts stay in services/gen-seat-mixed-quant/bench/.
Its single-turn numbers were real (104.22 tok/s, 52.3% MTP, 4/4
abliteration, 6/6 surface) -- they were just measuring the wrong thing.
Operator call: the incumbent abliterated model was the first one we could
find, not an optimised pick. sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-
UNCENSORED-NVFP4 (base AEON-7 BF16, abliterix-abliterated, Apache-2.0),
byte-verified at /tank/aimodels/qwen38-27b-aeon-ultimate-nvfp4.
Measured on the same harness, same GPU, cache-busted per playbook 5.
Baseline was RE-measured live before the swap rather than trusted:
incumbent (W4A4+FP8 mixed) AEON (W4A4)
decode bs=1 94.09 tok/s 104.22 +10.8%
MTP acceptance 47.7% 52.3% +4.6pp
abliteration 4/4 4/4
surface 6/6 6/6
weights 22.5 GB 20.6 GB -8.4%
AEON concurrency: conc=1 98.48 tok/s aggregate; conc=6 381.29 aggregate /
63.55 per-stream, MTP holding 50.6% under load.
Surface 6/6 includes vision (image-judge rides this seat) and a 36k-token
needle retrieval, which was the specific risk in going full-W4A4 -- the
packager only validated 32k, and W4A4 long-context collapse is in our own
notes from the Granite work. It held.
reasoning_effort: the AEON template defaults to xhigh (template line 47),
and at xhigh this model can spend its entire budget inside <think> and
emit no answer -- a silent-empty-response hazard for the automated
summarizer/classifier consumers. Seat now pins the default to medium via
--default-chat-template-kwargs, per-request overridable. Override PROVEN
live: chat_template_kwargs.reasoning_effort=bogus returns HTTP 400
carrying the template's own exception text, so caller values genuinely
reach the template and invalid ones fail loudly rather than silently
falling back. Empty GEN_REASONING_EFFORT omits the flag for models that do
not read the kwarg -- the Qwen3.6 line ignores it entirely, where setting
it would be a false lever.
All 7 aliases verified routing. Rollback is one .env line; the previous
build is untouched at /tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed.
TWO GAPS, declared:
- Incumbent concurrency was never captured before the swap (I baselined
bs=1 only), so the conc=1/6 figures have no same-hardware comparator.
- Perplexity NOT measured. eval_quality correctly refused it: under
--speculative-config prompt_logprobs come back ~uniform (median rank
~130k), playbook trap 2. A real PPL number needs both seats served
without spec-decode.
Adds concbench.py (concurrent throughput; wall-clock aggregate, not
sum-of-rates, and delta-based MTP accounting).
Re-quantizes the fleet `gen` seat from weight-only NVFP4A16 to a
mixed-precision build: NVFP4 W4A4 for layers 0-55 MLPs, FP8 W8A8 for the
attention projections / linear_attn / lm_head / layers 56-63 MLPs, FP8 KV
cache. Replicates the scheme of unsloth/Qwen3.8-27B-NVFP4 on the
abliterated weights.
The queued task named this "W4A8" (NVFP4 weights + FP8 activations). That
checkpoint cannot be served: vLLM 0.24's compressed-tensors dispatcher
(compressed_tensors.py:704-713) accepts NVFP4 weights with either no input
quantization (W4A16, which forces the Marlin kernel) or NVFP4 input
quantization (W4A4) -- anything else, FP8 included, raises ValueError at
load. CompressedTensorsW4A8Fp8 is INT4 weights gated on an exact-sm90
check, so it is closed on Blackwell twice over. The ~20% intuition was
correct; the scheme name was not. Getting FP8 into the mix has to be done
per-layer-group.
Established the gain before spending GPU time: unsloth's build was already
on-box, so serving it as a probe measured +19.1% over our seat at identical
MTP acceptance -- a kernel-level result, no requant needed to learn it.
Measured, cache-busted, bs=1:
decode 80.12 -> 94.53 tok/s (+18.0%)
MTP acceptance 47.8% -> 47.7% (unchanged)
perplexity (n=6) 6.941 -> 7.059 (+1.7%)
abliteration 4/4 -> 4/4 (preserved)
weights on disk 27.7 -> 22.5 GB (-19%)
Surface test green on the live seat: plain chat, vision, tool calling,
thinking split, 36K-token needle retrieval, streaming. All 7 LiteLLM
aliases verified routing.
GEN_GPU_MEM_UTIL 0.45 -> 0.43: the new weights are 5.2 GB smaller, and at
0.45 the seat absorbed that slack as KV, leaving meromero-charrp 0.18 GiB
short of its budget on the shared GPU0 -- it crash-looped. Handing the
space back leaves gen 422K tokens of KV (1.6x its 262K context) and both
seats co-resident at 89.8/97.9 GB.
Also records two measured negatives so they are not re-chased:
GEN_SPEC_TOKENS is already optimal at 3 (swept 2/3/4/5 -> 77.1/80.1/78.7/
75.9 tok/s), and vLLM's prompt_logprobs are ~uniform while speculative
decoding is on, so perplexity must be measured with spec off.
Pipeline, acceptance harness and raw measurements land in
services/gen-seat-mixed-quant/. Rollback is one .env line; the previous
build is untouched at /tank/aimodels/qwen38-27b-uncensored-nvfp4.
vh