docs(quant): correct 3.8 — TWO real causes, not a lone defective quant
Operator correction to the prior 3.8 framing (d28a371), which over-blamed
AEON and dismissed the vLLM bug as a mere amplifier. Both were real and
compounded:
- Cause 1 (real, upstream): the qwen3_5_mtp x GDN partial-accept bug
(#51113), architectural across vLLM/SGLang/llama.cpp, genuinely improved
by the nightly fix -- not just an amplifier.
- Cause 2 (real, quant): AEON is FULL W4A4 (A4 activations on attention),
the bottom of the KNOWN activation-precision gradient already in 1
(W4A4 < W4+FP8 < W4+bf16) -- mildly subpar, not 'defective'. On top of
Cause 1 it degenerated ~15-20% of real multi-turn generations.
The mixed FP8-attention build sits a rung up that gradient and is coherent;
a W4+bf16 build would be higher still at a prefill cost. Process lessons
retained (two causes mask each other; stochastic degeneration is invisible
to n=1 probes; isolate weights in parallel with serving flags -- but the
weight swap alone would NOT have found the real vLLM bug).
This commit is contained in:
@@ -131,40 +131,52 @@ There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not
|
||||
modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix
|
||||
(needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4.
|
||||
|
||||
### 3.8 ⭐⭐ A bad W4A4 quant degenerates STOCHASTICALLY — and masquerades as everything else
|
||||
### 3.8 ⭐⭐ Multi-turn degeneration from TWO real compounding causes — how they masked each other
|
||||
|
||||
The single most expensive misdiagnosis this project has had. A **defective NVFP4
|
||||
W4A4 quant** (`sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4`, full
|
||||
W4A4 incl. attention) went degenerate **~15-20% of generations** in real
|
||||
multi-turn use — collapse, cross-turn content bleed, early-ending — forcing a
|
||||
regenerate. It was **purged 2026-08-17** as no-good.
|
||||
The most expensive diagnosis this project has had, because there were **two real
|
||||
causes at once** and each partial fix moved the needle enough to look like *the*
|
||||
answer. Recorded precisely because the first write-up of this section
|
||||
over-attributed it to the quant alone; that was wrong.
|
||||
|
||||
The reason it cost days: **every other layer AMPLIFIED it, so each looked like the
|
||||
cause in turn.** MTP made it worse (so MTP-off "fixed" a synthetic probe); prefix
|
||||
caching made it worse (so APC-off "fixed" another probe); the vLLM #51113 mamba
|
||||
fix made it "significantly better." Each partial-improvement produced a plausible
|
||||
false root-cause that passed a clean run and then failed in real use. The actual
|
||||
cause was the **weights**: swapping to the FP8-attention mixed build
|
||||
(`qwen38-27b-uncensored-nvfp4-mixed`, same base model, same MTP, same nightly, MTP
|
||||
ON) is **coherent**. W4A4 *attention* was the defect; FP8 attention is not.
|
||||
**Cause 1 (real, upstream): the vLLM `qwen3_5_mtp` × Gated-DeltaNet bug.**
|
||||
Confirmed by two cross-frontier peers and the tracker (vllm#47087 symptom-twin,
|
||||
#43559 fix lineage, #51113 fix): the GDN recurrent state cannot roll back on a
|
||||
partial draft-accept, so speculative decoding corrupts it, worse with context.
|
||||
Architectural — vLLM/SGLang/llama.cpp mainline all shared it. **Genuinely fixed
|
||||
enough** by moving to vLLM **nightly** (`v0.27.2rc1.dev150+`, carries #51113):
|
||||
the operator reported it "significantly better" — this was a real bug, not just
|
||||
an amplifier.
|
||||
|
||||
**Lessons, dearly bought:**
|
||||
1. **Stochastic degeneration (15-20%) is nearly invisible to a small synthetic
|
||||
probe.** A single 7-turn run passes ~4 times in 5. n=1 "clean" proves nothing;
|
||||
this class of bug needs many runs or the operator's real high-volume use. Do
|
||||
NOT declare a fix from one passing probe — it validated three non-fixes here.
|
||||
2. **Isolate the WEIGHTS early.** When a quant degenerates, swap to a *different
|
||||
quant of the same base* before chasing serving flags (MTP, prefix-caching,
|
||||
engine version). Had the AEON→mixed weight swap come first, the serving-flag
|
||||
rabbit holes (MTP, APC, SGLang, nightly) would have been unnecessary.
|
||||
3. **W4A4 on attention is high-risk for coherence, independent of the acceptance
|
||||
gate.** AEON passed every static gate — abliteration 4/4, surface 6/6, a 36k
|
||||
needle, 52% MTP acceptance — and was still stochastically broken in
|
||||
conversation. Prefer FP8 attention (the mixed recipe, §2) over full W4A4; if
|
||||
W4A4 attention is used, gate on *many* real multi-turn generations.
|
||||
**Cause 2 (real, quant): full W4A4 is mildly subpar, per the known gradient.**
|
||||
`sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4` is **full** W4A4 — 4-bit
|
||||
*activations* on attention too, the bottom of the activation-precision ordering
|
||||
already in §1: **W4A4 (A4) < W4+FP8 (A8) < W4+bf16 (A16)**. Not "defective," just
|
||||
lowest-fidelity; on top of Cause 1 it degenerated ~15-20% of real multi-turn
|
||||
generations. The FP8-attention **mixed** build (`qwen38-27b-uncensored-nvfp4-mixed`,
|
||||
same base, same MTP, same nightly) sits a rung up that gradient and is coherent.
|
||||
AEON was purged 2026-08-17 (operator ruled it no-good; re-pullable from HF).
|
||||
|
||||
Current primary gen: the mixed FP8-attention build on vLLM nightly with MTP,
|
||||
until the DavidAU Qwen3.8 lands.
|
||||
**Why it cost days — and the process lessons that stand:**
|
||||
1. **Two real causes compound and mask each other.** Each mitigation (MTP-off,
|
||||
APC-off, the nightly #51113 fix) partially helped, so each looked like the fix
|
||||
and then failed in real use. When a mitigation "helps but doesn't fix," suspect
|
||||
a *second* cause rather than a wrong one.
|
||||
2. **Stochastic degeneration (~15-20%) is nearly invisible to a small synthetic
|
||||
probe** — a 7-turn run passes ~4 in 5. n=1 "clean" proves nothing; this class
|
||||
needs many runs or the operator's real high-volume use. Three non-fixes were
|
||||
"validated" by a single clean probe here.
|
||||
3. **Isolate the WEIGHTS in parallel with the serving flags, not after.** Swapping
|
||||
to a different quant of the same base (AEON→mixed) is what finally separated
|
||||
Cause 2 from Cause 1; doing it earlier would have shortened the hunt. But note
|
||||
it would NOT have found Cause 1 — the vLLM bug was real and needed the nightly.
|
||||
4. **Prefer FP8 attention (the §2 mixed recipe) over full W4A4** for a coherence-
|
||||
sensitive seat. AEON passed every static gate (abliteration 4/4, surface 6/6, a
|
||||
36k needle, 52% acceptance) and was still the lower-fidelity of the two.
|
||||
|
||||
Current primary gen: the mixed FP8-attention build on pinned vLLM nightly with
|
||||
MTP, until the DavidAU Qwen3.8 lands. A W4+bf16 (W4A16) build would be higher
|
||||
fidelity still (§1) at a prefill cost — an option if the mixed build ever proves
|
||||
marginal.
|
||||
|
||||
### 3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn
|
||||
|
||||
|
||||
Reference in New Issue
Block a user