docs(quant): correct 3.8 — TWO real causes, not a lone defective quant

Operator correction to the prior 3.8 framing (d28a371), which over-blamed
AEON and dismissed the vLLM bug as a mere amplifier. Both were real and
compounded:

- Cause 1 (real, upstream): the qwen3_5_mtp x GDN partial-accept bug
  (#51113), architectural across vLLM/SGLang/llama.cpp, genuinely improved
  by the nightly fix -- not just an amplifier.
- Cause 2 (real, quant): AEON is FULL W4A4 (A4 activations on attention),
  the bottom of the KNOWN activation-precision gradient already in 1
  (W4A4 < W4+FP8 < W4+bf16) -- mildly subpar, not 'defective'. On top of
  Cause 1 it degenerated ~15-20% of real multi-turn generations.

The mixed FP8-attention build sits a rung up that gradient and is coherent;
a W4+bf16 build would be higher still at a prefill cost. Process lessons
retained (two causes mask each other; stochastic degeneration is invisible
to n=1 probes; isolate weights in parallel with serving flags -- but the
weight swap alone would NOT have found the real vLLM bug).
This commit is contained in:
2026-08-17 01:25:37 -07:00
parent d28a371049
commit 2f2bbce73d
+42 -30
View File
@@ -131,40 +131,52 @@ There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not
modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix
(needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4.
### 3.8 ⭐⭐ A bad W4A4 quant degenerates STOCHASTICALLY — and masquerades as everything else
### 3.8 ⭐⭐ Multi-turn degeneration from TWO real compounding causes — how they masked each other
The single most expensive misdiagnosis this project has had. A **defective NVFP4
W4A4 quant** (`sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4`, full
W4A4 incl. attention) went degenerate **~15-20% of generations** in real
multi-turn use — collapse, cross-turn content bleed, early-ending — forcing a
regenerate. It was **purged 2026-08-17** as no-good.
The most expensive diagnosis this project has had, because there were **two real
causes at once** and each partial fix moved the needle enough to look like *the*
answer. Recorded precisely because the first write-up of this section
over-attributed it to the quant alone; that was wrong.
The reason it cost days: **every other layer AMPLIFIED it, so each looked like the
cause in turn.** MTP made it worse (so MTP-off "fixed" a synthetic probe); prefix
caching made it worse (so APC-off "fixed" another probe); the vLLM #51113 mamba
fix made it "significantly better." Each partial-improvement produced a plausible
false root-cause that passed a clean run and then failed in real use. The actual
cause was the **weights**: swapping to the FP8-attention mixed build
(`qwen38-27b-uncensored-nvfp4-mixed`, same base model, same MTP, same nightly, MTP
ON) is **coherent**. W4A4 *attention* was the defect; FP8 attention is not.
**Cause 1 (real, upstream): the vLLM `qwen3_5_mtp` × Gated-DeltaNet bug.**
Confirmed by two cross-frontier peers and the tracker (vllm#47087 symptom-twin,
#43559 fix lineage, #51113 fix): the GDN recurrent state cannot roll back on a
partial draft-accept, so speculative decoding corrupts it, worse with context.
Architectural — vLLM/SGLang/llama.cpp mainline all shared it. **Genuinely fixed
enough** by moving to vLLM **nightly** (`v0.27.2rc1.dev150+`, carries #51113):
the operator reported it "significantly better" — this was a real bug, not just
an amplifier.
**Lessons, dearly bought:**
1. **Stochastic degeneration (15-20%) is nearly invisible to a small synthetic
probe.** A single 7-turn run passes ~4 times in 5. n=1 "clean" proves nothing;
this class of bug needs many runs or the operator's real high-volume use. Do
NOT declare a fix from one passing probe it validated three non-fixes here.
2. **Isolate the WEIGHTS early.** When a quant degenerates, swap to a *different
quant of the same base* before chasing serving flags (MTP, prefix-caching,
engine version). Had the AEON→mixed weight swap come first, the serving-flag
rabbit holes (MTP, APC, SGLang, nightly) would have been unnecessary.
3. **W4A4 on attention is high-risk for coherence, independent of the acceptance
gate.** AEON passed every static gate — abliteration 4/4, surface 6/6, a 36k
needle, 52% MTP acceptance — and was still stochastically broken in
conversation. Prefer FP8 attention (the mixed recipe, §2) over full W4A4; if
W4A4 attention is used, gate on *many* real multi-turn generations.
**Cause 2 (real, quant): full W4A4 is mildly subpar, per the known gradient.**
`sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4` is **full** W4A4 — 4-bit
*activations* on attention too, the bottom of the activation-precision ordering
already in §1: **W4A4 (A4) < W4+FP8 (A8) < W4+bf16 (A16)**. Not "defective," just
lowest-fidelity; on top of Cause 1 it degenerated ~15-20% of real multi-turn
generations. The FP8-attention **mixed** build (`qwen38-27b-uncensored-nvfp4-mixed`,
same base, same MTP, same nightly) sits a rung up that gradient and is coherent.
AEON was purged 2026-08-17 (operator ruled it no-good; re-pullable from HF).
Current primary gen: the mixed FP8-attention build on vLLM nightly with MTP,
until the DavidAU Qwen3.8 lands.
**Why it cost days — and the process lessons that stand:**
1. **Two real causes compound and mask each other.** Each mitigation (MTP-off,
APC-off, the nightly #51113 fix) partially helped, so each looked like the fix
and then failed in real use. When a mitigation "helps but doesn't fix," suspect
a *second* cause rather than a wrong one.
2. **Stochastic degeneration (~15-20%) is nearly invisible to a small synthetic
probe** — a 7-turn run passes ~4 in 5. n=1 "clean" proves nothing; this class
needs many runs or the operator's real high-volume use. Three non-fixes were
"validated" by a single clean probe here.
3. **Isolate the WEIGHTS in parallel with the serving flags, not after.** Swapping
to a different quant of the same base (AEON→mixed) is what finally separated
Cause 2 from Cause 1; doing it earlier would have shortened the hunt. But note
it would NOT have found Cause 1 — the vLLM bug was real and needed the nightly.
4. **Prefer FP8 attention (the §2 mixed recipe) over full W4A4** for a coherence-
sensitive seat. AEON passed every static gate (abliteration 4/4, surface 6/6, a
36k needle, 52% acceptance) and was still the lower-fidelity of the two.
Current primary gen: the mixed FP8-attention build on pinned vLLM nightly with
MTP, until the DavidAU Qwen3.8 lands. A W4+bf16 (W4A16) build would be higher
fidelity still (§1) at a prefill cost — an option if the mixed build ever proves
marginal.
### 3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn