docs(quant): correct 3.8 — TWO real causes, not a lone defective quant

Operator correction to the prior 3.8 framing (d28a371), which over-blamed
AEON and dismissed the vLLM bug as a mere amplifier. Both were real and
compounded:

- Cause 1 (real, upstream): the qwen3_5_mtp x GDN partial-accept bug
  (#51113), architectural across vLLM/SGLang/llama.cpp, genuinely improved
  by the nightly fix -- not just an amplifier.
- Cause 2 (real, quant): AEON is FULL W4A4 (A4 activations on attention),
  the bottom of the KNOWN activation-precision gradient already in 1
  (W4A4 < W4+FP8 < W4+bf16) -- mildly subpar, not 'defective'. On top of
  Cause 1 it degenerated ~15-20% of real multi-turn generations.

The mixed FP8-attention build sits a rung up that gradient and is coherent;
a W4+bf16 build would be higher still at a prefill cost. Process lessons
retained (two causes mask each other; stochastic degeneration is invisible
to n=1 probes; isolate weights in parallel with serving flags -- but the
weight swap alone would NOT have found the real vLLM bug).
This commit is contained in:
vh
2026-08-17 01:25:37 -07:00
parent d28a371049
commit 2f2bbce73d
+42 -30
View File
@@ -131,40 +131,52 @@ There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not
modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix
(needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4. (needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4.
### 3.8 ⭐⭐ A bad W4A4 quant degenerates STOCHASTICALLY — and masquerades as everything else ### 3.8 ⭐⭐ Multi-turn degeneration from TWO real compounding causes — how they masked each other
The single most expensive misdiagnosis this project has had. A **defective NVFP4 The most expensive diagnosis this project has had, because there were **two real
W4A4 quant** (`sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4`, full causes at once** and each partial fix moved the needle enough to look like *the*
W4A4 incl. attention) went degenerate **~15-20% of generations** in real answer. Recorded precisely because the first write-up of this section
multi-turn use — collapse, cross-turn content bleed, early-ending — forcing a over-attributed it to the quant alone; that was wrong.
regenerate. It was **purged 2026-08-17** as no-good.
The reason it cost days: **every other layer AMPLIFIED it, so each looked like the **Cause 1 (real, upstream): the vLLM `qwen3_5_mtp` × Gated-DeltaNet bug.**
cause in turn.** MTP made it worse (so MTP-off "fixed" a synthetic probe); prefix Confirmed by two cross-frontier peers and the tracker (vllm#47087 symptom-twin,
caching made it worse (so APC-off "fixed" another probe); the vLLM #51113 mamba #43559 fix lineage, #51113 fix): the GDN recurrent state cannot roll back on a
fix made it "significantly better." Each partial-improvement produced a plausible partial draft-accept, so speculative decoding corrupts it, worse with context.
false root-cause that passed a clean run and then failed in real use. The actual Architectural — vLLM/SGLang/llama.cpp mainline all shared it. **Genuinely fixed
cause was the **weights**: swapping to the FP8-attention mixed build enough** by moving to vLLM **nightly** (`v0.27.2rc1.dev150+`, carries #51113):
(`qwen38-27b-uncensored-nvfp4-mixed`, same base model, same MTP, same nightly, MTP the operator reported it "significantly better" — this was a real bug, not just
ON) is **coherent**. W4A4 *attention* was the defect; FP8 attention is not. an amplifier.
**Lessons, dearly bought:** **Cause 2 (real, quant): full W4A4 is mildly subpar, per the known gradient.**
1. **Stochastic degeneration (15-20%) is nearly invisible to a small synthetic `sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4` is **full** W4A4 — 4-bit
probe.** A single 7-turn run passes ~4 times in 5. n=1 "clean" proves nothing; *activations* on attention too, the bottom of the activation-precision ordering
this class of bug needs many runs or the operator's real high-volume use. Do already in §1: **W4A4 (A4) < W4+FP8 (A8) < W4+bf16 (A16)**. Not "defective," just
NOT declare a fix from one passing probe — it validated three non-fixes here. lowest-fidelity; on top of Cause 1 it degenerated ~15-20% of real multi-turn
2. **Isolate the WEIGHTS early.** When a quant degenerates, swap to a *different generations. The FP8-attention **mixed** build (`qwen38-27b-uncensored-nvfp4-mixed`,
quant of the same base* before chasing serving flags (MTP, prefix-caching, same base, same MTP, same nightly) sits a rung up that gradient and is coherent.
engine version). Had the AEON→mixed weight swap come first, the serving-flag AEON was purged 2026-08-17 (operator ruled it no-good; re-pullable from HF).
rabbit holes (MTP, APC, SGLang, nightly) would have been unnecessary.
3. **W4A4 on attention is high-risk for coherence, independent of the acceptance
gate.** AEON passed every static gate — abliteration 4/4, surface 6/6, a 36k
needle, 52% MTP acceptance — and was still stochastically broken in
conversation. Prefer FP8 attention (the mixed recipe, §2) over full W4A4; if
W4A4 attention is used, gate on *many* real multi-turn generations.
Current primary gen: the mixed FP8-attention build on vLLM nightly with MTP, **Why it cost days — and the process lessons that stand:**
until the DavidAU Qwen3.8 lands. 1. **Two real causes compound and mask each other.** Each mitigation (MTP-off,
APC-off, the nightly #51113 fix) partially helped, so each looked like the fix
and then failed in real use. When a mitigation "helps but doesn't fix," suspect
a *second* cause rather than a wrong one.
2. **Stochastic degeneration (~15-20%) is nearly invisible to a small synthetic
probe** — a 7-turn run passes ~4 in 5. n=1 "clean" proves nothing; this class
needs many runs or the operator's real high-volume use. Three non-fixes were
"validated" by a single clean probe here.
3. **Isolate the WEIGHTS in parallel with the serving flags, not after.** Swapping
to a different quant of the same base (AEON→mixed) is what finally separated
Cause 2 from Cause 1; doing it earlier would have shortened the hunt. But note
it would NOT have found Cause 1 — the vLLM bug was real and needed the nightly.
4. **Prefer FP8 attention (the §2 mixed recipe) over full W4A4** for a coherence-
sensitive seat. AEON passed every static gate (abliteration 4/4, surface 6/6, a
36k needle, 52% acceptance) and was still the lower-fidelity of the two.
Current primary gen: the mixed FP8-attention build on pinned vLLM nightly with
MTP, until the DavidAU Qwen3.8 lands. A W4+bf16 (W4A16) build would be higher
fidelity still (§1) at a prefill cost — an option if the mixed build ever proves
marginal.
### 3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn ### 3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn