diff --git a/docs/pfi/model-quantization-playbook.md b/docs/pfi/model-quantization-playbook.md index 20b24aa..b6bcb7c 100644 --- a/docs/pfi/model-quantization-playbook.md +++ b/docs/pfi/model-quantization-playbook.md @@ -131,40 +131,52 @@ There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix (needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4. -### 3.8 ⭐⭐ A bad W4A4 quant degenerates STOCHASTICALLY — and masquerades as everything else +### 3.8 ⭐⭐ Multi-turn degeneration from TWO real compounding causes — how they masked each other -The single most expensive misdiagnosis this project has had. A **defective NVFP4 -W4A4 quant** (`sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4`, full -W4A4 incl. attention) went degenerate **~15-20% of generations** in real -multi-turn use — collapse, cross-turn content bleed, early-ending — forcing a -regenerate. It was **purged 2026-08-17** as no-good. +The most expensive diagnosis this project has had, because there were **two real +causes at once** and each partial fix moved the needle enough to look like *the* +answer. Recorded precisely because the first write-up of this section +over-attributed it to the quant alone; that was wrong. -The reason it cost days: **every other layer AMPLIFIED it, so each looked like the -cause in turn.** MTP made it worse (so MTP-off "fixed" a synthetic probe); prefix -caching made it worse (so APC-off "fixed" another probe); the vLLM #51113 mamba -fix made it "significantly better." Each partial-improvement produced a plausible -false root-cause that passed a clean run and then failed in real use. The actual -cause was the **weights**: swapping to the FP8-attention mixed build -(`qwen38-27b-uncensored-nvfp4-mixed`, same base model, same MTP, same nightly, MTP -ON) is **coherent**. W4A4 *attention* was the defect; FP8 attention is not. +**Cause 1 (real, upstream): the vLLM `qwen3_5_mtp` × Gated-DeltaNet bug.** +Confirmed by two cross-frontier peers and the tracker (vllm#47087 symptom-twin, +#43559 fix lineage, #51113 fix): the GDN recurrent state cannot roll back on a +partial draft-accept, so speculative decoding corrupts it, worse with context. +Architectural — vLLM/SGLang/llama.cpp mainline all shared it. **Genuinely fixed +enough** by moving to vLLM **nightly** (`v0.27.2rc1.dev150+`, carries #51113): +the operator reported it "significantly better" — this was a real bug, not just +an amplifier. -**Lessons, dearly bought:** -1. **Stochastic degeneration (15-20%) is nearly invisible to a small synthetic - probe.** A single 7-turn run passes ~4 times in 5. n=1 "clean" proves nothing; - this class of bug needs many runs or the operator's real high-volume use. Do - NOT declare a fix from one passing probe — it validated three non-fixes here. -2. **Isolate the WEIGHTS early.** When a quant degenerates, swap to a *different - quant of the same base* before chasing serving flags (MTP, prefix-caching, - engine version). Had the AEON→mixed weight swap come first, the serving-flag - rabbit holes (MTP, APC, SGLang, nightly) would have been unnecessary. -3. **W4A4 on attention is high-risk for coherence, independent of the acceptance - gate.** AEON passed every static gate — abliteration 4/4, surface 6/6, a 36k - needle, 52% MTP acceptance — and was still stochastically broken in - conversation. Prefer FP8 attention (the mixed recipe, §2) over full W4A4; if - W4A4 attention is used, gate on *many* real multi-turn generations. +**Cause 2 (real, quant): full W4A4 is mildly subpar, per the known gradient.** +`sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4` is **full** W4A4 — 4-bit +*activations* on attention too, the bottom of the activation-precision ordering +already in §1: **W4A4 (A4) < W4+FP8 (A8) < W4+bf16 (A16)**. Not "defective," just +lowest-fidelity; on top of Cause 1 it degenerated ~15-20% of real multi-turn +generations. The FP8-attention **mixed** build (`qwen38-27b-uncensored-nvfp4-mixed`, +same base, same MTP, same nightly) sits a rung up that gradient and is coherent. +AEON was purged 2026-08-17 (operator ruled it no-good; re-pullable from HF). -Current primary gen: the mixed FP8-attention build on vLLM nightly with MTP, -until the DavidAU Qwen3.8 lands. +**Why it cost days — and the process lessons that stand:** +1. **Two real causes compound and mask each other.** Each mitigation (MTP-off, + APC-off, the nightly #51113 fix) partially helped, so each looked like the fix + and then failed in real use. When a mitigation "helps but doesn't fix," suspect + a *second* cause rather than a wrong one. +2. **Stochastic degeneration (~15-20%) is nearly invisible to a small synthetic + probe** — a 7-turn run passes ~4 in 5. n=1 "clean" proves nothing; this class + needs many runs or the operator's real high-volume use. Three non-fixes were + "validated" by a single clean probe here. +3. **Isolate the WEIGHTS in parallel with the serving flags, not after.** Swapping + to a different quant of the same base (AEON→mixed) is what finally separated + Cause 2 from Cause 1; doing it earlier would have shortened the hunt. But note + it would NOT have found Cause 1 — the vLLM bug was real and needed the nightly. +4. **Prefer FP8 attention (the §2 mixed recipe) over full W4A4** for a coherence- + sensitive seat. AEON passed every static gate (abliteration 4/4, surface 6/6, a + 36k needle, 52% acceptance) and was still the lower-fidelity of the two. + +Current primary gen: the mixed FP8-attention build on pinned vLLM nightly with +MTP, until the DavidAU Qwen3.8 lands. A W4+bf16 (W4A16) build would be higher +fidelity still (§1) at a prefill cost — an option if the mixed build ever proves +marginal. ### 3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn