fix(gen-seat): AEON W4A4 was the defect — purged; mixed FP8-attn build is primary gen

Root cause of the multi-day degeneration hunt, operator-confirmed: the AEON
NVFP4 W4A4 quant (sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4,
full W4A4 incl. attention) went degenerate ~15-20% of generations in real
multi-turn use and forced regenerates. MTP, prefix-caching, and the gateway
all merely AMPLIFIED it, which is why MTP-off, APC-off, and the vLLM #51113
fix each 'helped' a synthetic probe without fixing it -- three plausible
false root-causes, each passing one clean run then failing in real use.

The fix was the WEIGHTS: the in-house JonathanColetti/Heretic mixed
NVFP4+FP8 build (qwen38-27b-uncensored-nvfp4-mixed, FP8 attention not W4A4,
same base, same MTP) is coherent through long multi-turn with MTP ON. W4A4
*attention* was the defect; FP8 attention is not.

This commit:
- GEN_MODEL -> the mixed FP8-attn build (primary gen until DavidAU 3.8 lands)
- GEN_IMAGE pinned to vllm/vllm-openai:nightly-311b3513... (v0.27.2rc1.dev150,
  carries #51113; pinned by sha so it does not drift on the next pull)
- AEON weights PURGED from /tank (no-good), safety-checked not-in-use first
- playbook 3.8: the stochastic-W4A4-degeneration lesson + isolate-weights-early
  + do-not-declare-a-fix-from-one-probe (it validated three non-fixes)

AEON is re-pullable from HF if ever needed, but the operator ruled it no-good.
This commit is contained in:
vh
2026-08-17 00:59:50 -07:00
parent 1f5b2cbcb0
commit d28a371049
3 changed files with 48 additions and 5 deletions
+35
View File
@@ -131,6 +131,41 @@ There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not
modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix
(needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4.
### 3.8 ⭐⭐ A bad W4A4 quant degenerates STOCHASTICALLY — and masquerades as everything else
The single most expensive misdiagnosis this project has had. A **defective NVFP4
W4A4 quant** (`sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4`, full
W4A4 incl. attention) went degenerate **~15-20% of generations** in real
multi-turn use — collapse, cross-turn content bleed, early-ending — forcing a
regenerate. It was **purged 2026-08-17** as no-good.
The reason it cost days: **every other layer AMPLIFIED it, so each looked like the
cause in turn.** MTP made it worse (so MTP-off "fixed" a synthetic probe); prefix
caching made it worse (so APC-off "fixed" another probe); the vLLM #51113 mamba
fix made it "significantly better." Each partial-improvement produced a plausible
false root-cause that passed a clean run and then failed in real use. The actual
cause was the **weights**: swapping to the FP8-attention mixed build
(`qwen38-27b-uncensored-nvfp4-mixed`, same base model, same MTP, same nightly, MTP
ON) is **coherent**. W4A4 *attention* was the defect; FP8 attention is not.
**Lessons, dearly bought:**
1. **Stochastic degeneration (15-20%) is nearly invisible to a small synthetic
probe.** A single 7-turn run passes ~4 times in 5. n=1 "clean" proves nothing;
this class of bug needs many runs or the operator's real high-volume use. Do
NOT declare a fix from one passing probe — it validated three non-fixes here.
2. **Isolate the WEIGHTS early.** When a quant degenerates, swap to a *different
quant of the same base* before chasing serving flags (MTP, prefix-caching,
engine version). Had the AEON→mixed weight swap come first, the serving-flag
rabbit holes (MTP, APC, SGLang, nightly) would have been unnecessary.
3. **W4A4 on attention is high-risk for coherence, independent of the acceptance
gate.** AEON passed every static gate — abliteration 4/4, surface 6/6, a 36k
needle, 52% MTP acceptance — and was still stochastically broken in
conversation. Prefer FP8 attention (the mixed recipe, §2) over full W4A4; if
W4A4 attention is used, gate on *many* real multi-turn generations.
Current primary gen: the mixed FP8-attention build on vLLM nightly with MTP,
until the DavidAU Qwen3.8 lands.
### 3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn
§3.3 is about *losing* the head (0% acceptance, silent). This is the opposite and