From d28a3710496e1ba24592fe36e0f9a7f05dc7f9f3 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 17 Aug 2026 00:59:50 -0700 Subject: [PATCH] =?UTF-8?q?fix(gen-seat):=20AEON=20W4A4=20was=20the=20defe?= =?UTF-8?q?ct=20=E2=80=94=20purged;=20mixed=20FP8-attn=20build=20is=20prim?= =?UTF-8?q?ary=20gen?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause of the multi-day degeneration hunt, operator-confirmed: the AEON NVFP4 W4A4 quant (sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4, full W4A4 incl. attention) went degenerate ~15-20% of generations in real multi-turn use and forced regenerates. MTP, prefix-caching, and the gateway all merely AMPLIFIED it, which is why MTP-off, APC-off, and the vLLM #51113 fix each 'helped' a synthetic probe without fixing it -- three plausible false root-causes, each passing one clean run then failing in real use. The fix was the WEIGHTS: the in-house JonathanColetti/Heretic mixed NVFP4+FP8 build (qwen38-27b-uncensored-nvfp4-mixed, FP8 attention not W4A4, same base, same MTP) is coherent through long multi-turn with MTP ON. W4A4 *attention* was the defect; FP8 attention is not. This commit: - GEN_MODEL -> the mixed FP8-attn build (primary gen until DavidAU 3.8 lands) - GEN_IMAGE pinned to vllm/vllm-openai:nightly-311b3513... (v0.27.2rc1.dev150, carries #51113; pinned by sha so it does not drift on the next pull) - AEON weights PURGED from /tank (no-good), safety-checked not-in-use first - playbook 3.8: the stochastic-W4A4-degeneration lesson + isolate-weights-early + do-not-declare-a-fix-from-one-probe (it validated three non-fixes) AEON is re-pullable from HF if ever needed, but the operator ruled it no-good. --- docs/pfi/model-quantization-playbook.md | 35 +++++++++++++++++++++++++ stacks/gen-seat/.env.example | 2 +- stacks/gen-seat/compose.yaml | 16 ++++++++--- 3 files changed, 48 insertions(+), 5 deletions(-) diff --git a/docs/pfi/model-quantization-playbook.md b/docs/pfi/model-quantization-playbook.md index 733f380..20b24aa 100644 --- a/docs/pfi/model-quantization-playbook.md +++ b/docs/pfi/model-quantization-playbook.md @@ -131,6 +131,41 @@ There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix (needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4. +### 3.8 ⭐⭐ A bad W4A4 quant degenerates STOCHASTICALLY — and masquerades as everything else + +The single most expensive misdiagnosis this project has had. A **defective NVFP4 +W4A4 quant** (`sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4`, full +W4A4 incl. attention) went degenerate **~15-20% of generations** in real +multi-turn use — collapse, cross-turn content bleed, early-ending — forcing a +regenerate. It was **purged 2026-08-17** as no-good. + +The reason it cost days: **every other layer AMPLIFIED it, so each looked like the +cause in turn.** MTP made it worse (so MTP-off "fixed" a synthetic probe); prefix +caching made it worse (so APC-off "fixed" another probe); the vLLM #51113 mamba +fix made it "significantly better." Each partial-improvement produced a plausible +false root-cause that passed a clean run and then failed in real use. The actual +cause was the **weights**: swapping to the FP8-attention mixed build +(`qwen38-27b-uncensored-nvfp4-mixed`, same base model, same MTP, same nightly, MTP +ON) is **coherent**. W4A4 *attention* was the defect; FP8 attention is not. + +**Lessons, dearly bought:** +1. **Stochastic degeneration (15-20%) is nearly invisible to a small synthetic + probe.** A single 7-turn run passes ~4 times in 5. n=1 "clean" proves nothing; + this class of bug needs many runs or the operator's real high-volume use. Do + NOT declare a fix from one passing probe — it validated three non-fixes here. +2. **Isolate the WEIGHTS early.** When a quant degenerates, swap to a *different + quant of the same base* before chasing serving flags (MTP, prefix-caching, + engine version). Had the AEON→mixed weight swap come first, the serving-flag + rabbit holes (MTP, APC, SGLang, nightly) would have been unnecessary. +3. **W4A4 on attention is high-risk for coherence, independent of the acceptance + gate.** AEON passed every static gate — abliteration 4/4, surface 6/6, a 36k + needle, 52% MTP acceptance — and was still stochastically broken in + conversation. Prefer FP8 attention (the mixed recipe, §2) over full W4A4; if + W4A4 attention is used, gate on *many* real multi-turn generations. + +Current primary gen: the mixed FP8-attention build on vLLM nightly with MTP, +until the DavidAU Qwen3.8 lands. + ### 3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn §3.3 is about *losing* the head (0% acceptance, silent). This is the opposite and diff --git a/stacks/gen-seat/.env.example b/stacks/gen-seat/.env.example index 4be3b61..d90cf1c 100644 --- a/stacks/gen-seat/.env.example +++ b/stacks/gen-seat/.env.example @@ -1,5 +1,5 @@ # gen-seat tunables — fleet `gen` seat (ana-ml2 GPU 0, :8015). Edit here, never commit. -GEN_IMAGE=vllm/vllm-openai:latest +GEN_IMAGE=vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0 API_KEY= GEN_GPU_ID=0 diff --git a/stacks/gen-seat/compose.yaml b/stacks/gen-seat/compose.yaml index 4d05f9e..84b5b0a 100644 --- a/stacks/gen-seat/compose.yaml +++ b/stacks/gen-seat/compose.yaml @@ -126,10 +126,18 @@ services: # in a stable release — so we do not jump the fleet gateway to an RC. # Cost of MTP-off: ~half decode tok/s. Accepted as the known-good. # - # TO RE-ENABLE (only with real-use verification): uncomment the two lines - # below. Do NOT trust a synthetic probe alone — it passed and was wrong. - # - --speculative-config - # - '{"method": "${GEN_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${GEN_SPEC_TOKENS:-3}}' + # MTP ON — PRIMARY GEN CONFIG (resolved 2026-08-17, operator-confirmed + # coherent). The whole multi-day degeneration hunt root-caused to the AEON + # W4A4 quant being DEFECTIVE: it went degenerate ~15-20% of generations and + # forced a regenerate. MTP / prefix-caching / gateway all merely AMPLIFIED + # it, which is why MTP-off and the vLLM #51113 fix each "helped" without + # fixing it. This build — the in-house JonathanColetti/Heretic mixed + # NVFP4+FP8 weights (FP8 attention, NOT W4A4) on vLLM nightly (#51113 fix) + # with MTP ON — is coherent through long multi-turn in real use. AEON + # (qwen38-27b-aeon-ultimate-nvfp4) was PURGED as no-good. Primary gen until + # the DavidAU Qwen3.8 lands. + - --speculative-config + - '{"method": "${GEN_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${GEN_SPEC_TOKENS:-3}}' deploy: resources: reservations: