fix(gen-seat): AEON W4A4 was the defect — purged; mixed FP8-attn build is primary gen
Root cause of the multi-day degeneration hunt, operator-confirmed: the AEON NVFP4 W4A4 quant (sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4, full W4A4 incl. attention) went degenerate ~15-20% of generations in real multi-turn use and forced regenerates. MTP, prefix-caching, and the gateway all merely AMPLIFIED it, which is why MTP-off, APC-off, and the vLLM #51113 fix each 'helped' a synthetic probe without fixing it -- three plausible false root-causes, each passing one clean run then failing in real use. The fix was the WEIGHTS: the in-house JonathanColetti/Heretic mixed NVFP4+FP8 build (qwen38-27b-uncensored-nvfp4-mixed, FP8 attention not W4A4, same base, same MTP) is coherent through long multi-turn with MTP ON. W4A4 *attention* was the defect; FP8 attention is not. This commit: - GEN_MODEL -> the mixed FP8-attn build (primary gen until DavidAU 3.8 lands) - GEN_IMAGE pinned to vllm/vllm-openai:nightly-311b3513... (v0.27.2rc1.dev150, carries #51113; pinned by sha so it does not drift on the next pull) - AEON weights PURGED from /tank (no-good), safety-checked not-in-use first - playbook 3.8: the stochastic-W4A4-degeneration lesson + isolate-weights-early + do-not-declare-a-fix-from-one-probe (it validated three non-fixes) AEON is re-pullable from HF if ever needed, but the operator ruled it no-good.
This commit is contained in:
@@ -1,5 +1,5 @@
|
||||
# gen-seat tunables — fleet `gen` seat (ana-ml2 GPU 0, :8015). Edit here, never commit.
|
||||
GEN_IMAGE=vllm/vllm-openai:latest
|
||||
GEN_IMAGE=vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0
|
||||
API_KEY=
|
||||
GEN_GPU_ID=0
|
||||
|
||||
|
||||
@@ -126,10 +126,18 @@ services:
|
||||
# in a stable release — so we do not jump the fleet gateway to an RC.
|
||||
# Cost of MTP-off: ~half decode tok/s. Accepted as the known-good.
|
||||
#
|
||||
# TO RE-ENABLE (only with real-use verification): uncomment the two lines
|
||||
# below. Do NOT trust a synthetic probe alone — it passed and was wrong.
|
||||
# - --speculative-config
|
||||
# - '{"method": "${GEN_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${GEN_SPEC_TOKENS:-3}}'
|
||||
# MTP ON — PRIMARY GEN CONFIG (resolved 2026-08-17, operator-confirmed
|
||||
# coherent). The whole multi-day degeneration hunt root-caused to the AEON
|
||||
# W4A4 quant being DEFECTIVE: it went degenerate ~15-20% of generations and
|
||||
# forced a regenerate. MTP / prefix-caching / gateway all merely AMPLIFIED
|
||||
# it, which is why MTP-off and the vLLM #51113 fix each "helped" without
|
||||
# fixing it. This build — the in-house JonathanColetti/Heretic mixed
|
||||
# NVFP4+FP8 weights (FP8 attention, NOT W4A4) on vLLM nightly (#51113 fix)
|
||||
# with MTP ON — is coherent through long multi-turn in real use. AEON
|
||||
# (qwen38-27b-aeon-ultimate-nvfp4) was PURGED as no-good. Primary gen until
|
||||
# the DavidAU Qwen3.8 lands.
|
||||
- --speculative-config
|
||||
- '{"method": "${GEN_SPEC_METHOD:-qwen3_5_mtp}", "num_speculative_tokens": ${GEN_SPEC_TOKENS:-3}}'
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
|
||||
Reference in New Issue
Block a user