vh
d28a371049
Root cause of the multi-day degeneration hunt, operator-confirmed: the AEON NVFP4 W4A4 quant (sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4, full W4A4 incl. attention) went degenerate ~15-20% of generations in real multi-turn use and forced regenerates. MTP, prefix-caching, and the gateway all merely AMPLIFIED it, which is why MTP-off, APC-off, and the vLLM #51113 fix each 'helped' a synthetic probe without fixing it -- three plausible false root-causes, each passing one clean run then failing in real use. The fix was the WEIGHTS: the in-house JonathanColetti/Heretic mixed NVFP4+FP8 build (qwen38-27b-uncensored-nvfp4-mixed, FP8 attention not W4A4, same base, same MTP) is coherent through long multi-turn with MTP ON. W4A4 *attention* was the defect; FP8 attention is not. This commit: - GEN_MODEL -> the mixed FP8-attn build (primary gen until DavidAU 3.8 lands) - GEN_IMAGE pinned to vllm/vllm-openai:nightly-311b3513... (v0.27.2rc1.dev150, carries #51113; pinned by sha so it does not drift on the next pull) - AEON weights PURGED from /tank (no-good), safety-checked not-in-use first - playbook 3.8: the stochastic-W4A4-degeneration lesson + isolate-weights-early + do-not-declare-a-fix-from-one-probe (it validated three non-fixes) AEON is re-pullable from HF if ever needed, but the operator ruled it no-good.
26 lines
1.0 KiB
Bash
26 lines
1.0 KiB
Bash
# gen-seat tunables — fleet `gen` seat (ana-ml2 GPU 0, :8015). Edit here, never commit.
|
|
GEN_IMAGE=vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0
|
|
API_KEY=
|
|
GEN_GPU_ID=0
|
|
|
|
GEN_CONTAINER_NAME=vllm-gen
|
|
GEN_PORT=8015
|
|
GEN_SERVED_NAME=qwen3.8-27b-uncensored
|
|
GEN_SERVED_NAME_THINK=qwen3.8-27b-uncensored-thinking
|
|
GEN_MODEL=/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed
|
|
GEN_QUANT=compressed-tensors
|
|
# 0.43 not 0.45: the mixed build's weights are 5.2 GB smaller, and at 0.45 the
|
|
# seat absorbed that slack as extra KV, leaving meromero-charrp 0.18 GiB short
|
|
# of its 0.52 budget on the shared GPU0 (it crash-looped). 0.43 still gives gen
|
|
# 422K tokens of KV = 1.6x its 262K context. Both seats: 89.8/97.9 GB.
|
|
GEN_GPU_MEM_UTIL=0.43
|
|
GEN_MAX_MODEL_LEN=262144
|
|
GEN_MAX_NUM_SEQS=16
|
|
GEN_KV_CACHE_DTYPE=fp8
|
|
GEN_REASONING_PARSER=qwen3
|
|
GEN_SPEC_METHOD=qwen3_5_mtp
|
|
# 3 is the measured optimum, not a default: swept n=2/3/4/5 on this seat ->
|
|
# 77.1 / 80.1 / 78.7 / 75.9 tok/s. Higher n trades acceptance for draft width
|
|
# and loses. Do not re-chase.
|
|
GEN_SPEC_TOKENS=3
|