feat(gen-seat): swap to AEON-ULTIMATE NVFP4 W4A4; default reasoning_effort=medium
Operator call: the incumbent abliterated model was the first one we could
find, not an optimised pick. sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-
UNCENSORED-NVFP4 (base AEON-7 BF16, abliterix-abliterated, Apache-2.0),
byte-verified at /tank/aimodels/qwen38-27b-aeon-ultimate-nvfp4.
Measured on the same harness, same GPU, cache-busted per playbook 5.
Baseline was RE-measured live before the swap rather than trusted:
incumbent (W4A4+FP8 mixed) AEON (W4A4)
decode bs=1 94.09 tok/s 104.22 +10.8%
MTP acceptance 47.7% 52.3% +4.6pp
abliteration 4/4 4/4
surface 6/6 6/6
weights 22.5 GB 20.6 GB -8.4%
AEON concurrency: conc=1 98.48 tok/s aggregate; conc=6 381.29 aggregate /
63.55 per-stream, MTP holding 50.6% under load.
Surface 6/6 includes vision (image-judge rides this seat) and a 36k-token
needle retrieval, which was the specific risk in going full-W4A4 -- the
packager only validated 32k, and W4A4 long-context collapse is in our own
notes from the Granite work. It held.
reasoning_effort: the AEON template defaults to xhigh (template line 47),
and at xhigh this model can spend its entire budget inside <think> and
emit no answer -- a silent-empty-response hazard for the automated
summarizer/classifier consumers. Seat now pins the default to medium via
--default-chat-template-kwargs, per-request overridable. Override PROVEN
live: chat_template_kwargs.reasoning_effort=bogus returns HTTP 400
carrying the template's own exception text, so caller values genuinely
reach the template and invalid ones fail loudly rather than silently
falling back. Empty GEN_REASONING_EFFORT omits the flag for models that do
not read the kwarg -- the Qwen3.6 line ignores it entirely, where setting
it would be a false lever.
All 7 aliases verified routing. Rollback is one .env line; the previous
build is untouched at /tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed.
TWO GAPS, declared:
- Incumbent concurrency was never captured before the swap (I baselined
bs=1 only), so the conc=1/6 figures have no same-hardware comparator.
- Perplexity NOT measured. eval_quality correctly refused it: under
--speculative-config prompt_logprobs come back ~uniform (median rank
~130k), playbook trap 2. A real PPL number needs both seats served
without spec-decode.
Adds concbench.py (concurrent throughput; wall-clock aggregate, not
sum-of-rates, and delta-based MTP accounting).
This commit is contained in:
@@ -62,6 +62,21 @@ services:
|
||||
- '{"image": 4}'
|
||||
- --reasoning-parser
|
||||
- ${GEN_REASONING_PARSER:-qwen3}
|
||||
# Seat-level DEFAULT reasoning effort, per-request overridable via
|
||||
# chat_template_kwargs. The AEON-ULTIMATE chat template defaults to
|
||||
# `xhigh` (template line 47), and at xhigh this model can spend its ENTIRE
|
||||
# token budget inside <think> -- the block runs past 18k chars, sometimes
|
||||
# degenerating into repeating one line, and no answer is ever emitted.
|
||||
# That is a silent-empty-response hazard for the automated consumers on
|
||||
# this seat (`summarizer`, `classifier`), so the default is pinned down a
|
||||
# notch rather than left at the model's own.
|
||||
# Template accepts ONLY xhigh | medium | low and RAISES on anything else,
|
||||
# so a typo in a caller's chat_template_kwargs is a 400, not a silent
|
||||
# fallback. Empty value = omit the flag entirely (pre-AEON models such as
|
||||
# the Qwen3.6 line do not read this kwarg at all, and setting it there
|
||||
# would be a false lever).
|
||||
- --default-chat-template-kwargs
|
||||
- '{"reasoning_effort": "${GEN_REASONING_EFFORT:-medium}"}'
|
||||
- --enable-auto-tool-choice
|
||||
- --tool-call-parser
|
||||
- qwen3_coder
|
||||
|
||||
Reference in New Issue
Block a user