vh
d47dd10795
Operator call: the incumbent abliterated model was the first one we could
find, not an optimised pick. sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-
UNCENSORED-NVFP4 (base AEON-7 BF16, abliterix-abliterated, Apache-2.0),
byte-verified at /tank/aimodels/qwen38-27b-aeon-ultimate-nvfp4.
Measured on the same harness, same GPU, cache-busted per playbook 5.
Baseline was RE-measured live before the swap rather than trusted:
incumbent (W4A4+FP8 mixed) AEON (W4A4)
decode bs=1 94.09 tok/s 104.22 +10.8%
MTP acceptance 47.7% 52.3% +4.6pp
abliteration 4/4 4/4
surface 6/6 6/6
weights 22.5 GB 20.6 GB -8.4%
AEON concurrency: conc=1 98.48 tok/s aggregate; conc=6 381.29 aggregate /
63.55 per-stream, MTP holding 50.6% under load.
Surface 6/6 includes vision (image-judge rides this seat) and a 36k-token
needle retrieval, which was the specific risk in going full-W4A4 -- the
packager only validated 32k, and W4A4 long-context collapse is in our own
notes from the Granite work. It held.
reasoning_effort: the AEON template defaults to xhigh (template line 47),
and at xhigh this model can spend its entire budget inside <think> and
emit no answer -- a silent-empty-response hazard for the automated
summarizer/classifier consumers. Seat now pins the default to medium via
--default-chat-template-kwargs, per-request overridable. Override PROVEN
live: chat_template_kwargs.reasoning_effort=bogus returns HTTP 400
carrying the template's own exception text, so caller values genuinely
reach the template and invalid ones fail loudly rather than silently
falling back. Empty GEN_REASONING_EFFORT omits the flag for models that do
not read the kwarg -- the Qwen3.6 line ignores it entirely, where setting
it would be a false lever.
All 7 aliases verified routing. Rollback is one .env line; the previous
build is untouched at /tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed.
TWO GAPS, declared:
- Incumbent concurrency was never captured before the swap (I baselined
bs=1 only), so the conc=1/6 figures have no same-hardware comparator.
- Perplexity NOT measured. eval_quality correctly refused it: under
--speculative-config prompt_logprobs come back ~uniform (median rank
~130k), playbook trap 2. A real PPL number needs both seats served
without spec-decode.
Adds concbench.py (concurrent throughput; wall-clock aggregate, not
sum-of-rates, and delta-based MTP accounting).
29 lines
667 B
JSON
29 lines
667 B
JSON
{
|
|
"tag": "aeon-2026-08-16",
|
|
"model": "qwen3.8-27b-uncensored",
|
|
"tok_s_median": 104.21612205891445,
|
|
"tok_s_mean": 103.84095336442763,
|
|
"tok_s_min": 93.73198544134813,
|
|
"tok_s_max": 111.54540137941143,
|
|
"mtp_accept_median": 0.5232481178125259,
|
|
"rates": [
|
|
101.61959217374029,
|
|
99.09260773424464,
|
|
106.33688838455522,
|
|
105.85190253179012,
|
|
93.73198544134813,
|
|
109.9689076842924,
|
|
102.58034158603878,
|
|
111.54540137941143
|
|
],
|
|
"accs": [
|
|
0.5020833333333333,
|
|
0.48261758691206547,
|
|
0.5460526315789473,
|
|
0.5359477124183006,
|
|
0.4393063583815029,
|
|
0.5736961451247166,
|
|
0.510548523206751,
|
|
0.5862068965517241
|
|
]
|
|
} |