vh
d47dd10795
Operator call: the incumbent abliterated model was the first one we could
find, not an optimised pick. sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-
UNCENSORED-NVFP4 (base AEON-7 BF16, abliterix-abliterated, Apache-2.0),
byte-verified at /tank/aimodels/qwen38-27b-aeon-ultimate-nvfp4.
Measured on the same harness, same GPU, cache-busted per playbook 5.
Baseline was RE-measured live before the swap rather than trusted:
incumbent (W4A4+FP8 mixed) AEON (W4A4)
decode bs=1 94.09 tok/s 104.22 +10.8%
MTP acceptance 47.7% 52.3% +4.6pp
abliteration 4/4 4/4
surface 6/6 6/6
weights 22.5 GB 20.6 GB -8.4%
AEON concurrency: conc=1 98.48 tok/s aggregate; conc=6 381.29 aggregate /
63.55 per-stream, MTP holding 50.6% under load.
Surface 6/6 includes vision (image-judge rides this seat) and a 36k-token
needle retrieval, which was the specific risk in going full-W4A4 -- the
packager only validated 32k, and W4A4 long-context collapse is in our own
notes from the Granite work. It held.
reasoning_effort: the AEON template defaults to xhigh (template line 47),
and at xhigh this model can spend its entire budget inside <think> and
emit no answer -- a silent-empty-response hazard for the automated
summarizer/classifier consumers. Seat now pins the default to medium via
--default-chat-template-kwargs, per-request overridable. Override PROVEN
live: chat_template_kwargs.reasoning_effort=bogus returns HTTP 400
carrying the template's own exception text, so caller values genuinely
reach the template and invalid ones fail loudly rather than silently
falling back. Empty GEN_REASONING_EFFORT omits the flag for models that do
not read the kwarg -- the Qwen3.6 line ignores it entirely, where setting
it would be a false lever.
All 7 aliases verified routing. Rollback is one .env line; the previous
build is untouched at /tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed.
TWO GAPS, declared:
- Incumbent concurrency was never captured before the swap (I baselined
bs=1 only), so the conc=1/6 figures have no same-hardware comparator.
- Perplexity NOT measured. eval_quality correctly refused it: under
--speculative-config prompt_logprobs come back ~uniform (median rank
~130k), playbook trap 2. A real PPL number needs both seats served
without spec-decode.
Adds concbench.py (concurrent throughput; wall-clock aggregate, not
sum-of-rates, and delta-based MTP accounting).
29 lines
669 B
JSON
29 lines
669 B
JSON
{
|
|
"tag": "incumbent-2026-08-16",
|
|
"model": "qwen3.8-27b-uncensored",
|
|
"tok_s_median": 94.09347994622732,
|
|
"tok_s_mean": 93.55745548449859,
|
|
"tok_s_min": 82.96871273617583,
|
|
"tok_s_max": 105.74160032081423,
|
|
"mtp_accept_median": 0.47653554175293306,
|
|
"rates": [
|
|
95.95445864393953,
|
|
101.0434335379699,
|
|
89.28808149441755,
|
|
95.61995719061252,
|
|
82.96871273617583,
|
|
85.27639725021699,
|
|
105.74160032081423,
|
|
92.56700270184211
|
|
],
|
|
"accs": [
|
|
0.5020833333333333,
|
|
0.5416666666666666,
|
|
0.43103448275862066,
|
|
0.4927536231884058,
|
|
0.3835125448028674,
|
|
0.42424242424242425,
|
|
0.5839080459770115,
|
|
0.4603174603174603
|
|
]
|
|
} |