Operator call: the incumbent abliterated model was the first one we could
find, not an optimised pick. sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-
UNCENSORED-NVFP4 (base AEON-7 BF16, abliterix-abliterated, Apache-2.0),
byte-verified at /tank/aimodels/qwen38-27b-aeon-ultimate-nvfp4.
Measured on the same harness, same GPU, cache-busted per playbook 5.
Baseline was RE-measured live before the swap rather than trusted:
incumbent (W4A4+FP8 mixed) AEON (W4A4)
decode bs=1 94.09 tok/s 104.22 +10.8%
MTP acceptance 47.7% 52.3% +4.6pp
abliteration 4/4 4/4
surface 6/6 6/6
weights 22.5 GB 20.6 GB -8.4%
AEON concurrency: conc=1 98.48 tok/s aggregate; conc=6 381.29 aggregate /
63.55 per-stream, MTP holding 50.6% under load.
Surface 6/6 includes vision (image-judge rides this seat) and a 36k-token
needle retrieval, which was the specific risk in going full-W4A4 -- the
packager only validated 32k, and W4A4 long-context collapse is in our own
notes from the Granite work. It held.
reasoning_effort: the AEON template defaults to xhigh (template line 47),
and at xhigh this model can spend its entire budget inside <think> and
emit no answer -- a silent-empty-response hazard for the automated
summarizer/classifier consumers. Seat now pins the default to medium via
--default-chat-template-kwargs, per-request overridable. Override PROVEN
live: chat_template_kwargs.reasoning_effort=bogus returns HTTP 400
carrying the template's own exception text, so caller values genuinely
reach the template and invalid ones fail loudly rather than silently
falling back. Empty GEN_REASONING_EFFORT omits the flag for models that do
not read the kwarg -- the Qwen3.6 line ignores it entirely, where setting
it would be a false lever.
All 7 aliases verified routing. Rollback is one .env line; the previous
build is untouched at /tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed.
TWO GAPS, declared:
- Incumbent concurrency was never captured before the swap (I baselined
bs=1 only), so the conc=1/6 figures have no same-hardware comparator.
- Perplexity NOT measured. eval_quality correctly refused it: under
--speculative-config prompt_logprobs come back ~uniform (median rank
~130k), playbook trap 2. A real PPL number needs both seats served
without spec-decode.
Adds concbench.py (concurrent throughput; wall-clock aggregate, not
sum-of-rates, and delta-based MTP accounting).
Closes the one axis of the original premise left unverified. Measured
cold (cache-busted) on both builds under matching serve configs:
~6.7k-token prompt 3,206 -> 6,334 tok/s prefill (+98%)
~27k-token prompt 2,862 -> 5,085 tok/s prefill (+78%)
TTFT on a ~27k doc 9.43 -> 5.31 s (-44%)
Prefill gains far exceed the +18% decode gain, and that ordering is the
expected one: decode at bs=1 is memory-bandwidth-bound and the weights
are 4-bit under either scheme, so little changes; prefill is
compute-bound, which is where native Blackwell FP4 tensor cores replace
the Marlin dequant-to-BF16 path. The summarizer aliases are the
consumers that feel this.
Adds bench/prefill_bench.py plus the raw JSON. The harness deliberately
uses SystemRandom: a seeded nonce regenerates the previous run's prompts
verbatim, prefix caching then serves them, and the first attempt read
~41k tok/s of cache-hit rather than ~5k of actual prefill.
Re-quantizes the fleet `gen` seat from weight-only NVFP4A16 to a
mixed-precision build: NVFP4 W4A4 for layers 0-55 MLPs, FP8 W8A8 for the
attention projections / linear_attn / lm_head / layers 56-63 MLPs, FP8 KV
cache. Replicates the scheme of unsloth/Qwen3.8-27B-NVFP4 on the
abliterated weights.
The queued task named this "W4A8" (NVFP4 weights + FP8 activations). That
checkpoint cannot be served: vLLM 0.24's compressed-tensors dispatcher
(compressed_tensors.py:704-713) accepts NVFP4 weights with either no input
quantization (W4A16, which forces the Marlin kernel) or NVFP4 input
quantization (W4A4) -- anything else, FP8 included, raises ValueError at
load. CompressedTensorsW4A8Fp8 is INT4 weights gated on an exact-sm90
check, so it is closed on Blackwell twice over. The ~20% intuition was
correct; the scheme name was not. Getting FP8 into the mix has to be done
per-layer-group.
Established the gain before spending GPU time: unsloth's build was already
on-box, so serving it as a probe measured +19.1% over our seat at identical
MTP acceptance -- a kernel-level result, no requant needed to learn it.
Measured, cache-busted, bs=1:
decode 80.12 -> 94.53 tok/s (+18.0%)
MTP acceptance 47.8% -> 47.7% (unchanged)
perplexity (n=6) 6.941 -> 7.059 (+1.7%)
abliteration 4/4 -> 4/4 (preserved)
weights on disk 27.7 -> 22.5 GB (-19%)
Surface test green on the live seat: plain chat, vision, tool calling,
thinking split, 36K-token needle retrieval, streaming. All 7 LiteLLM
aliases verified routing.
GEN_GPU_MEM_UTIL 0.45 -> 0.43: the new weights are 5.2 GB smaller, and at
0.45 the seat absorbed that slack as KV, leaving meromero-charrp 0.18 GiB
short of its budget on the shared GPU0 -- it crash-looped. Handing the
space back leaves gen 422K tokens of KV (1.6x its 262K context) and both
seats co-resident at 89.8/97.9 GB.
Also records two measured negatives so they are not re-chased:
GEN_SPEC_TOKENS is already optimal at 3 (swept 2/3/4/5 -> 77.1/80.1/78.7/
75.9 tok/s), and vLLM's prompt_logprobs are ~uniform while speculative
decoding is on, so perplexity must be measured with spec off.
Pipeline, acceptance harness and raw measurements land in
services/gen-seat-mixed-quant/. Rollback is one .env line; the previous
build is untouched at /tank/aimodels/qwen38-27b-uncensored-nvfp4.
vh