vh
4a5c3fcccf
Closes the one axis of the original premise left unverified. Measured cold (cache-busted) on both builds under matching serve configs: ~6.7k-token prompt 3,206 -> 6,334 tok/s prefill (+98%) ~27k-token prompt 2,862 -> 5,085 tok/s prefill (+78%) TTFT on a ~27k doc 9.43 -> 5.31 s (-44%) Prefill gains far exceed the +18% decode gain, and that ordering is the expected one: decode at bs=1 is memory-bandwidth-bound and the weights are 4-bit under either scheme, so little changes; prefill is compute-bound, which is where native Blackwell FP4 tensor cores replace the Marlin dequant-to-BF16 path. The summarizer aliases are the consumers that feel this. Adds bench/prefill_bench.py plus the raw JSON. The harness deliberately uses SystemRandom: a seeded nonce regenerates the previous run's prompts verbatim, prefix caching then serves them, and the first attempt read ~41k tok/s of cache-hit rather than ~5k of actual prefill.
29 lines
629 B
JSON
29 lines
629 B
JSON
{
|
|
"tag": "w4a16-recheck",
|
|
"model": "probe",
|
|
"tok_s_median": 77.51086756145159,
|
|
"tok_s_mean": 77.55102759761677,
|
|
"tok_s_min": 69.14223827875307,
|
|
"tok_s_max": 88.11959921725081,
|
|
"mtp_accept_median": 0.4493388052895577,
|
|
"rates": [
|
|
82.25341481185131,
|
|
77.12830062541333,
|
|
78.45783182678721,
|
|
69.14223827875307,
|
|
71.73740809898078,
|
|
77.89343449748986,
|
|
88.11959921725081,
|
|
75.67599342440771
|
|
],
|
|
"accs": [
|
|
0.494824016563147,
|
|
0.44573643410852715,
|
|
0.4536489151873767,
|
|
0.36300174520069806,
|
|
0.3894927536231884,
|
|
0.45294117647058824,
|
|
0.56,
|
|
0.43238095238095237
|
|
]
|
|
} |