perf(gen-seat): record prefill measurements — roughly doubled
Closes the one axis of the original premise left unverified. Measured cold (cache-busted) on both builds under matching serve configs: ~6.7k-token prompt 3,206 -> 6,334 tok/s prefill (+98%) ~27k-token prompt 2,862 -> 5,085 tok/s prefill (+78%) TTFT on a ~27k doc 9.43 -> 5.31 s (-44%) Prefill gains far exceed the +18% decode gain, and that ordering is the expected one: decode at bs=1 is memory-bandwidth-bound and the weights are 4-bit under either scheme, so little changes; prefill is compute-bound, which is where native Blackwell FP4 tensor cores replace the Marlin dequant-to-BF16 path. The summarizer aliases are the consumers that feel this. Adds bench/prefill_bench.py plus the raw JSON. The harness deliberately uses SystemRandom: a seeded nonce regenerates the previous run's prompts verbatim, prefix caching then serves them, and the first attempt read ~41k tok/s of cache-hit rather than ~5k of actual prefill.
This commit is contained in:
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"tag": "w4a16-recheck",
|
||||
"model": "probe",
|
||||
"tok_s_median": 77.51086756145159,
|
||||
"tok_s_mean": 77.55102759761677,
|
||||
"tok_s_min": 69.14223827875307,
|
||||
"tok_s_max": 88.11959921725081,
|
||||
"mtp_accept_median": 0.4493388052895577,
|
||||
"rates": [
|
||||
82.25341481185131,
|
||||
77.12830062541333,
|
||||
78.45783182678721,
|
||||
69.14223827875307,
|
||||
71.73740809898078,
|
||||
77.89343449748986,
|
||||
88.11959921725081,
|
||||
75.67599342440771
|
||||
],
|
||||
"accs": [
|
||||
0.494824016563147,
|
||||
0.44573643410852715,
|
||||
0.4536489151873767,
|
||||
0.36300174520069806,
|
||||
0.3894927536231884,
|
||||
0.45294117647058824,
|
||||
0.56,
|
||||
0.43238095238095237
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user