vh
4a5c3fcccf
Closes the one axis of the original premise left unverified. Measured cold (cache-busted) on both builds under matching serve configs: ~6.7k-token prompt 3,206 -> 6,334 tok/s prefill (+98%) ~27k-token prompt 2,862 -> 5,085 tok/s prefill (+78%) TTFT on a ~27k doc 9.43 -> 5.31 s (-44%) Prefill gains far exceed the +18% decode gain, and that ordering is the expected one: decode at bs=1 is memory-bandwidth-bound and the weights are 4-bit under either scheme, so little changes; prefill is compute-bound, which is where native Blackwell FP4 tensor cores replace the Marlin dequant-to-BF16 path. The summarizer aliases are the consumers that feel this. Adds bench/prefill_bench.py plus the raw JSON. The harness deliberately uses SystemRandom: a seeded nonce regenerates the previous run's prompts verbatim, prefix caching then serves them, and the first attempt read ~41k tok/s of cache-hit rather than ~5k of actual prefill.
13 lines
283 B
JSON
13 lines
283 B
JSON
{
|
|
"tag": "mixed",
|
|
"~6.5k": {
|
|
"approx_prompt_tokens": 6743,
|
|
"ttft_median_s": 1.0645924881100655,
|
|
"prefill_tok_s": 6333.878996244484
|
|
},
|
|
"~26k": {
|
|
"approx_prompt_tokens": 26993,
|
|
"ttft_median_s": 5.307923917658627,
|
|
"prefill_tok_s": 5085.415770598847
|
|
}
|
|
} |