Files
esh-pfi-infrastructure/stacks
vh a9a2be7060 tune(vllm): pin --enable-prefix-caching on granite + qwen
Benched granite prefix caching at ~6.5x faster TTFT (45ms cached vs 292ms
uncached) on a shared ~4.5k-token summarizer template. granite already had it
on by vLLM-v1 default; pinned explicit so a version flip can't silently disable
it. qwen (nightly) defaulted it OFF -> flipped on (free for the text-chat path,
marginal for vision where each image is a distinct prefix). Soft/evictable KV,
zero memory change (GPU1 still ~3.7GB free), all 5 services healthy.
2026-06-13 12:30:20 -07:00
..
2026-04-29 13:27:12 -07:00