tune(vllm): pin --enable-prefix-caching on granite + qwen

Benched granite prefix caching at ~6.5x faster TTFT (45ms cached vs 292ms
uncached) on a shared ~4.5k-token summarizer template. granite already had it
on by vLLM-v1 default; pinned explicit so a version flip can't silently disable
it. qwen (nightly) defaulted it OFF -> flipped on (free for the text-chat path,
marginal for vision where each image is a distinct prefix). Soft/evictable KV,
zero memory change (GPU1 still ~3.7GB free), all 5 services healthy.
This commit is contained in:
vh
2026-06-13 12:30:20 -07:00
parent 1e2a3a13b5
commit a9a2be7060
2 changed files with 8 additions and 0 deletions
+3
View File
@@ -54,6 +54,9 @@ services:
- ${QWEN_MAX_MODEL_LEN}
- --dtype
- auto
# Prefix caching pinned ON (the nightly defaults it OFF). Free win for the
# text-chat path; marginal for vision (each image is a distinct prefix).
- --enable-prefix-caching
deploy:
resources:
reservations: