tune(vllm): pin --enable-prefix-caching on granite + qwen

Benched granite prefix caching at ~6.5x faster TTFT (45ms cached vs 292ms
uncached) on a shared ~4.5k-token summarizer template. granite already had it
on by vLLM-v1 default; pinned explicit so a version flip can't silently disable
it. qwen (nightly) defaulted it OFF -> flipped on (free for the text-chat path,
marginal for vision where each image is a distinct prefix). Soft/evictable KV,
zero memory change (GPU1 still ~3.7GB free), all 5 services healthy.
This commit is contained in:
2026-06-13 12:30:20 -07:00
parent 1e2a3a13b5
commit a9a2be7060
2 changed files with 8 additions and 0 deletions
+3
View File
@@ -54,6 +54,9 @@ services:
- ${QWEN_MAX_MODEL_LEN}
- --dtype
- auto
# Prefix caching pinned ON (the nightly defaults it OFF). Free win for the
# text-chat path; marginal for vision (each image is a distinct prefix).
- --enable-prefix-caching
deploy:
resources:
reservations:
+5
View File
@@ -243,6 +243,11 @@ services:
# FP8 KV cache — halves KV memory; near-lossless on Ada (cc 8.9).
- --kv-cache-dtype
- ${GRANITE_KV_CACHE_DTYPE}
# Prefix caching pinned EXPLICIT (vLLM v1 defaults it on, but pin so a
# version flip can't silently disable it). Benched 2026-06-13: ~6.5x faster
# TTFT (45ms vs 292ms) on a shared ~4.5k-token summarizer template; soft/
# evictable KV, neutral when prefixes don't repeat — pure win for granite.
- --enable-prefix-caching
deploy:
resources:
reservations: