diff --git a/stacks/vllm/.env.example b/stacks/vllm/.env.example index ffdc53d..a9da0c3 100644 --- a/stacks/vllm/.env.example +++ b/stacks/vllm/.env.example @@ -38,9 +38,19 @@ RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B # EMBED 0.07 (~3.4 GB) — 0.6B Qwen3 embed @8k; 1.1GB weights + ~2GB KV (~3x) # RERANK 0.07 (~3.4 GB) — 0.6B Qwen3 rerank @8k; comfortable # REWARD 0.18 (~8.6 GB) — 8B Skywork AWQ @16k; 4.4GB weights + ~3.5GB KV (~1.5x) -EMBED_GPU_MEM_UTIL=0.07 -RERANK_GPU_MEM_UTIL=0.07 -REWARD_GPU_MEM_UTIL=0.18 +# +# RETUNED 2026-06-12 for the Ada->Blackwell swap (cards are now 96 GB, so the +# Ada-era fractions reserved ~2x the bytes for the same models). Empirically +# re-floored: util climbed in 0.01 steps until each service was stable under +# 20x parallel inference. +# embed/rerank 0.05 (~4.8 GB ea) — the load-floor for the 0.6B models (0.04 +# OOMs on engine init; their footprint is fixed weights+context, not KV). +# reward 0.10 (~9.6 GB) — the real over-provision was here (0.18 -> 0.10). +# Net: ~11 GB freed on GPU 1. (Fractions are of TOTAL card VRAM — re-floor if +# the cards change again.) +EMBED_GPU_MEM_UTIL=0.05 +RERANK_GPU_MEM_UTIL=0.05 +REWARD_GPU_MEM_UTIL=0.10 # Context length caps — lower these if VRAM is tight. # Qwen3-Embedding supports up to 32k; reranker up to 32k.