From 19a07b96abfa6ad2821821a22ffe3c452bb0c628 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 12 Jun 2026 17:41:13 -0700 Subject: [PATCH] tune(vllm): re-floor trio GPU util for Blackwell (96GB), 20x-parallel-stable Ada->Blackwell swap doubled card VRAM, so the Ada-era fractions (0.07/0.07/ 0.18) reserved ~2x the bytes for the same models. Empirically re-floored via 0.01-step climb until each service was stable under 20x parallel inference: embed/rerank 0.05 (load-floor for the 0.6B models), reward 0.10 (the real over-provision). Frees ~11 GB on GPU 1. Live .env on ana-ml2 already applied. --- stacks/vllm/.env.example | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/stacks/vllm/.env.example b/stacks/vllm/.env.example index ffdc53d..a9da0c3 100644 --- a/stacks/vllm/.env.example +++ b/stacks/vllm/.env.example @@ -38,9 +38,19 @@ RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B # EMBED 0.07 (~3.4 GB) — 0.6B Qwen3 embed @8k; 1.1GB weights + ~2GB KV (~3x) # RERANK 0.07 (~3.4 GB) — 0.6B Qwen3 rerank @8k; comfortable # REWARD 0.18 (~8.6 GB) — 8B Skywork AWQ @16k; 4.4GB weights + ~3.5GB KV (~1.5x) -EMBED_GPU_MEM_UTIL=0.07 -RERANK_GPU_MEM_UTIL=0.07 -REWARD_GPU_MEM_UTIL=0.18 +# +# RETUNED 2026-06-12 for the Ada->Blackwell swap (cards are now 96 GB, so the +# Ada-era fractions reserved ~2x the bytes for the same models). Empirically +# re-floored: util climbed in 0.01 steps until each service was stable under +# 20x parallel inference. +# embed/rerank 0.05 (~4.8 GB ea) — the load-floor for the 0.6B models (0.04 +# OOMs on engine init; their footprint is fixed weights+context, not KV). +# reward 0.10 (~9.6 GB) — the real over-provision was here (0.18 -> 0.10). +# Net: ~11 GB freed on GPU 1. (Fractions are of TOTAL card VRAM — re-floor if +# the cards change again.) +EMBED_GPU_MEM_UTIL=0.05 +RERANK_GPU_MEM_UTIL=0.05 +REWARD_GPU_MEM_UTIL=0.10 # Context length caps — lower these if VRAM is tight. # Qwen3-Embedding supports up to 32k; reranker up to 32k.