vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

tune(vllm): re-floor trio GPU util for Blackwell (96GB), 20x-parallel-stable

Browse Source 
Ada->Blackwell swap doubled card VRAM, so the Ada-era fractions (0.07/0.07/
0.18) reserved ~2x the bytes for the same models. Empirically re-floored via
0.01-step climb until each service was stable under 20x parallel inference:
embed/rerank 0.05 (load-floor for the 0.6B models), reward 0.10 (the real
over-provision). Frees ~11 GB on GPU 1. Live .env on ana-ml2 already applied.
This commit is contained in:
vh
2026-06-12 17:41:13 -07:00
parent edf0f912f8
commit 19a07b96ab
1 changed files with 13 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/vllm/.env.example
+13 -3
View File
@@ -38,9 +38,19 @@ RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B
# EMBED 0.07 (~3.4 GB) — 0.6B Qwen3 embed @8k; 1.1GB weights + ~2GB KV (~3x)
# RERANK 0.07 (~3.4 GB) — 0.6B Qwen3 rerank @8k; comfortable
# REWARD 0.18 (~8.6 GB) — 8B Skywork AWQ @16k; 4.4GB weights + ~3.5GB KV (~1.5x)
EMBED_GPU_MEM_UTIL=0.07
RERANK_GPU_MEM_UTIL=0.07
REWARD_GPU_MEM_UTIL=0.18
#
# RETUNED 2026-06-12 for the Ada->Blackwell swap (cards are now 96 GB, so the
# Ada-era fractions reserved ~2x the bytes for the same models). Empirically
# re-floored: util climbed in 0.01 steps until each service was stable under
# 20x parallel inference.
# embed/rerank 0.05 (~4.8 GB ea) — the load-floor for the 0.6B models (0.04
# OOMs on engine init; their footprint is fixed weights+context, not KV).
# reward 0.10 (~9.6 GB) — the real over-provision was here (0.18 -> 0.10).
# Net: ~11 GB freed on GPU 1. (Fractions are of TOTAL card VRAM — re-floor if
# the cards change again.)
EMBED_GPU_MEM_UTIL=0.05
RERANK_GPU_MEM_UTIL=0.05
REWARD_GPU_MEM_UTIL=0.10
# Context length caps — lower these if VRAM is tight.
# Qwen3-Embedding supports up to 32k; reranker up to 32k.