vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

tune(qwen3.5-122b): gpu-mem-util 0.90->0.95, max-num-seqs 4->8 (KV 260K->446K tokens, 3.4x concurrency @131K, no OOM)

Browse Source
This commit is contained in:
vh
2026-06-19 01:21:28 -07:00
parent 89c83c4271
commit 3ba0e544db
2 changed files with 6 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/qwen3.5-122b/.env.example
+4 -2
View File
@@ -15,8 +15,10 @@ QWEN35_GPU_ID=0
QWEN35_SERVED_NAME=qwen3.5-122-a10b
QWEN35_MAX_MODEL_LEN=131072
QWEN35_MAX_NUM_SEQS=4
QWEN35_GPU_MEM_UTIL=0.90
QWEN35_MAX_NUM_SEQS=8
# Single-tenant 96GB card; 0.95 is safe-aggressive (leaves ~5GB for activation
# spikes + fragmentation, esp. with 8 concurrent seqs). 0.97+ risks OOM under load.
QWEN35_GPU_MEM_UTIL=0.95
QWEN35_MAX_NUM_BATCHED_TOKENS=32768
# Optional upstream vLLM API key (empty = no auth; internal net only).