vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

tune(qwen3.5-122b): gpu-mem-util 0.90->0.95, max-num-seqs 4->8 (KV 260K->446K tokens, 3.4x concurrency @131K, no OOM)

Browse Source
This commit is contained in:
vh
2026-06-19 01:21:28 -07:00
parent 89c83c4271
commit 3ba0e544db
2 changed files with 6 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/qwen3.5-122b/compose.yaml
+2 -2
View File
@@ -35,8 +35,8 @@ services:
- SERVED_MODEL_NAME=${QWEN35_SERVED_NAME:-qwen3.5-122-a10b}
- HOST_PORT=8000
- MAX_MODEL_LEN=${QWEN35_MAX_MODEL_LEN:-131072}
- MAX_NUM_SEQS=${QWEN35_MAX_NUM_SEQS:-4}
- GPU_MEMORY_UTILIZATION=${QWEN35_GPU_MEM_UTIL:-0.90}
- MAX_NUM_SEQS=${QWEN35_MAX_NUM_SEQS:-8}
- GPU_MEMORY_UTILIZATION=${QWEN35_GPU_MEM_UTIL:-0.95}
- MAX_NUM_BATCHED_TOKENS=${QWEN35_MAX_NUM_BATCHED_TOKENS:-32768}
# --reasoning-parser qwen3 surfaces <think>…</think> as reasoning_content;
# the thinking on/off itself is per-request (litellm chat_template_kwargs).