feat(llama-swap): pin to GPU 0, reserving it for large-model hot-loads
ana-ml2's Ada->Blackwell swap (2x96GB) frees GPU 0 entirely. Pin llama-swap to GPU 0 via NVIDIA_VISIBLE_DEVICES so on-demand large-model hot-loads land there, off GPU 1 where the always-on vLLM services (granite + embed/rerank/ reward) live. Closes the long-standing 'pin llama-swap to GPU 0' item.
This commit is contained in:
@@ -29,6 +29,12 @@ services:
|
||||
environment:
|
||||
- HF_HOME=/hfcache
|
||||
- HF_HUB_CACHE=/hfcache/hub
|
||||
# Pin to GPU 0 — the reserved card for on-demand large-model hot-loads.
|
||||
# The always-on vLLM services (granite + embed/rerank/reward) own GPU 1;
|
||||
# keeping llama-swap off GPU 1 stops a hot-loaded model from contending
|
||||
# with them. llama.cpp then sees only GPU 0 (cuda:0), so --n-gpu-layers
|
||||
# 999 loads there with no per-model device targeting needed.
|
||||
- NVIDIA_VISIBLE_DEVICES=${LLAMA_SWAP_GPU:-0}
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -fsS http://localhost:8080/ >/dev/null || exit 1"]
|
||||
interval: 30s
|
||||
|
||||
Reference in New Issue
Block a user