feat(llama-swap): pin to GPU 0, reserving it for large-model hot-loads
ana-ml2's Ada->Blackwell swap (2x96GB) frees GPU 0 entirely. Pin llama-swap to GPU 0 via NVIDIA_VISIBLE_DEVICES so on-demand large-model hot-loads land there, off GPU 1 where the always-on vLLM services (granite + embed/rerank/ reward) live. Closes the long-standing 'pin llama-swap to GPU 0' item.
This commit is contained in:
@@ -29,6 +29,12 @@ services:
|
|||||||
environment:
|
environment:
|
||||||
- HF_HOME=/hfcache
|
- HF_HOME=/hfcache
|
||||||
- HF_HUB_CACHE=/hfcache/hub
|
- HF_HUB_CACHE=/hfcache/hub
|
||||||
|
# Pin to GPU 0 — the reserved card for on-demand large-model hot-loads.
|
||||||
|
# The always-on vLLM services (granite + embed/rerank/reward) own GPU 1;
|
||||||
|
# keeping llama-swap off GPU 1 stops a hot-loaded model from contending
|
||||||
|
# with them. llama.cpp then sees only GPU 0 (cuda:0), so --n-gpu-layers
|
||||||
|
# 999 loads there with no per-model device targeting needed.
|
||||||
|
- NVIDIA_VISIBLE_DEVICES=${LLAMA_SWAP_GPU:-0}
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD-SHELL", "curl -fsS http://localhost:8080/ >/dev/null || exit 1"]
|
test: ["CMD-SHELL", "curl -fsS http://localhost:8080/ >/dev/null || exit 1"]
|
||||||
interval: 30s
|
interval: 30s
|
||||||
|
|||||||
Reference in New Issue
Block a user