feat(llama-swap): pin to GPU 0, reserving it for large-model hot-loads

ana-ml2's Ada->Blackwell swap (2x96GB) frees GPU 0 entirely. Pin llama-swap
to GPU 0 via NVIDIA_VISIBLE_DEVICES so on-demand large-model hot-loads land
there, off GPU 1 where the always-on vLLM services (granite + embed/rerank/
reward) live. Closes the long-standing 'pin llama-swap to GPU 0' item.
This commit is contained in:
2026-06-12 15:17:46 -07:00
parent 922e8ad3d5
commit edf0f912f8
+6
View File
@@ -29,6 +29,12 @@ services:
environment:
- HF_HOME=/hfcache
- HF_HUB_CACHE=/hfcache/hub
# Pin to GPU 0 — the reserved card for on-demand large-model hot-loads.
# The always-on vLLM services (granite + embed/rerank/reward) own GPU 1;
# keeping llama-swap off GPU 1 stops a hot-loaded model from contending
# with them. llama.cpp then sees only GPU 0 (cuda:0), so --n-gpu-layers
# 999 loads there with no per-model device targeting needed.
- NVIDIA_VISIBLE_DEVICES=${LLAMA_SWAP_GPU:-0}
healthcheck:
test: ["CMD-SHELL", "curl -fsS http://localhost:8080/ >/dev/null || exit 1"]
interval: 30s