vh
8d6921cb75
- New: qwen3.6-35-a3b using -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q6_K_XL (~29 GB weights, thinking on, KB Qwen sampler defaults). - Swap: qwen3.6-35-a3b-abliterated from IIEleven11 Heretic Q8_0 to mradermacher/Qwen3.6-35B-A3B-abliterated-i1-GGUF:i1-Q6_K (already in the /hfcache on ana-ml2; ~27 GB). - Context: bumped both Qwen 3.6 entries 32768 -> 131072.
llama-swap
GGUF model server with on-demand model swapping. Served via llama.cpp's llama-server under the llama-swap proxy.
Server: ana-ml2
Port: 9292 (configurable via .env)
GPU: both (unpinned — runtime: nvidia grants access to all devices; per-model GPU selection happens inside config.yaml)
Files
compose.yaml— canonical compose. Deployed to/opt/docker/compose/llama-swap/compose.yamlon ana-ml2..env.example— template for the per-host.env. Copy to.envon the server and tweak.config.yaml— model definitions and groups. Deployed to/opt/docker/conf/llama-swap/config.yamlon the server.
Homepage labels are in the compose file under the AI Systems group, matching the convention used by vllm-qwen3 and infinity.
Deploy a fresh install
scripts/deploy-stack.sh ana-ml2 llama-swap
ssh ana-ml2 '
cd /opt/docker/compose/llama-swap && \
cp -n .env.example .env && \
docker compose config && \
docker compose up -d && \
docker compose logs --tail=30
'
Model reference conventions
- Modern entries: use
-hf <user>/<repo>[:<quant>]— reads from the shared HF cache, nothing to pre-stage outsidehf download - Legacy entries: use
--model /models/<dir>/<file>.gguf— reads GGUFs from/tank/aimodels/llm/(pre-HF-cache era, gradually being migrated)
New models should prefer the -hf pattern.
Deploy updates to config only
# After editing config.yaml here:
scp config.yaml ana-ml2:/opt/docker/conf/llama-swap/config.yaml
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose restart'
Deploy updates to compose only
# After editing compose.yaml or .env.example here:
scripts/deploy-stack.sh ana-ml2 llama-swap
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose up -d'