Files

T

vh 53f00b232b llama-swap: add selene-1-mini-8b (Atla judge model, imatrix Q6_K)

AtlaAI's Selene-1-Mini judge model for evaluation/scoring tasks.
Llama 3.1 8B base, mradermacher imatrix-quantized Q6_K (~6.5GB,
quality-leaning quant). Apache-2.0. Per Atla cookbook these defaults
hit 84% on RAGTruth hallucination eval.

New 'JUDGE / EVAL MODELS' section between the dense chat models and
the embedding models — separate category from chat/reasoning since
the run-params shape is different (deterministic-leaning: temp 0.01,
top-p 1.0, no repeat penalty).

q8_0 KV cache to fit 32K ctx cleanly on the 3090 with headroom.

Pre-pulled into the shared HF cache via the new
playbooks/pull-hf-model.yaml playbook (canonical replacement for
ad-hoc huggingface_hub.snapshot_download calls; see CHANGELOG).

Smoke-tested 2026-05-13: GET /v1/models lists selene-1-mini-8b,
POST /v1/chat/completions returns expected output cleanly.

2026-05-13 16:19:01 -07:00

conf

llama-swap: add selene-1-mini-8b (Atla judge model, imatrix Q6_K)

2026-05-13 16:19:01 -07:00

.env.example

Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline

2026-04-20 14:29:48 -07:00

compose.yaml

Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline

2026-04-20 14:29:48 -07:00

README.md

Initial commit: PFI fleet inventory, stacks, tooling, and backup pipeline

2026-04-20 14:29:48 -07:00

README.md

llama-swap

GGUF model server with on-demand model swapping. Served via llama.cpp's llama-server under the llama-swap proxy.

Server: ana-ml2 Port: 9292 (configurable via .env) GPU: both (unpinned — runtime: nvidia grants access to all devices; per-model GPU selection happens inside config.yaml)

Files

compose.yaml — canonical compose. Deployed to /opt/docker/compose/llama-swap/compose.yaml on ana-ml2.
.env.example — template for the per-host .env. Copy to .env on the server and tweak.
config.yaml — model definitions and groups. Deployed to /opt/docker/conf/llama-swap/config.yaml on the server.

Homepage labels are in the compose file under the AI Systems group, matching the convention used by vllm-qwen3 and infinity.

Deploy a fresh install

scripts/deploy-stack.sh ana-ml2 llama-swap

ssh ana-ml2 '
  cd /opt/docker/compose/llama-swap && \
  cp -n .env.example .env && \
  docker compose config && \
  docker compose up -d && \
  docker compose logs --tail=30
'

Model reference conventions

Modern entries: use -hf <user>/<repo>[:<quant>] — reads from the shared HF cache, nothing to pre-stage outside hf download
Legacy entries: use --model /models/<dir>/<file>.gguf — reads GGUFs from /tank/aimodels/llm/ (pre-HF-cache era, gradually being migrated)

New models should prefer the -hf pattern.

Deploy updates to config only

# After editing config.yaml here:
scp config.yaml ana-ml2:/opt/docker/conf/llama-swap/config.yaml
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose restart'

Deploy updates to compose only

# After editing compose.yaml or .env.example here:
scripts/deploy-stack.sh ana-ml2 llama-swap
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose up -d'