# llama-swap

GGUF model server with on-demand model swapping. Served via llama.cpp's `llama-server` under the llama-swap proxy.

**Server:** ana-ml2
**Port:** 9292 (configurable via `.env`)
**GPU:** both (unpinned — `runtime: nvidia` grants access to all devices; per-model GPU selection happens inside `config.yaml`)

## Files

- **`compose.yaml`** — canonical compose. Deployed to `/opt/docker/compose/llama-swap/compose.yaml` on ana-ml2.
- **`.env.example`** — template for the per-host `.env`. Copy to `.env` on the server and tweak.
- **`config.yaml`** — model definitions and groups. Deployed to `/opt/docker/conf/llama-swap/config.yaml` on the server.

Homepage labels are in the compose file under the `AI Systems` group, matching the convention used by `vllm` and `infinity`.

## Deploy a fresh install

```bash
scripts/deploy-stack.sh ana-ml2 llama-swap

ssh ana-ml2 '
  cd /opt/docker/compose/llama-swap && \
  cp -n .env.example .env && \
  docker compose config && \
  docker compose up -d && \
  docker compose logs --tail=30
'
```

## Model reference conventions

- **Modern entries:** use `-hf <user>/<repo>[:<quant>]` — reads from the shared HF cache, nothing to pre-stage outside `hf download`
- **Legacy entries:** use `--model /models/<dir>/<file>.gguf` — reads GGUFs from `/tank/aimodels/llm/` (pre-HF-cache era, gradually being migrated)

New models should prefer the `-hf` pattern.

## Deploy updates to config only

```bash
# After editing config.yaml here:
scp config.yaml ana-ml2:/opt/docker/conf/llama-swap/config.yaml
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose restart'
```

## Deploy updates to compose only

```bash
# After editing compose.yaml or .env.example here:
scripts/deploy-stack.sh ana-ml2 llama-swap
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose up -d'
```