vh
7e7130172e
Two related changes shipped together. The stack rename is independent but adding `vllm-reward` to the existing `vllm-qwen3` would have made that name actively misleading. **Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo references (README.md root, servers/ana-ml2/, stacks/llama-swap/, configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two intentional history mentions retained (servers/ana-ml2 + stacks/vllm README). **Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ on port 8003. The AWQ output is a locally-quantized model (not from HF), so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the shared HF cache. Model config.json declares LlamaForSequenceClassification which vLLM's pooling runner picks up automatically — produces a single reward score per input via /classify. **Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1 deprecated --task in favor of --runner pooling (model architecture in config.json drives the classification head). Compose uses --runner pooling with a comment explaining the substitution. **GPU memory:** no rebalance needed — production had already tuned EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB headroom on the 48 GB Ada. **Server-side:** brought existing vllm-qwen3 down, mv'd /opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed new compose via scripts/deploy-stack.sh, brought all 3 services up. **Smoke tests:** - /health on 8001/8002/8003 → 200 - /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ with max_model_len 16384 - /classify with a sample conversation → returns LABEL_0 with prob 0.9999 (single-output regression-style reward score, expected shape for a reward model)
53 lines
1.8 KiB
Markdown
53 lines
1.8 KiB
Markdown
# llama-swap
|
|
|
|
GGUF model server with on-demand model swapping. Served via llama.cpp's `llama-server` under the llama-swap proxy.
|
|
|
|
**Server:** ana-ml2
|
|
**Port:** 9292 (configurable via `.env`)
|
|
**GPU:** both (unpinned — `runtime: nvidia` grants access to all devices; per-model GPU selection happens inside `config.yaml`)
|
|
|
|
## Files
|
|
|
|
- **`compose.yaml`** — canonical compose. Deployed to `/opt/docker/compose/llama-swap/compose.yaml` on ana-ml2.
|
|
- **`.env.example`** — template for the per-host `.env`. Copy to `.env` on the server and tweak.
|
|
- **`config.yaml`** — model definitions and groups. Deployed to `/opt/docker/conf/llama-swap/config.yaml` on the server.
|
|
|
|
Homepage labels are in the compose file under the `AI Systems` group, matching the convention used by `vllm` and `infinity`.
|
|
|
|
## Deploy a fresh install
|
|
|
|
```bash
|
|
scripts/deploy-stack.sh ana-ml2 llama-swap
|
|
|
|
ssh ana-ml2 '
|
|
cd /opt/docker/compose/llama-swap && \
|
|
cp -n .env.example .env && \
|
|
docker compose config && \
|
|
docker compose up -d && \
|
|
docker compose logs --tail=30
|
|
'
|
|
```
|
|
|
|
## Model reference conventions
|
|
|
|
- **Modern entries:** use `-hf <user>/<repo>[:<quant>]` — reads from the shared HF cache, nothing to pre-stage outside `hf download`
|
|
- **Legacy entries:** use `--model /models/<dir>/<file>.gguf` — reads GGUFs from `/tank/aimodels/llm/` (pre-HF-cache era, gradually being migrated)
|
|
|
|
New models should prefer the `-hf` pattern.
|
|
|
|
## Deploy updates to config only
|
|
|
|
```bash
|
|
# After editing config.yaml here:
|
|
scp config.yaml ana-ml2:/opt/docker/conf/llama-swap/config.yaml
|
|
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose restart'
|
|
```
|
|
|
|
## Deploy updates to compose only
|
|
|
|
```bash
|
|
# After editing compose.yaml or .env.example here:
|
|
scripts/deploy-stack.sh ana-ml2 llama-swap
|
|
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose up -d'
|
|
```
|