Files

T

vh 7e7130172e vllm: rename stack from vllm-qwen3 → vllm + add Skywork reward classifier

Two related changes shipped together. The stack rename is independent
but adding `vllm-reward` to the existing `vllm-qwen3` would have made
that name actively misleading.

**Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo
references (README.md root, servers/ana-ml2/, stacks/llama-swap/,
configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two
intentional history mentions retained (servers/ana-ml2 + stacks/vllm
README).

**Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ
on port 8003. The AWQ output is a locally-quantized model (not from HF),
so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the
shared HF cache. Model config.json declares LlamaForSequenceClassification
which vLLM's pooling runner picks up automatically — produces a single
reward score per input via /classify.

**Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1
deprecated --task in favor of --runner pooling (model architecture in
config.json drives the classification head). Compose uses --runner
pooling with a comment explaining the substitution.

**GPU memory:** no rebalance needed — production had already tuned
EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now
matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB
headroom on the 48 GB Ada.

**Server-side:** brought existing vllm-qwen3 down, mv'd
/opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended
REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed
new compose via scripts/deploy-stack.sh, brought all 3 services up.

**Smoke tests:**
- /health on 8001/8002/8003 → 200
- /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
  with max_model_len 16384
- /classify with a sample conversation → returns LABEL_0 with prob 0.9999
  (single-output regression-style reward score, expected shape for a
  reward model)

2026-05-13 22:00:26 -07:00

1.8 KiB

Raw Blame History

llama-swap

GGUF model server with on-demand model swapping. Served via llama.cpp's llama-server under the llama-swap proxy.

Server: ana-ml2 Port: 9292 (configurable via .env) GPU: both (unpinned — runtime: nvidia grants access to all devices; per-model GPU selection happens inside config.yaml)

Files

compose.yaml — canonical compose. Deployed to /opt/docker/compose/llama-swap/compose.yaml on ana-ml2.
.env.example — template for the per-host .env. Copy to .env on the server and tweak.
config.yaml — model definitions and groups. Deployed to /opt/docker/conf/llama-swap/config.yaml on the server.

Homepage labels are in the compose file under the AI Systems group, matching the convention used by vllm and infinity.

Deploy a fresh install

scripts/deploy-stack.sh ana-ml2 llama-swap

ssh ana-ml2 '
  cd /opt/docker/compose/llama-swap && \
  cp -n .env.example .env && \
  docker compose config && \
  docker compose up -d && \
  docker compose logs --tail=30
'

Model reference conventions

Modern entries: use -hf <user>/<repo>[:<quant>] — reads from the shared HF cache, nothing to pre-stage outside hf download
Legacy entries: use --model /models/<dir>/<file>.gguf — reads GGUFs from /tank/aimodels/llm/ (pre-HF-cache era, gradually being migrated)

New models should prefer the -hf pattern.

Deploy updates to config only

# After editing config.yaml here:
scp config.yaml ana-ml2:/opt/docker/conf/llama-swap/config.yaml
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose restart'

Deploy updates to compose only

# After editing compose.yaml or .env.example here:
scripts/deploy-stack.sh ana-ml2 llama-swap
ssh ana-ml2 'cd /opt/docker/compose/llama-swap && docker compose up -d'

1.8 KiB Raw Blame History