vllm: rename stack from vllm-qwen3 → vllm + add Skywork reward classifier

Two related changes shipped together. The stack rename is independent
but adding `vllm-reward` to the existing `vllm-qwen3` would have made
that name actively misleading.

**Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo
references (README.md root, servers/ana-ml2/, stacks/llama-swap/,
configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two
intentional history mentions retained (servers/ana-ml2 + stacks/vllm
README).

**Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ
on port 8003. The AWQ output is a locally-quantized model (not from HF),
so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the
shared HF cache. Model config.json declares LlamaForSequenceClassification
which vLLM's pooling runner picks up automatically — produces a single
reward score per input via /classify.

**Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1
deprecated --task in favor of --runner pooling (model architecture in
config.json drives the classification head). Compose uses --runner
pooling with a comment explaining the substitution.

**GPU memory:** no rebalance needed — production had already tuned
EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now
matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB
headroom on the 48 GB Ada.

**Server-side:** brought existing vllm-qwen3 down, mv'd
/opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended
REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed
new compose via scripts/deploy-stack.sh, brought all 3 services up.

**Smoke tests:**
- /health on 8001/8002/8003 → 200
- /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
  with max_model_len 16384
- /classify with a sample conversation → returns LABEL_0 with prob 0.9999
  (single-output regression-style reward score, expected shape for a
  reward model)
This commit is contained in:
vh
2026-05-13 22:00:26 -07:00
parent 662a73ee0e
commit 7e7130172e
12 changed files with 428 additions and 135 deletions
+2 -2
View File
@@ -66,7 +66,7 @@ Per-host snapshots of the running system live under `servers/<host>/system-detai
**GPU (ana-ml2):**
- `llama-swap` — GGUF model swapper via llama.cpp (port 9292)
- `vllm-qwen3` — embeddings (8001) + reranker (8002) via vLLM
- `vllm` — embeddings (8001) + reranker (8002) + Skywork reward classifier (8003) via vLLM
**Anaheim non-GPU (ana-docker):**
- `traefik`, `crowdsec`, `gitea`, `vaultwarden`, `synapse`, `seafile`, `searxng`, `openwebui`, `sillytavern`, `mailrise`, `rustdesk`, `dockge`, `it-tools`
@@ -77,7 +77,7 @@ Per-host snapshots of the running system live under `servers/<host>/system-detai
- Backup target: `rest-server-ana` on port 8000
**GPU ana-ml2 (non-canonical for now):**
- `comfyui`, `kokoro`, `parakeet`, `vibevoice` alongside the canonical `llama-swap` + `vllm-qwen3`
- `comfyui`, `kokoro`, `parakeet`, `vibevoice` alongside the canonical `llama-swap` + `vllm`
**NH3 (nh3-docker):**
- `adguard`, `dockge`, plus Beszel/Dozzle agents