vllm: rename stack from vllm-qwen3 → vllm + add Skywork reward classifier
Two related changes shipped together. The stack rename is independent but adding `vllm-reward` to the existing `vllm-qwen3` would have made that name actively misleading. **Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo references (README.md root, servers/ana-ml2/, stacks/llama-swap/, configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two intentional history mentions retained (servers/ana-ml2 + stacks/vllm README). **Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ on port 8003. The AWQ output is a locally-quantized model (not from HF), so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the shared HF cache. Model config.json declares LlamaForSequenceClassification which vLLM's pooling runner picks up automatically — produces a single reward score per input via /classify. **Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1 deprecated --task in favor of --runner pooling (model architecture in config.json drives the classification head). Compose uses --runner pooling with a comment explaining the substitution. **GPU memory:** no rebalance needed — production had already tuned EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB headroom on the 48 GB Ada. **Server-side:** brought existing vllm-qwen3 down, mv'd /opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed new compose via scripts/deploy-stack.sh, brought all 3 services up. **Smoke tests:** - /health on 8001/8002/8003 → 200 - /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ with max_model_len 16384 - /classify with a sample conversation → returns LABEL_0 with prob 0.9999 (single-output regression-style reward score, expected shape for a reward model)
This commit is contained in:
@@ -29,7 +29,7 @@ pre-backup DB hook is required. Contrast with `configs/restic/ana-docker/`
|
||||
where synapse/seafile/vaultwarden DB dumps run first.
|
||||
|
||||
- `llama-swap` — GGUF swapper (llama.cpp)
|
||||
- `vllm-qwen3` — embedding + rerank
|
||||
- `vllm` — embedding + rerank + reward classifier
|
||||
- `comfyui`, `kokoro`, `parakeet`, `vibevoice`
|
||||
- `beszel-agent-ana`, `dozzle-agent-ana`, `dockge`
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
# caches, llama.cpp GGUFs, ComfyUI models, etc.) — all regenerable
|
||||
# from upstream. Backing them up would blow the repo size budget.
|
||||
# - No DB dumps needed. None of the stacks on this host (llama-swap,
|
||||
# vllm-qwen3, comfyui, kokoro, parakeet, vibevoice, beszel-agent,
|
||||
# vllm, comfyui, kokoro, parakeet, vibevoice, beszel-agent,
|
||||
# dozzle-agent, dockge) store relational data.
|
||||
|
||||
version: "1"
|
||||
|
||||
Reference in New Issue
Block a user