vh
7e7130172e
Two related changes shipped together. The stack rename is independent but adding `vllm-reward` to the existing `vllm-qwen3` would have made that name actively misleading. **Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo references (README.md root, servers/ana-ml2/, stacks/llama-swap/, configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two intentional history mentions retained (servers/ana-ml2 + stacks/vllm README). **Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ on port 8003. The AWQ output is a locally-quantized model (not from HF), so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the shared HF cache. Model config.json declares LlamaForSequenceClassification which vLLM's pooling runner picks up automatically — produces a single reward score per input via /classify. **Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1 deprecated --task in favor of --runner pooling (model architecture in config.json drives the classification head). Compose uses --runner pooling with a comment explaining the substitution. **GPU memory:** no rebalance needed — production had already tuned EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB headroom on the 48 GB Ada. **Server-side:** brought existing vllm-qwen3 down, mv'd /opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed new compose via scripts/deploy-stack.sh, brought all 3 services up. **Smoke tests:** - /health on 8001/8002/8003 → 200 - /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ with max_model_len 16384 - /classify with a sample conversation → returns LABEL_0 with prob 0.9999 (single-output regression-style reward score, expected shape for a reward model)
55 lines
2.2 KiB
Bash
55 lines
2.2 KiB
Bash
# vllm stack tunables. Copy this to `.env` on the server before deploying.
|
|
#
|
|
# cp .env.example .env
|
|
# # edit .env with real values
|
|
# docker compose up -d
|
|
|
|
# Image version — pin for reproducibility (`latest` for edge)
|
|
VLLM_VERSION=latest
|
|
|
|
# Host ports (container always listens on 8000 internally)
|
|
EMBED_PORT=8001
|
|
RERANK_PORT=8002
|
|
REWARD_PORT=8003
|
|
|
|
# GPU assignment — all services share this GPU
|
|
# (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work in llama-swap)
|
|
GPU_ID=1
|
|
|
|
# Models — reference by full repo name in API requests
|
|
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
|
|
RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B
|
|
# Skywork is a local-path AWQ output, not from HF Hub. Bind-mounted into the
|
|
# reward container at /local-models — see compose.yaml. No env var here for
|
|
# the model path itself since it's hard-coded in the compose command.
|
|
|
|
# GPU memory split — fractions are of TOTAL GPU memory, not free memory.
|
|
# vLLM profiles each service independently, so each slice must be large enough
|
|
# to fit that service's model + KV cache with no awareness of the others.
|
|
# Setting any one too low causes that container to OOM on KV cache allocation
|
|
# with `Available KV cache memory: -X.XX GiB`.
|
|
#
|
|
# Layout on a 48 GB Ada (~30% headroom kept free; matches current production):
|
|
# EMBED 0.20 (~9.6 GB) — 0.6B Qwen3 embed at 8k ctx; comfortable
|
|
# RERANK 0.20 (~9.6 GB) — 0.6B Qwen3 rerank at 8k ctx; comfortable
|
|
# REWARD 0.30 (~14 GB) — 8B Skywork AWQ at 16k ctx classify; tune up if OOM
|
|
EMBED_GPU_MEM_UTIL=0.20
|
|
RERANK_GPU_MEM_UTIL=0.20
|
|
REWARD_GPU_MEM_UTIL=0.30
|
|
|
|
# Context length caps — lower these if VRAM is tight.
|
|
# Qwen3-Embedding supports up to 32k; reranker up to 32k.
|
|
# Skywork capped at 16k server-side as defense-in-depth; JudgeClient also
|
|
# enforces the cap at dispatch time per spec.
|
|
EMBED_MAX_MODEL_LEN=8192
|
|
RERANK_MAX_MODEL_LEN=8192
|
|
REWARD_MAX_MODEL_LEN=16384
|
|
|
|
# Optional API key — leave blank for no auth (fine on the internal network).
|
|
# If set, all three services require `Authorization: Bearer <key>`.
|
|
API_KEY=
|
|
|
|
# HuggingFace token — only needed for gated models in the HF-Hub-loaded
|
|
# services (embed/rerank). Reward is local-path, ignores this.
|
|
HF_TOKEN=
|