vh
7e7130172e
Two related changes shipped together. The stack rename is independent but adding `vllm-reward` to the existing `vllm-qwen3` would have made that name actively misleading. **Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo references (README.md root, servers/ana-ml2/, stacks/llama-swap/, configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two intentional history mentions retained (servers/ana-ml2 + stacks/vllm README). **Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ on port 8003. The AWQ output is a locally-quantized model (not from HF), so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the shared HF cache. Model config.json declares LlamaForSequenceClassification which vLLM's pooling runner picks up automatically — produces a single reward score per input via /classify. **Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1 deprecated --task in favor of --runner pooling (model architecture in config.json drives the classification head). Compose uses --runner pooling with a comment explaining the substitution. **GPU memory:** no rebalance needed — production had already tuned EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB headroom on the 48 GB Ada. **Server-side:** brought existing vllm-qwen3 down, mv'd /opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed new compose via scripts/deploy-stack.sh, brought all 3 services up. **Smoke tests:** - /health on 8001/8002/8003 → 200 - /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ with max_model_len 16384 - /classify with a sample conversation → returns LABEL_0 with prob 0.9999 (single-output regression-style reward score, expected shape for a reward model)
134 lines
5.5 KiB
Markdown
134 lines
5.5 KiB
Markdown
# vllm
|
||
|
||
Multi-service vLLM stack on ana-ml2. Started as Qwen3 embedding + rerank
|
||
(replacing the unmaintained Infinity stack); generalized to host any vLLM
|
||
model on the box, currently three services:
|
||
|
||
- **`vllm-embed`** — Qwen3-Embedding 0.6B → OpenAI `/v1/embeddings`
|
||
- **`vllm-rerank`** — Qwen3-Reranker 0.6B → `/rerank` + `/score`
|
||
- **`vllm-reward`** — Skywork-Reward-V2-Llama-3.1-8B-AWQ → `/classify`
|
||
|
||
**Server:** ana-ml2
|
||
**Ports:** `8001` (embed), `8002` (rerank), `8003` (reward) — all configurable via `.env`
|
||
**GPU:** all three services share GPU 1 by default (configurable)
|
||
|
||
## Why one process per service
|
||
|
||
vLLM runs **one model per process**, so each model gets its own container.
|
||
All three pin to the same GPU and split VRAM via `--gpu-memory-utilization`.
|
||
Embed + rerank use `--runner pooling`; reward uses `--task classify` (newer
|
||
vLLM flag for sequence-classification heads).
|
||
|
||
## Reranker caveat
|
||
|
||
Qwen/Qwen3-Reranker-0.6B is a causal-LM checkpoint. The `--hf-overrides`
|
||
flag in `compose.yaml` re-maps it to `Qwen3ForSequenceClassification` so
|
||
vLLM's `/rerank` and `/score` endpoints work and the model emits only
|
||
`no`/`yes` class logits instead of the full 151k-token distribution.
|
||
|
||
If that override breaks after a vLLM upgrade, the pre-converted checkpoint
|
||
`tomaarsen/Qwen3-Reranker-0.6B-seq-cls` is a drop-in replacement that needs
|
||
no overrides — set `RERANK_MODEL=tomaarsen/Qwen3-Reranker-0.6B-seq-cls` in
|
||
`.env` and remove the `--hf-overrides` line from the compose.
|
||
|
||
## Reward / Skywork specifics
|
||
|
||
`vllm-reward` serves `Skywork-Reward-V2-Llama-3.1-8B-AWQ` — an AWQ
|
||
quantization produced locally (not pulled from HF). The quant output lives
|
||
at `/tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ` on ana-ml2 and
|
||
is bind-mounted read-only into the container at `/local-models`. vLLM
|
||
loads it as a local-path HF-format model (config.json + safetensors).
|
||
|
||
`--max-model-len 16384` is a server-side cap; JudgeClient on the consumer
|
||
side also enforces this at dispatch time. Defense-in-depth.
|
||
|
||
`--dtype auto` lets vLLM pick the right path for AWQ-quantized weights.
|
||
|
||
## Deploy
|
||
|
||
```bash
|
||
# Sync canonical → ana-ml2
|
||
scripts/deploy-stack.sh ana-ml2 vllm
|
||
|
||
# Pre-download Qwen3 models from HF (Skywork is local-path, no pull needed)
|
||
scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
|
||
--var hf_repo=Qwen/Qwen3-Embedding-0.6B
|
||
scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
|
||
--var hf_repo=Qwen/Qwen3-Reranker-0.6B
|
||
|
||
# Launch
|
||
ssh ana-ml2 'cd /opt/docker/compose/vllm && docker compose up -d && docker compose logs --tail=30'
|
||
```
|
||
|
||
First boot compiles CUDA graphs and can take 2–3 minutes per service
|
||
(longer for the 8B reward — `start_period: 240s`).
|
||
|
||
## Verify
|
||
|
||
```bash
|
||
# Health (each on its own port)
|
||
curl -s http://localhost:8001/health
|
||
curl -s http://localhost:8002/health
|
||
curl -s http://localhost:8003/health
|
||
|
||
# Embedding (OpenAI-compatible)
|
||
curl -s http://localhost:8001/v1/embeddings \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq .
|
||
|
||
# Reranker
|
||
curl -s http://localhost:8002/rerank \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq .
|
||
|
||
# Reward classify (Skywork)
|
||
curl -s http://localhost:8003/classify \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"model":"Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ","input":"User: hello\nAssistant: hi there"}' | jq .
|
||
|
||
# Listed models
|
||
curl -s http://localhost:8001/v1/models | jq .
|
||
curl -s http://localhost:8002/v1/models | jq .
|
||
curl -s http://localhost:8003/v1/models | jq .
|
||
```
|
||
|
||
## Scaling knobs
|
||
|
||
- **`EMBED_GPU_MEM_UTIL` / `RERANK_GPU_MEM_UTIL` / `REWARD_GPU_MEM_UTIL`** —
|
||
fractions of **total** GPU VRAM each service reserves (not of free VRAM).
|
||
Each profiler runs independently with no awareness of the others, so any
|
||
slice that's too small to fit `model + KV cache` will OOM the
|
||
second-to-start container with
|
||
`Available KV cache memory: -X.XX GiB`. Default 0.20/0.20/0.30 totals
|
||
0.70, leaving ~14 GB headroom on a 48 GB Ada (matches the production
|
||
tune-down from the original 0.40/0.40 defaults — embed/rerank fit
|
||
comfortably in 0.20 each). Bump REWARD up first if you see OOM — the
|
||
8B AWQ model's KV slice at 16k ctx is the tightest. Drop embed/rerank
|
||
further only if you've extended REWARD past 0.40 and still need room.
|
||
|
||
- **`EMBED_MAX_MODEL_LEN` / `RERANK_MAX_MODEL_LEN` / `REWARD_MAX_MODEL_LEN`** —
|
||
lower to reduce KV-cache allocation if VRAM is tight. Qwen3 supports up
|
||
to 32k natively; Skywork's reward cap of 16k is intentional (matches
|
||
JudgeClient's dispatch-side cap).
|
||
|
||
- **Larger Qwen3 models** — embedding/reranker come in 0.6B / 4B / 8B.
|
||
Swap `EMBED_MODEL` / `RERANK_MODEL` and bump the memory fractions
|
||
accordingly.
|
||
|
||
- **Separate GPUs** — if contention hurts latency, split them. Today all
|
||
three share `${GPU_ID}`. Adding `GPU_ID_REWARD`/etc. is a small compose
|
||
edit.
|
||
|
||
## History
|
||
|
||
- **2026-05-13 rename + extend** — stack renamed from `vllm-qwen3` to
|
||
`vllm` and gained the `vllm-reward` service (Skywork-Reward-V2 8B AWQ).
|
||
No GPU memory rebalance needed in practice — production had already
|
||
tuned EMBED/RERANK down from 0.40/0.40 to 0.20/0.20; adding REWARD at
|
||
0.30 fits cleanly with ~14 GB headroom on the 48 GB Ada.
|
||
|
||
- **Migrated off Infinity** — Infinity stopped shipping a `transformers`
|
||
build that knew Qwen3; this stack is the canonical replacement for both
|
||
embed and rerank. Consumers (AIPA agents, LibreChat RAG) point at
|
||
`:8001`/`:8002`.
|