vh
c35f6510bf
Layout: GPU 0 = cyberprev (KV 16 -> 15 GiB, 1.29x), erp-seat (moved from GPU 1, KV 8.49 -> 5.5 GiB, 1.32x), voices (ctx 8,192 -> 40,960, its native maximum, KV 6 GiB, 1.07x), coder (moved from GPU 1, ctx 16,384 -> 32,768, the card maximum, 2.35x), parakeet-nemo. GPU 1 = gen-small (moved from GPU 0, 5.5 GiB, 1.76x), meromero-rp (22 GiB, 1.32x), intern-decision. Sizing rule: KV >= 1.25x max context rounded up to 0.5 GiB, checked against 7 days of LiteLLM traffic and the engines' own KV-usage logs. Each card keeps spare memory for its bursty tenant (parakeet's cap on GPU 0, intern-decision's 32k call on GPU 1). YaRN extensions are not enabled: they do not fit, and the Qwen cards warn static YaRN degrades short prompts. gen-small gets --override-generation-config max_new_tokens 32768. It applies only to requests that omit max_tokens (vLLM get_max_tokens) and answers 9 runaway ~260k-token replies in 7 days; the longest normal reply was 6,357. Applied with playbooks/fv-ml1-rebalance-20261005.yaml (boot-check guard and health/KV verification per seat); all nine seats healthy, gateway smoke tests pass.
140 lines
6.0 KiB
Markdown
140 lines
6.0 KiB
Markdown
# vllm
|
||
|
||
> ⚠ **2026-09-25: embedding + reranking LEFT this stack.** `vllm-embed` and
|
||
> `vllm-rerank-a3` were retired; the fleet's embed/rerank now runs on **TEI on
|
||
> esh-ml1** (`stacks/embed-rerank`), and TEI is the fleet engine for those from
|
||
> now on (Prime). What remains here is `vllm-reward` and `vllm-coder` on fv-ml1.
|
||
> The history below predates that.
|
||
|
||
Multi-service vLLM stack on ana-ml2. Started as Qwen3 embedding + rerank
|
||
(replacing the unmaintained Infinity stack); generalized to host any vLLM
|
||
model on the box, currently three services:
|
||
|
||
- **`vllm-embed`** — Qwen3-Embedding 0.6B → OpenAI `/v1/embeddings`
|
||
- **`vllm-rerank`** — Qwen3-Reranker 0.6B → `/rerank` + `/score`
|
||
- **`vllm-reward`** — Skywork-Reward-V2-Llama-3.1-8B-AWQ → `/classify`
|
||
|
||
**Server:** ana-ml2
|
||
**Ports:** `8001` (embed), `8002` (rerank), `8003` (reward) — all configurable via `.env`
|
||
**GPU:** only `vllm-coder` is live in this stack; since 2026-10-05 it runs on GPU 0 (`CODER_GPU_ID`) at the card-published maximum context of 32,768 (KV pinned 77,056 tokens, 2.35×). The embed/rerank/reward services here are retired
|
||
|
||
## Why one process per service
|
||
|
||
vLLM runs **one model per process**, so each model gets its own container.
|
||
All three pin to the same GPU and split VRAM via `--gpu-memory-utilization`.
|
||
Embed + rerank use `--runner pooling`; reward uses `--task classify` (newer
|
||
vLLM flag for sequence-classification heads).
|
||
|
||
## Reranker caveat
|
||
|
||
Qwen/Qwen3-Reranker-0.6B is a causal-LM checkpoint. The `--hf-overrides`
|
||
flag in `compose.yaml` re-maps it to `Qwen3ForSequenceClassification` so
|
||
vLLM's `/rerank` and `/score` endpoints work and the model emits only
|
||
`no`/`yes` class logits instead of the full 151k-token distribution.
|
||
|
||
If that override breaks after a vLLM upgrade, the pre-converted checkpoint
|
||
`tomaarsen/Qwen3-Reranker-0.6B-seq-cls` is a drop-in replacement that needs
|
||
no overrides — set `RERANK_MODEL=tomaarsen/Qwen3-Reranker-0.6B-seq-cls` in
|
||
`.env` and remove the `--hf-overrides` line from the compose.
|
||
|
||
## Reward / Skywork specifics
|
||
|
||
`vllm-reward` serves `Skywork-Reward-V2-Llama-3.1-8B-AWQ` — an AWQ
|
||
quantization produced locally (not pulled from HF). The quant output lives
|
||
at `/tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ` on ana-ml2 and
|
||
is bind-mounted read-only into the container at `/local-models`. vLLM
|
||
loads it as a local-path HF-format model (config.json + safetensors).
|
||
|
||
`--max-model-len 16384` is a server-side cap; JudgeClient on the consumer
|
||
side also enforces this at dispatch time. Defense-in-depth.
|
||
|
||
`--dtype auto` lets vLLM pick the right path for AWQ-quantized weights.
|
||
|
||
## Deploy
|
||
|
||
```bash
|
||
# Sync canonical → ana-ml2
|
||
scripts/deploy-stack.sh ana-ml2 vllm
|
||
|
||
# Pre-download Qwen3 models from HF (Skywork is local-path, no pull needed)
|
||
scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
|
||
--var hf_repo=Qwen/Qwen3-Embedding-0.6B
|
||
scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
|
||
--var hf_repo=Qwen/Qwen3-Reranker-0.6B
|
||
|
||
# Launch
|
||
ssh ana-ml2 'cd /opt/docker/compose/vllm && docker compose up -d && docker compose logs --tail=30'
|
||
```
|
||
|
||
First boot compiles CUDA graphs and can take 2–3 minutes per service
|
||
(longer for the 8B reward — `start_period: 240s`).
|
||
|
||
## Verify
|
||
|
||
```bash
|
||
# Health (each on its own port)
|
||
curl -s http://localhost:8001/health
|
||
curl -s http://localhost:8002/health
|
||
curl -s http://localhost:8003/health
|
||
|
||
# Embedding (OpenAI-compatible)
|
||
curl -s http://localhost:8001/v1/embeddings \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq .
|
||
|
||
# Reranker
|
||
curl -s http://localhost:8002/rerank \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq .
|
||
|
||
# Reward classify (Skywork)
|
||
curl -s http://localhost:8003/classify \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"model":"Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ","input":"User: hello\nAssistant: hi there"}' | jq .
|
||
|
||
# Listed models
|
||
curl -s http://localhost:8001/v1/models | jq .
|
||
curl -s http://localhost:8002/v1/models | jq .
|
||
curl -s http://localhost:8003/v1/models | jq .
|
||
```
|
||
|
||
## Scaling knobs
|
||
|
||
- **`EMBED_GPU_MEM_UTIL` / `RERANK_GPU_MEM_UTIL` / `REWARD_GPU_MEM_UTIL`** —
|
||
fractions of **total** GPU VRAM each service reserves (not of free VRAM).
|
||
Each profiler runs independently with no awareness of the others, so any
|
||
slice that's too small to fit `model + KV cache` will OOM the
|
||
second-to-start container with
|
||
`Available KV cache memory: -X.XX GiB`. Default 0.20/0.20/0.30 totals
|
||
0.70, leaving ~14 GB headroom on a 48 GB Ada (matches the production
|
||
tune-down from the original 0.40/0.40 defaults — embed/rerank fit
|
||
comfortably in 0.20 each). Bump REWARD up first if you see OOM — the
|
||
8B AWQ model's KV slice at 16k ctx is the tightest. Drop embed/rerank
|
||
further only if you've extended REWARD past 0.40 and still need room.
|
||
|
||
- **`EMBED_MAX_MODEL_LEN` / `RERANK_MAX_MODEL_LEN` / `REWARD_MAX_MODEL_LEN`** —
|
||
lower to reduce KV-cache allocation if VRAM is tight. Qwen3 supports up
|
||
to 32k natively; Skywork's reward cap of 16k is intentional (matches
|
||
JudgeClient's dispatch-side cap).
|
||
|
||
- **Larger Qwen3 models** — embedding/reranker come in 0.6B / 4B / 8B.
|
||
Swap `EMBED_MODEL` / `RERANK_MODEL` and bump the memory fractions
|
||
accordingly.
|
||
|
||
- **Separate GPUs** — if contention hurts latency, split them. Today all
|
||
three share `${GPU_ID}`. Adding `GPU_ID_REWARD`/etc. is a small compose
|
||
edit.
|
||
|
||
## History
|
||
|
||
- **2026-05-13 rename + extend** — stack renamed from `vllm-qwen3` to
|
||
`vllm` and gained the `vllm-reward` service (Skywork-Reward-V2 8B AWQ).
|
||
No GPU memory rebalance needed in practice — production had already
|
||
tuned EMBED/RERANK down from 0.40/0.40 to 0.20/0.20; adding REWARD at
|
||
0.30 fits cleanly with ~14 GB headroom on the 48 GB Ada.
|
||
|
||
- **Migrated off Infinity** — Infinity stopped shipping a `transformers`
|
||
build that knew Qwen3; this stack is the canonical replacement for both
|
||
embed and rerank. Consumers (AIPA agents, LibreChat RAG) point at
|
||
`:8001`/`:8002`.
|