vh
7bdac80878
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet from now on; fv-ml1 retires both once esh-ml1 is up. - stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports 8001/8013, fail-closed truncation (--auto-truncate false; embed --max-batch-tokens 32768). - litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address); reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target. - Verified via the gateway against the retiring fv-ml1 seats: embed cosine median 0.999927 (n=203); rerank top-1/top-3 29/30. - stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged. - Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md; CLAUDE.md gains the TEI convention.
140 lines
5.8 KiB
Markdown
140 lines
5.8 KiB
Markdown
# vllm
|
||
|
||
> ⚠ **2026-09-25: embedding + reranking LEFT this stack.** `vllm-embed` and
|
||
> `vllm-rerank-a3` were retired; the fleet's embed/rerank now runs on **TEI on
|
||
> esh-ml1** (`stacks/embed-rerank`), and TEI is the fleet engine for those from
|
||
> now on (Prime). What remains here is `vllm-reward` and `vllm-coder` on fv-ml1.
|
||
> The history below predates that.
|
||
|
||
Multi-service vLLM stack on ana-ml2. Started as Qwen3 embedding + rerank
|
||
(replacing the unmaintained Infinity stack); generalized to host any vLLM
|
||
model on the box, currently three services:
|
||
|
||
- **`vllm-embed`** — Qwen3-Embedding 0.6B → OpenAI `/v1/embeddings`
|
||
- **`vllm-rerank`** — Qwen3-Reranker 0.6B → `/rerank` + `/score`
|
||
- **`vllm-reward`** — Skywork-Reward-V2-Llama-3.1-8B-AWQ → `/classify`
|
||
|
||
**Server:** ana-ml2
|
||
**Ports:** `8001` (embed), `8002` (rerank), `8003` (reward) — all configurable via `.env`
|
||
**GPU:** all three services share GPU 1 by default (configurable)
|
||
|
||
## Why one process per service
|
||
|
||
vLLM runs **one model per process**, so each model gets its own container.
|
||
All three pin to the same GPU and split VRAM via `--gpu-memory-utilization`.
|
||
Embed + rerank use `--runner pooling`; reward uses `--task classify` (newer
|
||
vLLM flag for sequence-classification heads).
|
||
|
||
## Reranker caveat
|
||
|
||
Qwen/Qwen3-Reranker-0.6B is a causal-LM checkpoint. The `--hf-overrides`
|
||
flag in `compose.yaml` re-maps it to `Qwen3ForSequenceClassification` so
|
||
vLLM's `/rerank` and `/score` endpoints work and the model emits only
|
||
`no`/`yes` class logits instead of the full 151k-token distribution.
|
||
|
||
If that override breaks after a vLLM upgrade, the pre-converted checkpoint
|
||
`tomaarsen/Qwen3-Reranker-0.6B-seq-cls` is a drop-in replacement that needs
|
||
no overrides — set `RERANK_MODEL=tomaarsen/Qwen3-Reranker-0.6B-seq-cls` in
|
||
`.env` and remove the `--hf-overrides` line from the compose.
|
||
|
||
## Reward / Skywork specifics
|
||
|
||
`vllm-reward` serves `Skywork-Reward-V2-Llama-3.1-8B-AWQ` — an AWQ
|
||
quantization produced locally (not pulled from HF). The quant output lives
|
||
at `/tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ` on ana-ml2 and
|
||
is bind-mounted read-only into the container at `/local-models`. vLLM
|
||
loads it as a local-path HF-format model (config.json + safetensors).
|
||
|
||
`--max-model-len 16384` is a server-side cap; JudgeClient on the consumer
|
||
side also enforces this at dispatch time. Defense-in-depth.
|
||
|
||
`--dtype auto` lets vLLM pick the right path for AWQ-quantized weights.
|
||
|
||
## Deploy
|
||
|
||
```bash
|
||
# Sync canonical → ana-ml2
|
||
scripts/deploy-stack.sh ana-ml2 vllm
|
||
|
||
# Pre-download Qwen3 models from HF (Skywork is local-path, no pull needed)
|
||
scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
|
||
--var hf_repo=Qwen/Qwen3-Embedding-0.6B
|
||
scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
|
||
--var hf_repo=Qwen/Qwen3-Reranker-0.6B
|
||
|
||
# Launch
|
||
ssh ana-ml2 'cd /opt/docker/compose/vllm && docker compose up -d && docker compose logs --tail=30'
|
||
```
|
||
|
||
First boot compiles CUDA graphs and can take 2–3 minutes per service
|
||
(longer for the 8B reward — `start_period: 240s`).
|
||
|
||
## Verify
|
||
|
||
```bash
|
||
# Health (each on its own port)
|
||
curl -s http://localhost:8001/health
|
||
curl -s http://localhost:8002/health
|
||
curl -s http://localhost:8003/health
|
||
|
||
# Embedding (OpenAI-compatible)
|
||
curl -s http://localhost:8001/v1/embeddings \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq .
|
||
|
||
# Reranker
|
||
curl -s http://localhost:8002/rerank \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq .
|
||
|
||
# Reward classify (Skywork)
|
||
curl -s http://localhost:8003/classify \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"model":"Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ","input":"User: hello\nAssistant: hi there"}' | jq .
|
||
|
||
# Listed models
|
||
curl -s http://localhost:8001/v1/models | jq .
|
||
curl -s http://localhost:8002/v1/models | jq .
|
||
curl -s http://localhost:8003/v1/models | jq .
|
||
```
|
||
|
||
## Scaling knobs
|
||
|
||
- **`EMBED_GPU_MEM_UTIL` / `RERANK_GPU_MEM_UTIL` / `REWARD_GPU_MEM_UTIL`** —
|
||
fractions of **total** GPU VRAM each service reserves (not of free VRAM).
|
||
Each profiler runs independently with no awareness of the others, so any
|
||
slice that's too small to fit `model + KV cache` will OOM the
|
||
second-to-start container with
|
||
`Available KV cache memory: -X.XX GiB`. Default 0.20/0.20/0.30 totals
|
||
0.70, leaving ~14 GB headroom on a 48 GB Ada (matches the production
|
||
tune-down from the original 0.40/0.40 defaults — embed/rerank fit
|
||
comfortably in 0.20 each). Bump REWARD up first if you see OOM — the
|
||
8B AWQ model's KV slice at 16k ctx is the tightest. Drop embed/rerank
|
||
further only if you've extended REWARD past 0.40 and still need room.
|
||
|
||
- **`EMBED_MAX_MODEL_LEN` / `RERANK_MAX_MODEL_LEN` / `REWARD_MAX_MODEL_LEN`** —
|
||
lower to reduce KV-cache allocation if VRAM is tight. Qwen3 supports up
|
||
to 32k natively; Skywork's reward cap of 16k is intentional (matches
|
||
JudgeClient's dispatch-side cap).
|
||
|
||
- **Larger Qwen3 models** — embedding/reranker come in 0.6B / 4B / 8B.
|
||
Swap `EMBED_MODEL` / `RERANK_MODEL` and bump the memory fractions
|
||
accordingly.
|
||
|
||
- **Separate GPUs** — if contention hurts latency, split them. Today all
|
||
three share `${GPU_ID}`. Adding `GPU_ID_REWARD`/etc. is a small compose
|
||
edit.
|
||
|
||
## History
|
||
|
||
- **2026-05-13 rename + extend** — stack renamed from `vllm-qwen3` to
|
||
`vllm` and gained the `vllm-reward` service (Skywork-Reward-V2 8B AWQ).
|
||
No GPU memory rebalance needed in practice — production had already
|
||
tuned EMBED/RERANK down from 0.40/0.40 to 0.20/0.20; adding REWARD at
|
||
0.30 fits cleanly with ~14 GB headroom on the 48 GB Ada.
|
||
|
||
- **Migrated off Infinity** — Infinity stopped shipping a `transformers`
|
||
build that knew Qwen3; this stack is the canonical replacement for both
|
||
embed and rerank. Consumers (AIPA agents, LibreChat RAG) point at
|
||
`:8001`/`:8002`.
|