esh-pfi-infrastructure/stacks/vllm

vllm

Multi-service vLLM stack on ana-ml2. Started as Qwen3 embedding + rerank (replacing the unmaintained Infinity stack); generalized to host any vLLM model on the box, currently three services:

• vllm-embed — Qwen3-Embedding 0.6B → OpenAI /v1/embeddings
• vllm-rerank — Qwen3-Reranker 0.6B → /rerank + /score
• vllm-reward — Skywork-Reward-V2-Llama-3.1-8B-AWQ → /classify

Server: ana-ml2 Ports: 8001 (embed), 8002 (rerank), 8003 (reward) — all configurable via .env GPU: all three services share GPU 1 by default (configurable)

Why one process per service

vLLM runs one model per process, so each model gets its own container. All three pin to the same GPU and split VRAM via --gpu-memory-utilization. Embed + rerank use --runner pooling; reward uses --task classify (newer vLLM flag for sequence-classification heads).

Reranker caveat

Qwen/Qwen3-Reranker-0.6B is a causal-LM checkpoint. The --hf-overrides flag in compose.yaml re-maps it to Qwen3ForSequenceClassification so vLLM's /rerank and /score endpoints work and the model emits only no/yes class logits instead of the full 151k-token distribution.

If that override breaks after a vLLM upgrade, the pre-converted checkpoint tomaarsen/Qwen3-Reranker-0.6B-seq-cls is a drop-in replacement that needs no overrides — set RERANK_MODEL=tomaarsen/Qwen3-Reranker-0.6B-seq-cls in .env and remove the --hf-overrides line from the compose.

Reward / Skywork specifics

vllm-reward serves Skywork-Reward-V2-Llama-3.1-8B-AWQ — an AWQ quantization produced locally (not pulled from HF). The quant output lives at /tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ on ana-ml2 and is bind-mounted read-only into the container at /local-models. vLLM loads it as a local-path HF-format model (config.json + safetensors).

--max-model-len 16384 is a server-side cap; JudgeClient on the consumer side also enforces this at dispatch time. Defense-in-depth.

--dtype auto lets vLLM pick the right path for AWQ-quantized weights.

Deploy

# Sync canonical → ana-ml2
scripts/deploy-stack.sh ana-ml2 vllm

# Pre-download Qwen3 models from HF (Skywork is local-path, no pull needed)
scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
    --var hf_repo=Qwen/Qwen3-Embedding-0.6B
scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
    --var hf_repo=Qwen/Qwen3-Reranker-0.6B

# Launch
ssh ana-ml2 'cd /opt/docker/compose/vllm && docker compose up -d && docker compose logs --tail=30'

First boot compiles CUDA graphs and can take 2–3 minutes per service (longer for the 8B reward — start_period: 240s).

Verify

# Health (each on its own port)
curl -s http://localhost:8001/health
curl -s http://localhost:8002/health
curl -s http://localhost:8003/health

# Embedding (OpenAI-compatible)
curl -s http://localhost:8001/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq .

# Reranker
curl -s http://localhost:8002/rerank \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq .

# Reward classify (Skywork)
curl -s http://localhost:8003/classify \
  -H "Content-Type: application/json" \
  -d '{"model":"Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ","input":"User: hello\nAssistant: hi there"}' | jq .

# Listed models
curl -s http://localhost:8001/v1/models | jq .
curl -s http://localhost:8002/v1/models | jq .
curl -s http://localhost:8003/v1/models | jq .

Scaling knobs

• EMBED_GPU_MEM_UTIL / RERANK_GPU_MEM_UTIL / REWARD_GPU_MEM_UTIL — fractions of total GPU VRAM each service reserves (not of free VRAM). Each profiler runs independently with no awareness of the others, so any slice that's too small to fit model + KV cache will OOM the second-to-start container with Available KV cache memory: -X.XX GiB. Default 0.20/0.20/0.30 totals 0.70, leaving ~14 GB headroom on a 48 GB Ada (matches the production tune-down from the original 0.40/0.40 defaults — embed/rerank fit comfortably in 0.20 each). Bump REWARD up first if you see OOM — the 8B AWQ model's KV slice at 16k ctx is the tightest. Drop embed/rerank further only if you've extended REWARD past 0.40 and still need room.

• EMBED_MAX_MODEL_LEN / RERANK_MAX_MODEL_LEN / REWARD_MAX_MODEL_LEN — lower to reduce KV-cache allocation if VRAM is tight. Qwen3 supports up to 32k natively; Skywork's reward cap of 16k is intentional (matches JudgeClient's dispatch-side cap).

• Larger Qwen3 models — embedding/reranker come in 0.6B / 4B / 8B. Swap EMBED_MODEL / RERANK_MODEL and bump the memory fractions accordingly.

• Separate GPUs — if contention hurts latency, split them. Today all three share ${GPU_ID}. Adding GPU_ID_REWARD/etc. is a small compose edit.

History

• 2026-05-13 rename + extend — stack renamed from vllm-qwen3 to vllm and gained the vllm-reward service (Skywork-Reward-V2 8B AWQ). No GPU memory rebalance needed in practice — production had already tuned EMBED/RERANK down from 0.40/0.40 to 0.20/0.20; adding REWARD at 0.30 fits cleanly with ~14 GB headroom on the 48 GB Ada.

• Migrated off Infinity — Infinity stopped shipping a transformers build that knew Qwen3; this stack is the canonical replacement for both embed and rerank. Consumers (AIPA agents, LibreChat RAG) point at :8001/:8002.