# vllm

Multi-service vLLM stack on ana-ml2. Started as Qwen3 embedding + rerank
(replacing the unmaintained Infinity stack); generalized to host any vLLM
model on the box, currently three services:

- **`vllm-embed`** — Qwen3-Embedding 0.6B → OpenAI `/v1/embeddings`
- **`vllm-rerank`** — Qwen3-Reranker 0.6B → `/rerank` + `/score`
- **`vllm-reward`** — Skywork-Reward-V2-Llama-3.1-8B-AWQ → `/classify`

**Server:** ana-ml2
**Ports:** `8001` (embed), `8002` (rerank), `8003` (reward) — all configurable via `.env`
**GPU:** all three services share GPU 1 by default (configurable)

## Why one process per service

vLLM runs **one model per process**, so each model gets its own container.
All three pin to the same GPU and split VRAM via `--gpu-memory-utilization`.
Embed + rerank use `--runner pooling`; reward uses `--task classify` (newer
vLLM flag for sequence-classification heads).

## Reranker caveat

Qwen/Qwen3-Reranker-0.6B is a causal-LM checkpoint. The `--hf-overrides`
flag in `compose.yaml` re-maps it to `Qwen3ForSequenceClassification` so
vLLM's `/rerank` and `/score` endpoints work and the model emits only
`no`/`yes` class logits instead of the full 151k-token distribution.

If that override breaks after a vLLM upgrade, the pre-converted checkpoint
`tomaarsen/Qwen3-Reranker-0.6B-seq-cls` is a drop-in replacement that needs
no overrides — set `RERANK_MODEL=tomaarsen/Qwen3-Reranker-0.6B-seq-cls` in
`.env` and remove the `--hf-overrides` line from the compose.

## Reward / Skywork specifics

`vllm-reward` serves `Skywork-Reward-V2-Llama-3.1-8B-AWQ` — an AWQ
quantization produced locally (not pulled from HF). The quant output lives
at `/tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ` on ana-ml2 and
is bind-mounted read-only into the container at `/local-models`. vLLM
loads it as a local-path HF-format model (config.json + safetensors).

`--max-model-len 16384` is a server-side cap; JudgeClient on the consumer
side also enforces this at dispatch time. Defense-in-depth.

`--dtype auto` lets vLLM pick the right path for AWQ-quantized weights.

## Deploy

```bash
# Sync canonical → ana-ml2
scripts/deploy-stack.sh ana-ml2 vllm

# Pre-download Qwen3 models from HF (Skywork is local-path, no pull needed)
scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
    --var hf_repo=Qwen/Qwen3-Embedding-0.6B
scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
    --var hf_repo=Qwen/Qwen3-Reranker-0.6B

# Launch
ssh ana-ml2 'cd /opt/docker/compose/vllm && docker compose up -d && docker compose logs --tail=30'
```

First boot compiles CUDA graphs and can take 2–3 minutes per service
(longer for the 8B reward — `start_period: 240s`).

## Verify

```bash
# Health (each on its own port)
curl -s http://localhost:8001/health
curl -s http://localhost:8002/health
curl -s http://localhost:8003/health

# Embedding (OpenAI-compatible)
curl -s http://localhost:8001/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq .

# Reranker
curl -s http://localhost:8002/rerank \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq .

# Reward classify (Skywork)
curl -s http://localhost:8003/classify \
  -H "Content-Type: application/json" \
  -d '{"model":"Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ","input":"User: hello\nAssistant: hi there"}' | jq .

# Listed models
curl -s http://localhost:8001/v1/models | jq .
curl -s http://localhost:8002/v1/models | jq .
curl -s http://localhost:8003/v1/models | jq .
```

## Scaling knobs

- **`EMBED_GPU_MEM_UTIL` / `RERANK_GPU_MEM_UTIL` / `REWARD_GPU_MEM_UTIL`** —
  fractions of **total** GPU VRAM each service reserves (not of free VRAM).
  Each profiler runs independently with no awareness of the others, so any
  slice that's too small to fit `model + KV cache` will OOM the
  second-to-start container with
  `Available KV cache memory: -X.XX GiB`. Default 0.20/0.20/0.30 totals
  0.70, leaving ~14 GB headroom on a 48 GB Ada (matches the production
  tune-down from the original 0.40/0.40 defaults — embed/rerank fit
  comfortably in 0.20 each). Bump REWARD up first if you see OOM — the
  8B AWQ model's KV slice at 16k ctx is the tightest. Drop embed/rerank
  further only if you've extended REWARD past 0.40 and still need room.

- **`EMBED_MAX_MODEL_LEN` / `RERANK_MAX_MODEL_LEN` / `REWARD_MAX_MODEL_LEN`** —
  lower to reduce KV-cache allocation if VRAM is tight. Qwen3 supports up
  to 32k natively; Skywork's reward cap of 16k is intentional (matches
  JudgeClient's dispatch-side cap).

- **Larger Qwen3 models** — embedding/reranker come in 0.6B / 4B / 8B.
  Swap `EMBED_MODEL` / `RERANK_MODEL` and bump the memory fractions
  accordingly.

- **Separate GPUs** — if contention hurts latency, split them. Today all
  three share `${GPU_ID}`. Adding `GPU_ID_REWARD`/etc. is a small compose
  edit.

## History

- **2026-05-13 rename + extend** — stack renamed from `vllm-qwen3` to
  `vllm` and gained the `vllm-reward` service (Skywork-Reward-V2 8B AWQ).
  No GPU memory rebalance needed in practice — production had already
  tuned EMBED/RERANK down from 0.40/0.40 to 0.20/0.20; adding REWARD at
  0.30 fits cleanly with ~14 GB headroom on the 48 GB Ada.

- **Migrated off Infinity** — Infinity stopped shipping a `transformers`
  build that knew Qwen3; this stack is the canonical replacement for both
  embed and rerank. Consumers (AIPA agents, LibreChat RAG) point at
  `:8001`/`:8002`.