esh-pfi-infrastructure/stacks/vllm-qwen3/README.md

vllm-qwen3

Qwen3 embedding + reranker served via vLLM. Replaces the unmaintained Infinity stack.

Server: ana-ml2 Ports: 8001 (embed), 8002 (rerank) — both configurable via .env GPU: both services share GPU 1 by default (configurable)

Why two services

vLLM runs one model per process, so embedding and reranking each get their own container. Both pin to the same GPU and split VRAM via --gpu-memory-utilization. Both use --runner pooling so the OpenAI server exposes /v1/embeddings (for the embedder) and /rerank, /score (for the reranker, which also needs the --hf-overrides described below).

Reranker caveat

Qwen/Qwen3-Reranker-0.6B is a causal-LM checkpoint. The --hf-overrides flag in compose.yaml re-maps it to Qwen3ForSequenceClassification so vLLM's /rerank and /score endpoints work and the model emits only no/yes class logits instead of the full 151k-token distribution.

If that override breaks after a vLLM upgrade, the pre-converted checkpoint tomaarsen/Qwen3-Reranker-0.6B-seq-cls is a drop-in replacement that needs no overrides — set RERANK_MODEL=tomaarsen/Qwen3-Reranker-0.6B-seq-cls in .env and remove the --hf-overrides line from the compose.

Deploy

# On ana-ml2:
sudo mkdir -p /opt/docker/compose/vllm-qwen3
sudo chown $USER /opt/docker/compose/vllm-qwen3
cd /opt/docker/compose/vllm-qwen3

# Copy compose.yaml + .env.example here (e.g. via scp from this workspace)
cp .env.example .env
# edit .env — pick GPU, ports, memory split, etc.

# Pre-download models (optional, speeds first boot)
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^EMBED_MODEL .env | cut -d= -f2)"
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^RERANK_MODEL .env | cut -d= -f2)"

# Dry-parse
docker compose config

# Launch
docker compose up -d
docker compose logs -f

First boot compiles CUDA graphs and can take 2–3 minutes per service. The start_period: 180s healthcheck grace reflects that.

Verify

# Health
curl -s http://localhost:8001/health
curl -s http://localhost:8002/health

# Embedding (OpenAI-compatible)
curl -s http://localhost:8001/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq .

# Reranker
curl -s http://localhost:8002/rerank \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq .

# Listed models
curl -s http://localhost:8001/v1/models | jq .
curl -s http://localhost:8002/v1/models | jq .

Scaling knobs

• EMBED_GPU_MEM_UTIL / RERANK_GPU_MEM_UTIL — fractions of total GPU VRAM each service reserves (not of free VRAM). Both services profile independently, so each slice must be large enough to fit that service's model + KV cache with no knowledge of the other. Setting them too low causes the second-to-start container to OOM on KV cache allocation with Available KV cache memory: -X.XX GiB. Default 0.40/0.40 (= 0.80 total) leaves ~20% GPU headroom and works cleanly for the 0.6B pair at 8k context; raise for 4B/8B variants or drop max-model-len if you need more room.
• EMBED_MAX_MODEL_LEN / RERANK_MAX_MODEL_LEN — lower to reduce KV-cache allocation if VRAM is tight. Qwen3 supports up to 32k natively.
• Larger models — Qwen3-Embedding/Reranker come in 0.6B / 4B / 8B. Swap EMBED_MODEL / RERANK_MODEL and bump the memory fractions accordingly.
• Separate GPUs — if contention hurts latency, split them: add a second GPU_ID_RERANK variable and point each service at its own device. (Requires a small compose edit; currently both share ${GPU_ID}.)

Migrating off Infinity

Once this stack is verified stable:

1. Stop the infinity stack (docker compose down under /opt/docker/compose/infinity/).
2. Update consumers (AIPA agents, LibreChat RAG) to point at :8001 for embeddings and :8002 for rerank.
3. Delete stacks/infinity/ from this workspace.