vllm: rename stack from vllm-qwen3 → vllm + add Skywork reward classifier
Two related changes shipped together. The stack rename is independent but adding `vllm-reward` to the existing `vllm-qwen3` would have made that name actively misleading. **Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo references (README.md root, servers/ana-ml2/, stacks/llama-swap/, configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two intentional history mentions retained (servers/ana-ml2 + stacks/vllm README). **Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ on port 8003. The AWQ output is a locally-quantized model (not from HF), so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the shared HF cache. Model config.json declares LlamaForSequenceClassification which vLLM's pooling runner picks up automatically — produces a single reward score per input via /classify. **Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1 deprecated --task in favor of --runner pooling (model architecture in config.json drives the classification head). Compose uses --runner pooling with a comment explaining the substitution. **GPU memory:** no rebalance needed — production had already tuned EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB headroom on the 48 GB Ada. **Server-side:** brought existing vllm-qwen3 down, mv'd /opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed new compose via scripts/deploy-stack.sh, brought all 3 services up. **Smoke tests:** - /health on 8001/8002/8003 → 200 - /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ with max_model_len 16384 - /classify with a sample conversation → returns LABEL_0 with prob 0.9999 (single-output regression-style reward score, expected shape for a reward model)
This commit is contained in:
@@ -66,7 +66,7 @@ Per-host snapshots of the running system live under `servers/<host>/system-detai
|
||||
|
||||
**GPU (ana-ml2):**
|
||||
- `llama-swap` — GGUF model swapper via llama.cpp (port 9292)
|
||||
- `vllm-qwen3` — embeddings (8001) + reranker (8002) via vLLM
|
||||
- `vllm` — embeddings (8001) + reranker (8002) + Skywork reward classifier (8003) via vLLM
|
||||
|
||||
**Anaheim non-GPU (ana-docker):**
|
||||
- `traefik`, `crowdsec`, `gitea`, `vaultwarden`, `synapse`, `seafile`, `searxng`, `openwebui`, `sillytavern`, `mailrise`, `rustdesk`, `dockge`, `it-tools`
|
||||
@@ -77,7 +77,7 @@ Per-host snapshots of the running system live under `servers/<host>/system-detai
|
||||
- Backup target: `rest-server-ana` on port 8000
|
||||
|
||||
**GPU ana-ml2 (non-canonical for now):**
|
||||
- `comfyui`, `kokoro`, `parakeet`, `vibevoice` alongside the canonical `llama-swap` + `vllm-qwen3`
|
||||
- `comfyui`, `kokoro`, `parakeet`, `vibevoice` alongside the canonical `llama-swap` + `vllm`
|
||||
|
||||
**NH3 (nh3-docker):**
|
||||
- `adguard`, `dockge`, plus Beszel/Dozzle agents
|
||||
|
||||
@@ -29,7 +29,7 @@ pre-backup DB hook is required. Contrast with `configs/restic/ana-docker/`
|
||||
where synapse/seafile/vaultwarden DB dumps run first.
|
||||
|
||||
- `llama-swap` — GGUF swapper (llama.cpp)
|
||||
- `vllm-qwen3` — embedding + rerank
|
||||
- `vllm` — embedding + rerank + reward classifier
|
||||
- `comfyui`, `kokoro`, `parakeet`, `vibevoice`
|
||||
- `beszel-agent-ana`, `dozzle-agent-ana`, `dockge`
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
# caches, llama.cpp GGUFs, ComfyUI models, etc.) — all regenerable
|
||||
# from upstream. Backing them up would blow the repo size budget.
|
||||
# - No DB dumps needed. None of the stacks on this host (llama-swap,
|
||||
# vllm-qwen3, comfyui, kokoro, parakeet, vibevoice, beszel-agent,
|
||||
# vllm, comfyui, kokoro, parakeet, vibevoice, beszel-agent,
|
||||
# dozzle-agent, dockge) store relational data.
|
||||
|
||||
version: "1"
|
||||
|
||||
@@ -209,7 +209,7 @@ Essentially **all Anaheim primary services** go offline. Because ana-nas lives h
|
||||
|
||||
### ana-ml2 (bare metal Supermicro, 10.250.50.54, BMC 10.250.250.50)
|
||||
|
||||
**Blast radius:** AI inference services (llama-swap, vllm-qwen3). Consumer-facing chat/embedding endpoints fail.
|
||||
**Blast radius:** AI inference services (llama-swap, vllm). Consumer-facing chat/embedding/reward-scoring endpoints fail.
|
||||
|
||||
**Recovery:**
|
||||
1. Check OS via SSH. If unresponsive, BMC console at <https://10.250.250.50>.
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-05-08_
|
||||
|
||||
## Repo purpose
|
||||
|
||||
Reference workspace for PFI infrastructure: server inventory, canonical
|
||||
Docker Compose stacks, ops playbooks, and conventions. Authoritative
|
||||
copies of compose files live on the servers under
|
||||
`/opt/docker/compose/<stack>/`; this repo mirrors them for version
|
||||
control, editing, planning, and CI-driven deploys.
|
||||
|
||||
## Tools and conventions
|
||||
|
||||
Sister repos (separate gitea repos, deployed by playbooks here):
|
||||
|
||||
| Repo | Role | CI status |
|
||||
|---|---|---|
|
||||
| `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) |
|
||||
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
|
||||
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
|
||||
|
||||
- **Two-layer backups** — Backrest orchestrates restic for file+DB (5
|
||||
fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
|
||||
VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
|
||||
cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
|
||||
for the blast-radius matrix.
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-05-08:_
|
||||
|
||||
- **CI/CD pipe is live for three repos** (task-board, vor, nevermore).
|
||||
Pattern: workflow checks out triggering repo + `vh/esh-pfi-infrastructure`
|
||||
(for elway), SSHes to ana-docker, runs an elway playbook pinned to the
|
||||
triggering commit SHA. User-scope secrets `DEPLOY_SSH_KEY` and
|
||||
`MGMT_REPO_TOKEN` cover all repos under `vh/`.
|
||||
- **Open**: rotate `MINIFLUX_PASSWORD` — leaked in a prior session's
|
||||
transcript while seeding nevermore's `.env` (full leak once, partial
|
||||
leak once).
|
||||
- **Open**: clean up old `news-digest` detritus on ana-docker —
|
||||
`/opt/docker/compose/news-digest/`, `/opt/docker/data/news-digest/`,
|
||||
image `local/news-digest:v5`. Migrate the historical `edition-*.html`
|
||||
archive files into `/opt/docker/data/nevermore/` first.
|
||||
- **Watch**: nh3-nas `/volume1` at 65%; plan retention review or
|
||||
capacity expansion before ~80%.
|
||||
- **Possible next**: aggregator service + homepage widget for live
|
||||
"fleet backup health" dashboard. Discussed but not committed; user
|
||||
weighing against a `/schedule` weekly digest as the lighter
|
||||
alternative.
|
||||
- **Open mystery**: `docker push` from outside the LAN to
|
||||
`gitea.phasefinal.com` has a **~60s client-side per-PATCH ceiling**
|
||||
for chunked blob uploads. We confirmed the timer is client-side
|
||||
(Traefik logged `499 / 60012ms` — client closed; Gitea logged the
|
||||
same as `unexpected EOF`). Where exactly the 60s lives in the docker
|
||||
daemon / containerd stack — and whether it's tunable — is
|
||||
unidentified. Worked around for now by shrinking images below the
|
||||
cliff; if it bites again, instrument with `dockerd -D` + strace on a
|
||||
fresh push to find the actual timer.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-05-08]` Pin `torch` to CPU-only via
|
||||
`--index-url https://download.pytorch.org/whl/cpu` in
|
||||
`Worldtree/Dockerfile` before the rest of `requirements.txt`.
|
||||
`sentence-transformers` transitively pulls torch and grabs the CUDA
|
||||
flavor by default (~2.5GB of nvidia libs). Worldtree runs on
|
||||
`ana-docker` (no GPU reservation in `compose.yaml`) so CUDA is dead
|
||||
weight. Image: 5.62GB → 1.6GB. The failing push from off-LAN now
|
||||
succeeds.
|
||||
- `[2026-05-08]` Split `Worldtree/requirements.txt` into runtime +
|
||||
`requirements-dev.txt` (pytest, pytest-asyncio, ruff, pylint).
|
||||
~50MB out of the runtime image; smaller attack surface.
|
||||
- `[2026-05-01]` nevermore **replaces** news-digest at port 8181 (not
|
||||
coexist on 8182). User chose one stack at a time over soak window.
|
||||
~30s briefing gap accepted as the cost.
|
||||
- `[2026-04-30]` Bumped nevermore deploy playbook's first-render wait
|
||||
from 60s to 6 min — first deploy with empty `/output` requires a
|
||||
worker LLM cycle (2-4 min) before `index.html` exists.
|
||||
- `[2026-04-29]` Workflow drops explicit `container:` directive after
|
||||
runner re-registration with `:docker://node:20-bookworm-slim` schema
|
||||
labels — single source of truth for the build environment.
|
||||
- `[2026-04-29]` Deploy playbooks accept SHA refs in addition to branch
|
||||
names (`git rev-parse <ref>` then `origin/<ref>^{commit}` fallback).
|
||||
CI passes `--var ref=${{ github.sha }}`; manual runs pass `main` /
|
||||
`v0.1.0`. Same code path either way.
|
||||
- `[2026-04-29]` Single central Gitea Actions runner on ana-docker
|
||||
(`f014d55`). Parameterized `playbooks/deploy-gitea-runner.yaml` so
|
||||
site-local runners (nh3, esh) drop in via `--var` overrides — not
|
||||
copy-pasted playbooks.
|
||||
- `[2026-04-29]` `nevermore` carries forward `hidden.json` and
|
||||
historical `MINIFLUX_PASSWORD` from `news-digest` during cutover
|
||||
(preserves user state across stack rename).
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT
|
||||
timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero
|
||||
matches and led to a wrong "no /v2/ traffic in 12 days" conclusion.
|
||||
**Gitea logs in PDT, Traefik logs in UTC** — same host, different
|
||||
timezones. Always normalize timezones (UTC) when correlating logs
|
||||
across services on the same box. Cost: ~30 min in the wrong
|
||||
direction.
|
||||
- `[2026-05-08]` Validating a user-proposed Traefik/Gitea timeout bump
|
||||
for "Traefik is dropping connections on big docker pushes" without
|
||||
first verifying which component was actually in the failure path.
|
||||
Traefik turned out to be innocent (`499 / 60012ms` = client closed,
|
||||
Traefik never timed out), Gitea's `PER_WRITE_TIMEOUT` governs
|
||||
response writes (wrong direction), and the real ceiling was a ~60s
|
||||
client-side timer no compose change can reach. **Lesson: validate
|
||||
the diagnostic premise — which component is actually in the failure
|
||||
path? — before refining the proposed fix.**
|
||||
- `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` /
|
||||
`PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on
|
||||
`/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern
|
||||
**response writes**, not request body reads. `unexpected EOF` from
|
||||
Go's HTTP server means the client closed mid-body-upload; not a
|
||||
knob Gitea exposes server-side.
|
||||
- `[2026-04-30]` task-board workflow with
|
||||
`container: image: debian:bookworm-slim` — fails:
|
||||
`actions/checkout@v4` needs `node` at runtime, slim image lacks it.
|
||||
Switched to `node:20-bookworm-slim` (has node + apt) or runner-label
|
||||
default.
|
||||
- `[2026-04-30]` Dropping the `container:` directive before runner
|
||||
re-registration with docker-schema labels — runner silently falls
|
||||
back to **host mode** (jobs run inside the alpine `act_runner`
|
||||
container itself, no apt). The `:host` suffix in startup logs
|
||||
(`labels updated to: [pfi-fleet:host ana-docker:host]`) is the
|
||||
giveaway. Fix: register with `pfi-fleet:docker://<image>` schema
|
||||
labels.
|
||||
- `[2026-04-30]` Updating runner labels by editing `.env` and bouncing
|
||||
— doesn't take. The `.runner` registration cache pins labels at
|
||||
first registration; env-var updates are read each start but the
|
||||
stored token + UUID are tied to the original label set on the gitea
|
||||
side. Fix: stop runner, delete `.runner`, generate new admin
|
||||
registration token, redeploy.
|
||||
- `[2026-04-30]` `git reset --hard origin/<sha>` in
|
||||
`deploy-task-board.yaml` (and the in-repo nevermore playbook before
|
||||
fix) — invalid syntax: `origin/` prefix only works for branch refs.
|
||||
SHAs need `git reset --hard <sha>` directly. Resolved with
|
||||
`git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first,
|
||||
then bare `"{{ ref }}^{commit}"` fallback.
|
||||
- `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after
|
||||
task-board wiring — it was actually only repo-scope on
|
||||
`vh/task-board`. vor's first CI run failed with empty SSH key
|
||||
(`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user
|
||||
scope at `gitea.phasefinal.com/user/settings/actions/secrets`.
|
||||
- `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity —
|
||||
leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a
|
||||
follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1<redacted>/"`
|
||||
still leaked the first 4 chars. Lesson: when probing secret-bearing
|
||||
files, use field-by-field SELECTIVE inspection
|
||||
(`grep -E "^(KEY1|KEY2)="`) rather than negative filters; for any
|
||||
password line, `grep -c` (existence) or `test -n "$(...)"`
|
||||
(non-empty), never `cat` or value-printing.
|
||||
@@ -35,14 +35,15 @@ Primary AI inference host for PFI.
|
||||
| Stack | Port | Notes |
|
||||
|-------|------|-------|
|
||||
| llama-swap | 9292 | GGUF model server via llama.cpp |
|
||||
| vllm-embed (Qwen3) | 8001 | OpenAI-compatible embeddings; part of the `vllm-qwen3` stack (GPU 1) |
|
||||
| vllm-rerank (Qwen3) | 8002 | OpenAI-compatible reranker; part of the `vllm-qwen3` stack (GPU 1) |
|
||||
| vllm-embed (Qwen3) | 8001 | OpenAI-compatible embeddings; part of the `vllm` stack (GPU 1) |
|
||||
| vllm-rerank (Qwen3) | 8002 | OpenAI-compatible reranker; part of the `vllm` stack (GPU 1) |
|
||||
| vllm-reward (Skywork) | 8003 | Skywork-Reward-V2-8B-AWQ classifier; part of the `vllm` stack (GPU 1) |
|
||||
| dockge | 5001 | Docker stack management UI |
|
||||
| dozzle-agent | 7007 | Log agent; reports to the Dozzle hub on ana-docker |
|
||||
| beszel-agent | 45876 | Metrics agent; reports to the Beszel hub on ana-docker |
|
||||
|
||||
**Retired since last README update:**
|
||||
- `infinity` — replaced by `vllm-qwen3` after the upstream image stopped shipping a `transformers` build that knew Qwen3.
|
||||
- `infinity` — replaced by the `vllm` stack (originally `vllm-qwen3`, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a `transformers` build that knew Qwen3.
|
||||
- `LibreChat (+ rag_api, vectordb, mongodb, meilisearch)` — removed from this host.
|
||||
- `searxng` — now hosted on ana-docker for the whole fleet.
|
||||
- Residual networks (`librechat_default`, `kokoro-tts-gpu_default`) from prior experiments are still present; safe to `docker network rm` at leisure.
|
||||
@@ -60,6 +61,6 @@ Latest snapshot: `system-details.txt` (regenerate as needed).
|
||||
By default, no container is pinned. For predictable performance when multiple GPU workloads run concurrently:
|
||||
|
||||
- **GPU 0:** heavy LLM (llama-swap big models).
|
||||
- **GPU 1:** light services (both vllm-qwen3 services share this GPU via `--gpu-memory-utilization`).
|
||||
- **GPU 1:** light services (the three `vllm` services share this GPU via `--gpu-memory-utilization`).
|
||||
|
||||
Use `deploy.resources.reservations.devices[].device_ids: ["<id>"]` in compose to pin.
|
||||
|
||||
@@ -12,7 +12,7 @@ GGUF model server with on-demand model swapping. Served via llama.cpp's `llama-s
|
||||
- **`.env.example`** — template for the per-host `.env`. Copy to `.env` on the server and tweak.
|
||||
- **`config.yaml`** — model definitions and groups. Deployed to `/opt/docker/conf/llama-swap/config.yaml` on the server.
|
||||
|
||||
Homepage labels are in the compose file under the `AI Systems` group, matching the convention used by `vllm-qwen3` and `infinity`.
|
||||
Homepage labels are in the compose file under the `AI Systems` group, matching the convention used by `vllm` and `infinity`.
|
||||
|
||||
## Deploy a fresh install
|
||||
|
||||
|
||||
@@ -1,40 +0,0 @@
|
||||
# vllm-qwen3 stack tunables. Copy this to `.env` on the server before deploying.
|
||||
#
|
||||
# cp .env.example .env
|
||||
# # edit .env with real values
|
||||
# docker compose up -d
|
||||
|
||||
# Image version — pin for reproducibility (`latest` for edge)
|
||||
VLLM_VERSION=latest
|
||||
|
||||
# Host ports (container always listens on 8000 internally)
|
||||
EMBED_PORT=8001
|
||||
RERANK_PORT=8002
|
||||
|
||||
# GPU assignment — both services share this GPU
|
||||
# (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work)
|
||||
GPU_ID=1
|
||||
|
||||
# Models — reference by full repo name in API requests
|
||||
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
|
||||
RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B
|
||||
|
||||
# GPU memory split — fractions are of TOTAL GPU memory, not free memory.
|
||||
# When two vLLM services share a GPU, each profiler needs its own slice to
|
||||
# fit both the model and KV cache, so small values cause the second-to-start
|
||||
# service to OOM on KV cache allocation. 0.40 + 0.40 leaves ~20% headroom
|
||||
# and is comfortably above the minimum for two 0.6B Qwen3 models at 8k ctx.
|
||||
EMBED_GPU_MEM_UTIL=0.40
|
||||
RERANK_GPU_MEM_UTIL=0.40
|
||||
|
||||
# Context length caps — lower these if VRAM is tight.
|
||||
# Qwen3-Embedding supports up to 32k; reranker up to 32k.
|
||||
EMBED_MAX_MODEL_LEN=8192
|
||||
RERANK_MAX_MODEL_LEN=8192
|
||||
|
||||
# Optional API key — leave blank for no auth (fine on the internal network).
|
||||
# If set, both services require `Authorization: Bearer <key>`.
|
||||
API_KEY=
|
||||
|
||||
# HuggingFace token — only needed for gated models
|
||||
HF_TOKEN=
|
||||
@@ -1,80 +0,0 @@
|
||||
# vllm-qwen3
|
||||
|
||||
Qwen3 embedding + reranker served via vLLM. Replaces the unmaintained Infinity stack.
|
||||
|
||||
**Server:** ana-ml2
|
||||
**Ports:** `8001` (embed), `8002` (rerank) — both configurable via `.env`
|
||||
**GPU:** both services share GPU 1 by default (configurable)
|
||||
|
||||
## Why two services
|
||||
|
||||
vLLM runs **one model per process**, so embedding and reranking each get their own container. Both pin to the same GPU and split VRAM via `--gpu-memory-utilization`. Both use `--runner pooling` so the OpenAI server exposes `/v1/embeddings` (for the embedder) and `/rerank`, `/score` (for the reranker, which also needs the `--hf-overrides` described below).
|
||||
|
||||
## Reranker caveat
|
||||
|
||||
Qwen/Qwen3-Reranker-0.6B is a causal-LM checkpoint. The `--hf-overrides` flag in `compose.yaml` re-maps it to `Qwen3ForSequenceClassification` so vLLM's `/rerank` and `/score` endpoints work and the model emits only `no`/`yes` class logits instead of the full 151k-token distribution.
|
||||
|
||||
If that override breaks after a vLLM upgrade, the pre-converted checkpoint `tomaarsen/Qwen3-Reranker-0.6B-seq-cls` is a drop-in replacement that needs no overrides — set `RERANK_MODEL=tomaarsen/Qwen3-Reranker-0.6B-seq-cls` in `.env` and remove the `--hf-overrides` line from the compose.
|
||||
|
||||
## Deploy
|
||||
|
||||
```bash
|
||||
# On ana-ml2:
|
||||
sudo mkdir -p /opt/docker/compose/vllm-qwen3
|
||||
sudo chown $USER /opt/docker/compose/vllm-qwen3
|
||||
cd /opt/docker/compose/vllm-qwen3
|
||||
|
||||
# Copy compose.yaml + .env.example here (e.g. via scp from this workspace)
|
||||
cp .env.example .env
|
||||
# edit .env — pick GPU, ports, memory split, etc.
|
||||
|
||||
# Pre-download models (optional, speeds first boot)
|
||||
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^EMBED_MODEL .env | cut -d= -f2)"
|
||||
HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^RERANK_MODEL .env | cut -d= -f2)"
|
||||
|
||||
# Dry-parse
|
||||
docker compose config
|
||||
|
||||
# Launch
|
||||
docker compose up -d
|
||||
docker compose logs -f
|
||||
```
|
||||
|
||||
First boot compiles CUDA graphs and can take 2–3 minutes per service. The `start_period: 180s` healthcheck grace reflects that.
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
# Health
|
||||
curl -s http://localhost:8001/health
|
||||
curl -s http://localhost:8002/health
|
||||
|
||||
# Embedding (OpenAI-compatible)
|
||||
curl -s http://localhost:8001/v1/embeddings \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq .
|
||||
|
||||
# Reranker
|
||||
curl -s http://localhost:8002/rerank \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq .
|
||||
|
||||
# Listed models
|
||||
curl -s http://localhost:8001/v1/models | jq .
|
||||
curl -s http://localhost:8002/v1/models | jq .
|
||||
```
|
||||
|
||||
## Scaling knobs
|
||||
|
||||
- **`EMBED_GPU_MEM_UTIL` / `RERANK_GPU_MEM_UTIL`** — fractions of **total** GPU VRAM each service reserves (not of free VRAM). Both services profile independently, so each slice must be large enough to fit that service's model + KV cache with no knowledge of the other. Setting them too low causes the second-to-start container to OOM on KV cache allocation with `Available KV cache memory: -X.XX GiB`. Default 0.40/0.40 (= 0.80 total) leaves ~20% GPU headroom and works cleanly for the 0.6B pair at 8k context; raise for 4B/8B variants or drop `max-model-len` if you need more room.
|
||||
- **`EMBED_MAX_MODEL_LEN` / `RERANK_MAX_MODEL_LEN`** — lower to reduce KV-cache allocation if VRAM is tight. Qwen3 supports up to 32k natively.
|
||||
- **Larger models** — Qwen3-Embedding/Reranker come in 0.6B / 4B / 8B. Swap `EMBED_MODEL` / `RERANK_MODEL` and bump the memory fractions accordingly.
|
||||
- **Separate GPUs** — if contention hurts latency, split them: add a second `GPU_ID_RERANK` variable and point each service at its own device. (Requires a small compose edit; currently both share `${GPU_ID}`.)
|
||||
|
||||
## Migrating off Infinity
|
||||
|
||||
Once this stack is verified stable:
|
||||
|
||||
1. Stop the infinity stack (`docker compose down` under `/opt/docker/compose/infinity/`).
|
||||
2. Update consumers (AIPA agents, LibreChat RAG) to point at `:8001` for embeddings and `:8002` for rerank.
|
||||
3. Delete `stacks/infinity/` from this workspace.
|
||||
@@ -0,0 +1,54 @@
|
||||
# vllm stack tunables. Copy this to `.env` on the server before deploying.
|
||||
#
|
||||
# cp .env.example .env
|
||||
# # edit .env with real values
|
||||
# docker compose up -d
|
||||
|
||||
# Image version — pin for reproducibility (`latest` for edge)
|
||||
VLLM_VERSION=latest
|
||||
|
||||
# Host ports (container always listens on 8000 internally)
|
||||
EMBED_PORT=8001
|
||||
RERANK_PORT=8002
|
||||
REWARD_PORT=8003
|
||||
|
||||
# GPU assignment — all services share this GPU
|
||||
# (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work in llama-swap)
|
||||
GPU_ID=1
|
||||
|
||||
# Models — reference by full repo name in API requests
|
||||
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
|
||||
RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B
|
||||
# Skywork is a local-path AWQ output, not from HF Hub. Bind-mounted into the
|
||||
# reward container at /local-models — see compose.yaml. No env var here for
|
||||
# the model path itself since it's hard-coded in the compose command.
|
||||
|
||||
# GPU memory split — fractions are of TOTAL GPU memory, not free memory.
|
||||
# vLLM profiles each service independently, so each slice must be large enough
|
||||
# to fit that service's model + KV cache with no awareness of the others.
|
||||
# Setting any one too low causes that container to OOM on KV cache allocation
|
||||
# with `Available KV cache memory: -X.XX GiB`.
|
||||
#
|
||||
# Layout on a 48 GB Ada (~30% headroom kept free; matches current production):
|
||||
# EMBED 0.20 (~9.6 GB) — 0.6B Qwen3 embed at 8k ctx; comfortable
|
||||
# RERANK 0.20 (~9.6 GB) — 0.6B Qwen3 rerank at 8k ctx; comfortable
|
||||
# REWARD 0.30 (~14 GB) — 8B Skywork AWQ at 16k ctx classify; tune up if OOM
|
||||
EMBED_GPU_MEM_UTIL=0.20
|
||||
RERANK_GPU_MEM_UTIL=0.20
|
||||
REWARD_GPU_MEM_UTIL=0.30
|
||||
|
||||
# Context length caps — lower these if VRAM is tight.
|
||||
# Qwen3-Embedding supports up to 32k; reranker up to 32k.
|
||||
# Skywork capped at 16k server-side as defense-in-depth; JudgeClient also
|
||||
# enforces the cap at dispatch time per spec.
|
||||
EMBED_MAX_MODEL_LEN=8192
|
||||
RERANK_MAX_MODEL_LEN=8192
|
||||
REWARD_MAX_MODEL_LEN=16384
|
||||
|
||||
# Optional API key — leave blank for no auth (fine on the internal network).
|
||||
# If set, all three services require `Authorization: Bearer <key>`.
|
||||
API_KEY=
|
||||
|
||||
# HuggingFace token — only needed for gated models in the HF-Hub-loaded
|
||||
# services (embed/rerank). Reward is local-path, ignores this.
|
||||
HF_TOKEN=
|
||||
@@ -0,0 +1,133 @@
|
||||
# vllm
|
||||
|
||||
Multi-service vLLM stack on ana-ml2. Started as Qwen3 embedding + rerank
|
||||
(replacing the unmaintained Infinity stack); generalized to host any vLLM
|
||||
model on the box, currently three services:
|
||||
|
||||
- **`vllm-embed`** — Qwen3-Embedding 0.6B → OpenAI `/v1/embeddings`
|
||||
- **`vllm-rerank`** — Qwen3-Reranker 0.6B → `/rerank` + `/score`
|
||||
- **`vllm-reward`** — Skywork-Reward-V2-Llama-3.1-8B-AWQ → `/classify`
|
||||
|
||||
**Server:** ana-ml2
|
||||
**Ports:** `8001` (embed), `8002` (rerank), `8003` (reward) — all configurable via `.env`
|
||||
**GPU:** all three services share GPU 1 by default (configurable)
|
||||
|
||||
## Why one process per service
|
||||
|
||||
vLLM runs **one model per process**, so each model gets its own container.
|
||||
All three pin to the same GPU and split VRAM via `--gpu-memory-utilization`.
|
||||
Embed + rerank use `--runner pooling`; reward uses `--task classify` (newer
|
||||
vLLM flag for sequence-classification heads).
|
||||
|
||||
## Reranker caveat
|
||||
|
||||
Qwen/Qwen3-Reranker-0.6B is a causal-LM checkpoint. The `--hf-overrides`
|
||||
flag in `compose.yaml` re-maps it to `Qwen3ForSequenceClassification` so
|
||||
vLLM's `/rerank` and `/score` endpoints work and the model emits only
|
||||
`no`/`yes` class logits instead of the full 151k-token distribution.
|
||||
|
||||
If that override breaks after a vLLM upgrade, the pre-converted checkpoint
|
||||
`tomaarsen/Qwen3-Reranker-0.6B-seq-cls` is a drop-in replacement that needs
|
||||
no overrides — set `RERANK_MODEL=tomaarsen/Qwen3-Reranker-0.6B-seq-cls` in
|
||||
`.env` and remove the `--hf-overrides` line from the compose.
|
||||
|
||||
## Reward / Skywork specifics
|
||||
|
||||
`vllm-reward` serves `Skywork-Reward-V2-Llama-3.1-8B-AWQ` — an AWQ
|
||||
quantization produced locally (not pulled from HF). The quant output lives
|
||||
at `/tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ` on ana-ml2 and
|
||||
is bind-mounted read-only into the container at `/local-models`. vLLM
|
||||
loads it as a local-path HF-format model (config.json + safetensors).
|
||||
|
||||
`--max-model-len 16384` is a server-side cap; JudgeClient on the consumer
|
||||
side also enforces this at dispatch time. Defense-in-depth.
|
||||
|
||||
`--dtype auto` lets vLLM pick the right path for AWQ-quantized weights.
|
||||
|
||||
## Deploy
|
||||
|
||||
```bash
|
||||
# Sync canonical → ana-ml2
|
||||
scripts/deploy-stack.sh ana-ml2 vllm
|
||||
|
||||
# Pre-download Qwen3 models from HF (Skywork is local-path, no pull needed)
|
||||
scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
|
||||
--var hf_repo=Qwen/Qwen3-Embedding-0.6B
|
||||
scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
|
||||
--var hf_repo=Qwen/Qwen3-Reranker-0.6B
|
||||
|
||||
# Launch
|
||||
ssh ana-ml2 'cd /opt/docker/compose/vllm && docker compose up -d && docker compose logs --tail=30'
|
||||
```
|
||||
|
||||
First boot compiles CUDA graphs and can take 2–3 minutes per service
|
||||
(longer for the 8B reward — `start_period: 240s`).
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
# Health (each on its own port)
|
||||
curl -s http://localhost:8001/health
|
||||
curl -s http://localhost:8002/health
|
||||
curl -s http://localhost:8003/health
|
||||
|
||||
# Embedding (OpenAI-compatible)
|
||||
curl -s http://localhost:8001/v1/embeddings \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq .
|
||||
|
||||
# Reranker
|
||||
curl -s http://localhost:8002/rerank \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq .
|
||||
|
||||
# Reward classify (Skywork)
|
||||
curl -s http://localhost:8003/classify \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ","input":"User: hello\nAssistant: hi there"}' | jq .
|
||||
|
||||
# Listed models
|
||||
curl -s http://localhost:8001/v1/models | jq .
|
||||
curl -s http://localhost:8002/v1/models | jq .
|
||||
curl -s http://localhost:8003/v1/models | jq .
|
||||
```
|
||||
|
||||
## Scaling knobs
|
||||
|
||||
- **`EMBED_GPU_MEM_UTIL` / `RERANK_GPU_MEM_UTIL` / `REWARD_GPU_MEM_UTIL`** —
|
||||
fractions of **total** GPU VRAM each service reserves (not of free VRAM).
|
||||
Each profiler runs independently with no awareness of the others, so any
|
||||
slice that's too small to fit `model + KV cache` will OOM the
|
||||
second-to-start container with
|
||||
`Available KV cache memory: -X.XX GiB`. Default 0.20/0.20/0.30 totals
|
||||
0.70, leaving ~14 GB headroom on a 48 GB Ada (matches the production
|
||||
tune-down from the original 0.40/0.40 defaults — embed/rerank fit
|
||||
comfortably in 0.20 each). Bump REWARD up first if you see OOM — the
|
||||
8B AWQ model's KV slice at 16k ctx is the tightest. Drop embed/rerank
|
||||
further only if you've extended REWARD past 0.40 and still need room.
|
||||
|
||||
- **`EMBED_MAX_MODEL_LEN` / `RERANK_MAX_MODEL_LEN` / `REWARD_MAX_MODEL_LEN`** —
|
||||
lower to reduce KV-cache allocation if VRAM is tight. Qwen3 supports up
|
||||
to 32k natively; Skywork's reward cap of 16k is intentional (matches
|
||||
JudgeClient's dispatch-side cap).
|
||||
|
||||
- **Larger Qwen3 models** — embedding/reranker come in 0.6B / 4B / 8B.
|
||||
Swap `EMBED_MODEL` / `RERANK_MODEL` and bump the memory fractions
|
||||
accordingly.
|
||||
|
||||
- **Separate GPUs** — if contention hurts latency, split them. Today all
|
||||
three share `${GPU_ID}`. Adding `GPU_ID_REWARD`/etc. is a small compose
|
||||
edit.
|
||||
|
||||
## History
|
||||
|
||||
- **2026-05-13 rename + extend** — stack renamed from `vllm-qwen3` to
|
||||
`vllm` and gained the `vllm-reward` service (Skywork-Reward-V2 8B AWQ).
|
||||
No GPU memory rebalance needed in practice — production had already
|
||||
tuned EMBED/RERANK down from 0.40/0.40 to 0.20/0.20; adding REWARD at
|
||||
0.30 fits cleanly with ~14 GB headroom on the 48 GB Ada.
|
||||
|
||||
- **Migrated off Infinity** — Infinity stopped shipping a `transformers`
|
||||
build that knew Qwen3; this stack is the canonical replacement for both
|
||||
embed and rerank. Consumers (AIPA agents, LibreChat RAG) point at
|
||||
`:8001`/`:8002`.
|
||||
@@ -1,10 +1,16 @@
|
||||
# vLLM — Qwen3 Embedding + Reranker (one stack, two services).
|
||||
# vLLM — Qwen3 Embedding + Reranker + Skywork Reward-V2 classifier.
|
||||
#
|
||||
# Replaces the unmaintained Infinity stack. vLLM runs one model per process,
|
||||
# so this stack brings up two containers sharing a single GPU:
|
||||
# Originally created to replace the unmaintained Infinity stack (embed +
|
||||
# rerank); generalized 2026-05-13 to host any vLLM-served model on ana-ml2,
|
||||
# starting with the Skywork-Reward-V2-Llama-3.1-8B reward classifier
|
||||
# (AWQ-quantized locally, served from /tank/aimodels/llm/).
|
||||
#
|
||||
# vLLM runs one model per process, so this stack brings up three containers
|
||||
# sharing a single GPU:
|
||||
#
|
||||
# vllm-embed — Qwen3-Embedding served as an OpenAI /v1/embeddings server
|
||||
# vllm-rerank — Qwen3-Reranker served as a /rerank + /score server
|
||||
# vllm-reward — Skywork-Reward-V2-Llama-3.1-8B-AWQ served as a /classify scorer
|
||||
#
|
||||
# The reranker is a causal-LM checkpoint; --hf-overrides re-maps it to
|
||||
# Qwen3ForSequenceClassification so vLLM's reranking endpoints work and the
|
||||
@@ -13,8 +19,14 @@
|
||||
# All tunables live in .env — edit that, not this file.
|
||||
#
|
||||
# Pre-download models to avoid first-run delay:
|
||||
# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Embedding-0.6B
|
||||
# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Reranker-0.6B
|
||||
# scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
|
||||
# --var hf_repo=Qwen/Qwen3-Embedding-0.6B
|
||||
# scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \
|
||||
# --var hf_repo=Qwen/Qwen3-Reranker-0.6B
|
||||
#
|
||||
# Skywork-Reward-V2-Llama-3.1-8B-AWQ is a locally-quantized model — lives at
|
||||
# /tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ on ana-ml2 and is
|
||||
# bind-mounted into the reward service at /local-models. Not from HF Hub.
|
||||
|
||||
services:
|
||||
vllm-embed:
|
||||
@@ -127,6 +139,64 @@ services:
|
||||
- homepage.description=Qwen3 Reranker via vLLM (ana-ml2)
|
||||
- homepage.href=http://10.250.50.54:${RERANK_PORT}/docs
|
||||
|
||||
vllm-reward:
|
||||
image: vllm/vllm-openai:${VLLM_VERSION}
|
||||
container_name: vllm-reward
|
||||
restart: unless-stopped
|
||||
ipc: host
|
||||
ports:
|
||||
- "${REWARD_PORT}:8000"
|
||||
volumes:
|
||||
# AWQ output lives in the legacy llama-swap models tree, not the HF cache
|
||||
# — bind-mount the LLM models dir read-only so the reward service can
|
||||
# load it as a local-path HF-format model.
|
||||
- /tank/aimodels/llm:/local-models:ro
|
||||
environment:
|
||||
- VLLM_API_KEY=${API_KEY:-}
|
||||
command:
|
||||
- /local-models/Skywork-Reward-V2-Llama-3.1-8B-AWQ
|
||||
- --served-model-name
|
||||
- Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
|
||||
# vLLM 0.19.1 deprecated --task in favor of --runner. The model's
|
||||
# config.json declares `LlamaForSequenceClassification` so the
|
||||
# pooling runner uses it as a classifier (single-label reward score)
|
||||
# without needing an explicit task flag.
|
||||
- --runner
|
||||
- pooling
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- "8000"
|
||||
- --gpu-memory-utilization
|
||||
- ${REWARD_GPU_MEM_UTIL}
|
||||
- --max-model-len
|
||||
- ${REWARD_MAX_MODEL_LEN}
|
||||
- --dtype
|
||||
- auto
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids:
|
||||
- "${GPU_ID}"
|
||||
capabilities:
|
||||
- gpu
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 240s
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI Systems
|
||||
- homepage.name=vLLM Reward (Skywork)
|
||||
- homepage.icon=mdi-scale-balance
|
||||
- homepage.description=Skywork-Reward-V2 8B classifier via vLLM (ana-ml2)
|
||||
- homepage.href=http://10.250.50.54:${REWARD_PORT}/docs
|
||||
|
||||
networks:
|
||||
tnet:
|
||||
name: traefik-net
|
||||
Reference in New Issue
Block a user