From 7e7130172e81c8d96ad8b8af1b6cf8cfad729cfc Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 13 May 2026 22:00:26 -0700 Subject: [PATCH] =?UTF-8?q?vllm:=20rename=20stack=20from=20vllm-qwen3=20?= =?UTF-8?q?=E2=86=92=20vllm=20+=20add=20Skywork=20reward=20classifier?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two related changes shipped together. The stack rename is independent but adding `vllm-reward` to the existing `vllm-qwen3` would have made that name actively misleading. **Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo references (README.md root, servers/ana-ml2/, stacks/llama-swap/, configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two intentional history mentions retained (servers/ana-ml2 + stacks/vllm README). **Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ on port 8003. The AWQ output is a locally-quantized model (not from HF), so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the shared HF cache. Model config.json declares LlamaForSequenceClassification which vLLM's pooling runner picks up automatically — produces a single reward score per input via /classify. **Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1 deprecated --task in favor of --runner pooling (model architecture in config.json drives the classification head). Compose uses --runner pooling with a comment explaining the substitution. **GPU memory:** no rebalance needed — production had already tuned EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB headroom on the 48 GB Ada. **Server-side:** brought existing vllm-qwen3 down, mv'd /opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed new compose via scripts/deploy-stack.sh, brought all 3 services up. **Smoke tests:** - /health on 8001/8002/8003 → 200 - /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ with max_model_len 16384 - /classify with a sample conversation → returns LABEL_0 with prob 0.9999 (single-output regression-style reward score, expected shape for a reward model) --- README.md | 4 +- configs/restic/ana-ml2/README.md | 2 +- configs/restic/ana-ml2/profiles.yaml | 2 +- docs/runbooks/disaster-recovery.md | 2 +- persistent-memory.md | 155 +++++++++++++++++++++++ servers/ana-ml2/README.md | 9 +- stacks/llama-swap/README.md | 2 +- stacks/vllm-qwen3/.env.example | 40 ------ stacks/vllm-qwen3/README.md | 80 ------------ stacks/vllm/.env.example | 54 ++++++++ stacks/vllm/README.md | 133 +++++++++++++++++++ stacks/{vllm-qwen3 => vllm}/compose.yaml | 80 +++++++++++- 12 files changed, 428 insertions(+), 135 deletions(-) create mode 100644 persistent-memory.md delete mode 100644 stacks/vllm-qwen3/.env.example delete mode 100644 stacks/vllm-qwen3/README.md create mode 100644 stacks/vllm/.env.example create mode 100644 stacks/vllm/README.md rename stacks/{vllm-qwen3 => vllm}/compose.yaml (55%) diff --git a/README.md b/README.md index eb4e5ba..4f40c93 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,7 @@ Per-host snapshots of the running system live under `servers//system-detai **GPU (ana-ml2):** - `llama-swap` — GGUF model swapper via llama.cpp (port 9292) -- `vllm-qwen3` — embeddings (8001) + reranker (8002) via vLLM +- `vllm` — embeddings (8001) + reranker (8002) + Skywork reward classifier (8003) via vLLM **Anaheim non-GPU (ana-docker):** - `traefik`, `crowdsec`, `gitea`, `vaultwarden`, `synapse`, `seafile`, `searxng`, `openwebui`, `sillytavern`, `mailrise`, `rustdesk`, `dockge`, `it-tools` @@ -77,7 +77,7 @@ Per-host snapshots of the running system live under `servers//system-detai - Backup target: `rest-server-ana` on port 8000 **GPU ana-ml2 (non-canonical for now):** -- `comfyui`, `kokoro`, `parakeet`, `vibevoice` alongside the canonical `llama-swap` + `vllm-qwen3` +- `comfyui`, `kokoro`, `parakeet`, `vibevoice` alongside the canonical `llama-swap` + `vllm` **NH3 (nh3-docker):** - `adguard`, `dockge`, plus Beszel/Dozzle agents diff --git a/configs/restic/ana-ml2/README.md b/configs/restic/ana-ml2/README.md index 5cbe6a7..8f8f228 100644 --- a/configs/restic/ana-ml2/README.md +++ b/configs/restic/ana-ml2/README.md @@ -29,7 +29,7 @@ pre-backup DB hook is required. Contrast with `configs/restic/ana-docker/` where synapse/seafile/vaultwarden DB dumps run first. - `llama-swap` — GGUF swapper (llama.cpp) -- `vllm-qwen3` — embedding + rerank +- `vllm` — embedding + rerank + reward classifier - `comfyui`, `kokoro`, `parakeet`, `vibevoice` - `beszel-agent-ana`, `dozzle-agent-ana`, `dockge` diff --git a/configs/restic/ana-ml2/profiles.yaml b/configs/restic/ana-ml2/profiles.yaml index ca12dcb..a30284c 100644 --- a/configs/restic/ana-ml2/profiles.yaml +++ b/configs/restic/ana-ml2/profiles.yaml @@ -12,7 +12,7 @@ # caches, llama.cpp GGUFs, ComfyUI models, etc.) — all regenerable # from upstream. Backing them up would blow the repo size budget. # - No DB dumps needed. None of the stacks on this host (llama-swap, -# vllm-qwen3, comfyui, kokoro, parakeet, vibevoice, beszel-agent, +# vllm, comfyui, kokoro, parakeet, vibevoice, beszel-agent, # dozzle-agent, dockge) store relational data. version: "1" diff --git a/docs/runbooks/disaster-recovery.md b/docs/runbooks/disaster-recovery.md index 1838e4b..372912e 100644 --- a/docs/runbooks/disaster-recovery.md +++ b/docs/runbooks/disaster-recovery.md @@ -209,7 +209,7 @@ Essentially **all Anaheim primary services** go offline. Because ana-nas lives h ### ana-ml2 (bare metal Supermicro, 10.250.50.54, BMC 10.250.250.50) -**Blast radius:** AI inference services (llama-swap, vllm-qwen3). Consumer-facing chat/embedding endpoints fail. +**Blast radius:** AI inference services (llama-swap, vllm). Consumer-facing chat/embedding/reward-scoring endpoints fail. **Recovery:** 1. Check OS via SSH. If unresponsive, BMC console at . diff --git a/persistent-memory.md b/persistent-memory.md new file mode 100644 index 0000000..cbf7696 --- /dev/null +++ b/persistent-memory.md @@ -0,0 +1,155 @@ +# Persistent memory — eshpfi-management + +_Last updated: 2026-05-08_ + +## Repo purpose + +Reference workspace for PFI infrastructure: server inventory, canonical +Docker Compose stacks, ops playbooks, and conventions. Authoritative +copies of compose files live on the servers under +`/opt/docker/compose//`; this repo mirrors them for version +control, editing, planning, and CI-driven deploys. + +## Tools and conventions + +Sister repos (separate gitea repos, deployed by playbooks here): + +| Repo | Role | CI status | +|---|---|---| +| `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) | +| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) | +| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) | + +- **Two-layer backups** — Backrest orchestrates restic for file+DB (5 + fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for + VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + + cross-site restic targets — see `docs/runbooks/disaster-recovery.md` + for the blast-radius matrix. + +## Current state / in-flight + +_As of 2026-05-08:_ + +- **CI/CD pipe is live for three repos** (task-board, vor, nevermore). + Pattern: workflow checks out triggering repo + `vh/esh-pfi-infrastructure` + (for elway), SSHes to ana-docker, runs an elway playbook pinned to the + triggering commit SHA. User-scope secrets `DEPLOY_SSH_KEY` and + `MGMT_REPO_TOKEN` cover all repos under `vh/`. +- **Open**: rotate `MINIFLUX_PASSWORD` — leaked in a prior session's + transcript while seeding nevermore's `.env` (full leak once, partial + leak once). +- **Open**: clean up old `news-digest` detritus on ana-docker — + `/opt/docker/compose/news-digest/`, `/opt/docker/data/news-digest/`, + image `local/news-digest:v5`. Migrate the historical `edition-*.html` + archive files into `/opt/docker/data/nevermore/` first. +- **Watch**: nh3-nas `/volume1` at 65%; plan retention review or + capacity expansion before ~80%. +- **Possible next**: aggregator service + homepage widget for live + "fleet backup health" dashboard. Discussed but not committed; user + weighing against a `/schedule` weekly digest as the lighter + alternative. +- **Open mystery**: `docker push` from outside the LAN to + `gitea.phasefinal.com` has a **~60s client-side per-PATCH ceiling** + for chunked blob uploads. We confirmed the timer is client-side + (Traefik logged `499 / 60012ms` — client closed; Gitea logged the + same as `unexpected EOF`). Where exactly the 60s lives in the docker + daemon / containerd stack — and whether it's tunable — is + unidentified. Worked around for now by shrinking images below the + cliff; if it bites again, instrument with `dockerd -D` + strace on a + fresh push to find the actual timer. + +## Recent decisions + +- `[2026-05-08]` Pin `torch` to CPU-only via + `--index-url https://download.pytorch.org/whl/cpu` in + `Worldtree/Dockerfile` before the rest of `requirements.txt`. + `sentence-transformers` transitively pulls torch and grabs the CUDA + flavor by default (~2.5GB of nvidia libs). Worldtree runs on + `ana-docker` (no GPU reservation in `compose.yaml`) so CUDA is dead + weight. Image: 5.62GB → 1.6GB. The failing push from off-LAN now + succeeds. +- `[2026-05-08]` Split `Worldtree/requirements.txt` into runtime + + `requirements-dev.txt` (pytest, pytest-asyncio, ruff, pylint). + ~50MB out of the runtime image; smaller attack surface. +- `[2026-05-01]` nevermore **replaces** news-digest at port 8181 (not + coexist on 8182). User chose one stack at a time over soak window. + ~30s briefing gap accepted as the cost. +- `[2026-04-30]` Bumped nevermore deploy playbook's first-render wait + from 60s to 6 min — first deploy with empty `/output` requires a + worker LLM cycle (2-4 min) before `index.html` exists. +- `[2026-04-29]` Workflow drops explicit `container:` directive after + runner re-registration with `:docker://node:20-bookworm-slim` schema + labels — single source of truth for the build environment. +- `[2026-04-29]` Deploy playbooks accept SHA refs in addition to branch + names (`git rev-parse ` then `origin/^{commit}` fallback). + CI passes `--var ref=${{ github.sha }}`; manual runs pass `main` / + `v0.1.0`. Same code path either way. +- `[2026-04-29]` Single central Gitea Actions runner on ana-docker + (`f014d55`). Parameterized `playbooks/deploy-gitea-runner.yaml` so + site-local runners (nh3, esh) drop in via `--var` overrides — not + copy-pasted playbooks. +- `[2026-04-29]` `nevermore` carries forward `hidden.json` and + historical `MINIFLUX_PASSWORD` from `news-digest` during cutover + (preserves user state across stack rename). + +## Tried and abandoned + +- `[2026-05-08]` Filtering Traefik's UTC access log by Gitea-local-PDT + timestamp substrings (`grep "2026/05/08 15:1[2-7]"`) returned zero + matches and led to a wrong "no /v2/ traffic in 12 days" conclusion. + **Gitea logs in PDT, Traefik logs in UTC** — same host, different + timezones. Always normalize timezones (UTC) when correlating logs + across services on the same box. Cost: ~30 min in the wrong + direction. +- `[2026-05-08]` Validating a user-proposed Traefik/Gitea timeout bump + for "Traefik is dropping connections on big docker pushes" without + first verifying which component was actually in the failure path. + Traefik turned out to be innocent (`499 / 60012ms` = client closed, + Traefik never timed out), Gitea's `PER_WRITE_TIMEOUT` governs + response writes (wrong direction), and the real ceiling was a ~60s + client-side timer no compose change can reach. **Lesson: validate + the diagnostic premise — which component is actually in the failure + path? — before refining the proposed fix.** +- `[2026-05-08]` Bumping Gitea `PER_WRITE_TIMEOUT` / + `PER_WRITE_PER_KB_TIMEOUT` to address `unexpected EOF` on + `/v2/.../blobs/uploads/` PATCH — wrong direction. Both govern + **response writes**, not request body reads. `unexpected EOF` from + Go's HTTP server means the client closed mid-body-upload; not a + knob Gitea exposes server-side. +- `[2026-04-30]` task-board workflow with + `container: image: debian:bookworm-slim` — fails: + `actions/checkout@v4` needs `node` at runtime, slim image lacks it. + Switched to `node:20-bookworm-slim` (has node + apt) or runner-label + default. +- `[2026-04-30]` Dropping the `container:` directive before runner + re-registration with docker-schema labels — runner silently falls + back to **host mode** (jobs run inside the alpine `act_runner` + container itself, no apt). The `:host` suffix in startup logs + (`labels updated to: [pfi-fleet:host ana-docker:host]`) is the + giveaway. Fix: register with `pfi-fleet:docker://` schema + labels. +- `[2026-04-30]` Updating runner labels by editing `.env` and bouncing + — doesn't take. The `.runner` registration cache pins labels at + first registration; env-var updates are read each start but the + stored token + UUID are tied to the original label set on the gitea + side. Fix: stop runner, delete `.runner`, generate new admin + registration token, redeploy. +- `[2026-04-30]` `git reset --hard origin/` in + `deploy-task-board.yaml` (and the in-repo nevermore playbook before + fix) — invalid syntax: `origin/` prefix only works for branch refs. + SHAs need `git reset --hard ` directly. Resolved with + `git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"` first, + then bare `"{{ ref }}^{commit}"` fallback. +- `[2026-04-30]` Assuming `DEPLOY_SSH_KEY` was at user scope after + task-board wiring — it was actually only repo-scope on + `vh/task-board`. vor's first CI run failed with empty SSH key + (`printf '%s\n' "" > ~/.ssh/id_ed25519`). Fix: copy secret to user + scope at `gitea.phasefinal.com/user/settings/actions/secrets`. +- `[2026-04-30]` `grep -vE "^(#|$)"` to inspect `.env` for sanity — + leaked the full `MINIFLUX_PASSWORD` line into the transcript. Then a + follow-up redaction attempt with `sed -E "s/=(.{4}).*$/=\1/"` + still leaked the first 4 chars. Lesson: when probing secret-bearing + files, use field-by-field SELECTIVE inspection + (`grep -E "^(KEY1|KEY2)="`) rather than negative filters; for any + password line, `grep -c` (existence) or `test -n "$(...)"` + (non-empty), never `cat` or value-printing. diff --git a/servers/ana-ml2/README.md b/servers/ana-ml2/README.md index 1609ac0..2aa1b1b 100644 --- a/servers/ana-ml2/README.md +++ b/servers/ana-ml2/README.md @@ -35,14 +35,15 @@ Primary AI inference host for PFI. | Stack | Port | Notes | |-------|------|-------| | llama-swap | 9292 | GGUF model server via llama.cpp | -| vllm-embed (Qwen3) | 8001 | OpenAI-compatible embeddings; part of the `vllm-qwen3` stack (GPU 1) | -| vllm-rerank (Qwen3) | 8002 | OpenAI-compatible reranker; part of the `vllm-qwen3` stack (GPU 1) | +| vllm-embed (Qwen3) | 8001 | OpenAI-compatible embeddings; part of the `vllm` stack (GPU 1) | +| vllm-rerank (Qwen3) | 8002 | OpenAI-compatible reranker; part of the `vllm` stack (GPU 1) | +| vllm-reward (Skywork) | 8003 | Skywork-Reward-V2-8B-AWQ classifier; part of the `vllm` stack (GPU 1) | | dockge | 5001 | Docker stack management UI | | dozzle-agent | 7007 | Log agent; reports to the Dozzle hub on ana-docker | | beszel-agent | 45876 | Metrics agent; reports to the Beszel hub on ana-docker | **Retired since last README update:** -- `infinity` — replaced by `vllm-qwen3` after the upstream image stopped shipping a `transformers` build that knew Qwen3. +- `infinity` — replaced by the `vllm` stack (originally `vllm-qwen3`, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a `transformers` build that knew Qwen3. - `LibreChat (+ rag_api, vectordb, mongodb, meilisearch)` — removed from this host. - `searxng` — now hosted on ana-docker for the whole fleet. - Residual networks (`librechat_default`, `kokoro-tts-gpu_default`) from prior experiments are still present; safe to `docker network rm` at leisure. @@ -60,6 +61,6 @@ Latest snapshot: `system-details.txt` (regenerate as needed). By default, no container is pinned. For predictable performance when multiple GPU workloads run concurrently: - **GPU 0:** heavy LLM (llama-swap big models). -- **GPU 1:** light services (both vllm-qwen3 services share this GPU via `--gpu-memory-utilization`). +- **GPU 1:** light services (the three `vllm` services share this GPU via `--gpu-memory-utilization`). Use `deploy.resources.reservations.devices[].device_ids: [""]` in compose to pin. diff --git a/stacks/llama-swap/README.md b/stacks/llama-swap/README.md index c04a7e6..200b6f0 100644 --- a/stacks/llama-swap/README.md +++ b/stacks/llama-swap/README.md @@ -12,7 +12,7 @@ GGUF model server with on-demand model swapping. Served via llama.cpp's `llama-s - **`.env.example`** — template for the per-host `.env`. Copy to `.env` on the server and tweak. - **`config.yaml`** — model definitions and groups. Deployed to `/opt/docker/conf/llama-swap/config.yaml` on the server. -Homepage labels are in the compose file under the `AI Systems` group, matching the convention used by `vllm-qwen3` and `infinity`. +Homepage labels are in the compose file under the `AI Systems` group, matching the convention used by `vllm` and `infinity`. ## Deploy a fresh install diff --git a/stacks/vllm-qwen3/.env.example b/stacks/vllm-qwen3/.env.example deleted file mode 100644 index 2a655c7..0000000 --- a/stacks/vllm-qwen3/.env.example +++ /dev/null @@ -1,40 +0,0 @@ -# vllm-qwen3 stack tunables. Copy this to `.env` on the server before deploying. -# -# cp .env.example .env -# # edit .env with real values -# docker compose up -d - -# Image version — pin for reproducibility (`latest` for edge) -VLLM_VERSION=latest - -# Host ports (container always listens on 8000 internally) -EMBED_PORT=8001 -RERANK_PORT=8002 - -# GPU assignment — both services share this GPU -# (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work) -GPU_ID=1 - -# Models — reference by full repo name in API requests -EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B -RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B - -# GPU memory split — fractions are of TOTAL GPU memory, not free memory. -# When two vLLM services share a GPU, each profiler needs its own slice to -# fit both the model and KV cache, so small values cause the second-to-start -# service to OOM on KV cache allocation. 0.40 + 0.40 leaves ~20% headroom -# and is comfortably above the minimum for two 0.6B Qwen3 models at 8k ctx. -EMBED_GPU_MEM_UTIL=0.40 -RERANK_GPU_MEM_UTIL=0.40 - -# Context length caps — lower these if VRAM is tight. -# Qwen3-Embedding supports up to 32k; reranker up to 32k. -EMBED_MAX_MODEL_LEN=8192 -RERANK_MAX_MODEL_LEN=8192 - -# Optional API key — leave blank for no auth (fine on the internal network). -# If set, both services require `Authorization: Bearer `. -API_KEY= - -# HuggingFace token — only needed for gated models -HF_TOKEN= diff --git a/stacks/vllm-qwen3/README.md b/stacks/vllm-qwen3/README.md deleted file mode 100644 index f53fb42..0000000 --- a/stacks/vllm-qwen3/README.md +++ /dev/null @@ -1,80 +0,0 @@ -# vllm-qwen3 - -Qwen3 embedding + reranker served via vLLM. Replaces the unmaintained Infinity stack. - -**Server:** ana-ml2 -**Ports:** `8001` (embed), `8002` (rerank) — both configurable via `.env` -**GPU:** both services share GPU 1 by default (configurable) - -## Why two services - -vLLM runs **one model per process**, so embedding and reranking each get their own container. Both pin to the same GPU and split VRAM via `--gpu-memory-utilization`. Both use `--runner pooling` so the OpenAI server exposes `/v1/embeddings` (for the embedder) and `/rerank`, `/score` (for the reranker, which also needs the `--hf-overrides` described below). - -## Reranker caveat - -Qwen/Qwen3-Reranker-0.6B is a causal-LM checkpoint. The `--hf-overrides` flag in `compose.yaml` re-maps it to `Qwen3ForSequenceClassification` so vLLM's `/rerank` and `/score` endpoints work and the model emits only `no`/`yes` class logits instead of the full 151k-token distribution. - -If that override breaks after a vLLM upgrade, the pre-converted checkpoint `tomaarsen/Qwen3-Reranker-0.6B-seq-cls` is a drop-in replacement that needs no overrides — set `RERANK_MODEL=tomaarsen/Qwen3-Reranker-0.6B-seq-cls` in `.env` and remove the `--hf-overrides` line from the compose. - -## Deploy - -```bash -# On ana-ml2: -sudo mkdir -p /opt/docker/compose/vllm-qwen3 -sudo chown $USER /opt/docker/compose/vllm-qwen3 -cd /opt/docker/compose/vllm-qwen3 - -# Copy compose.yaml + .env.example here (e.g. via scp from this workspace) -cp .env.example .env -# edit .env — pick GPU, ports, memory split, etc. - -# Pre-download models (optional, speeds first boot) -HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^EMBED_MODEL .env | cut -d= -f2)" -HF_HOME=/tank/aimodels/huggingface hf download "$(grep ^RERANK_MODEL .env | cut -d= -f2)" - -# Dry-parse -docker compose config - -# Launch -docker compose up -d -docker compose logs -f -``` - -First boot compiles CUDA graphs and can take 2–3 minutes per service. The `start_period: 180s` healthcheck grace reflects that. - -## Verify - -```bash -# Health -curl -s http://localhost:8001/health -curl -s http://localhost:8002/health - -# Embedding (OpenAI-compatible) -curl -s http://localhost:8001/v1/embeddings \ - -H "Content-Type: application/json" \ - -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq . - -# Reranker -curl -s http://localhost:8002/rerank \ - -H "Content-Type: application/json" \ - -d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq . - -# Listed models -curl -s http://localhost:8001/v1/models | jq . -curl -s http://localhost:8002/v1/models | jq . -``` - -## Scaling knobs - -- **`EMBED_GPU_MEM_UTIL` / `RERANK_GPU_MEM_UTIL`** — fractions of **total** GPU VRAM each service reserves (not of free VRAM). Both services profile independently, so each slice must be large enough to fit that service's model + KV cache with no knowledge of the other. Setting them too low causes the second-to-start container to OOM on KV cache allocation with `Available KV cache memory: -X.XX GiB`. Default 0.40/0.40 (= 0.80 total) leaves ~20% GPU headroom and works cleanly for the 0.6B pair at 8k context; raise for 4B/8B variants or drop `max-model-len` if you need more room. -- **`EMBED_MAX_MODEL_LEN` / `RERANK_MAX_MODEL_LEN`** — lower to reduce KV-cache allocation if VRAM is tight. Qwen3 supports up to 32k natively. -- **Larger models** — Qwen3-Embedding/Reranker come in 0.6B / 4B / 8B. Swap `EMBED_MODEL` / `RERANK_MODEL` and bump the memory fractions accordingly. -- **Separate GPUs** — if contention hurts latency, split them: add a second `GPU_ID_RERANK` variable and point each service at its own device. (Requires a small compose edit; currently both share `${GPU_ID}`.) - -## Migrating off Infinity - -Once this stack is verified stable: - -1. Stop the infinity stack (`docker compose down` under `/opt/docker/compose/infinity/`). -2. Update consumers (AIPA agents, LibreChat RAG) to point at `:8001` for embeddings and `:8002` for rerank. -3. Delete `stacks/infinity/` from this workspace. diff --git a/stacks/vllm/.env.example b/stacks/vllm/.env.example new file mode 100644 index 0000000..016d7dc --- /dev/null +++ b/stacks/vllm/.env.example @@ -0,0 +1,54 @@ +# vllm stack tunables. Copy this to `.env` on the server before deploying. +# +# cp .env.example .env +# # edit .env with real values +# docker compose up -d + +# Image version — pin for reproducibility (`latest` for edge) +VLLM_VERSION=latest + +# Host ports (container always listens on 8000 internally) +EMBED_PORT=8001 +RERANK_PORT=8002 +REWARD_PORT=8003 + +# GPU assignment — all services share this GPU +# (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work in llama-swap) +GPU_ID=1 + +# Models — reference by full repo name in API requests +EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B +RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B +# Skywork is a local-path AWQ output, not from HF Hub. Bind-mounted into the +# reward container at /local-models — see compose.yaml. No env var here for +# the model path itself since it's hard-coded in the compose command. + +# GPU memory split — fractions are of TOTAL GPU memory, not free memory. +# vLLM profiles each service independently, so each slice must be large enough +# to fit that service's model + KV cache with no awareness of the others. +# Setting any one too low causes that container to OOM on KV cache allocation +# with `Available KV cache memory: -X.XX GiB`. +# +# Layout on a 48 GB Ada (~30% headroom kept free; matches current production): +# EMBED 0.20 (~9.6 GB) — 0.6B Qwen3 embed at 8k ctx; comfortable +# RERANK 0.20 (~9.6 GB) — 0.6B Qwen3 rerank at 8k ctx; comfortable +# REWARD 0.30 (~14 GB) — 8B Skywork AWQ at 16k ctx classify; tune up if OOM +EMBED_GPU_MEM_UTIL=0.20 +RERANK_GPU_MEM_UTIL=0.20 +REWARD_GPU_MEM_UTIL=0.30 + +# Context length caps — lower these if VRAM is tight. +# Qwen3-Embedding supports up to 32k; reranker up to 32k. +# Skywork capped at 16k server-side as defense-in-depth; JudgeClient also +# enforces the cap at dispatch time per spec. +EMBED_MAX_MODEL_LEN=8192 +RERANK_MAX_MODEL_LEN=8192 +REWARD_MAX_MODEL_LEN=16384 + +# Optional API key — leave blank for no auth (fine on the internal network). +# If set, all three services require `Authorization: Bearer `. +API_KEY= + +# HuggingFace token — only needed for gated models in the HF-Hub-loaded +# services (embed/rerank). Reward is local-path, ignores this. +HF_TOKEN= diff --git a/stacks/vllm/README.md b/stacks/vllm/README.md new file mode 100644 index 0000000..3e0111c --- /dev/null +++ b/stacks/vllm/README.md @@ -0,0 +1,133 @@ +# vllm + +Multi-service vLLM stack on ana-ml2. Started as Qwen3 embedding + rerank +(replacing the unmaintained Infinity stack); generalized to host any vLLM +model on the box, currently three services: + +- **`vllm-embed`** — Qwen3-Embedding 0.6B → OpenAI `/v1/embeddings` +- **`vllm-rerank`** — Qwen3-Reranker 0.6B → `/rerank` + `/score` +- **`vllm-reward`** — Skywork-Reward-V2-Llama-3.1-8B-AWQ → `/classify` + +**Server:** ana-ml2 +**Ports:** `8001` (embed), `8002` (rerank), `8003` (reward) — all configurable via `.env` +**GPU:** all three services share GPU 1 by default (configurable) + +## Why one process per service + +vLLM runs **one model per process**, so each model gets its own container. +All three pin to the same GPU and split VRAM via `--gpu-memory-utilization`. +Embed + rerank use `--runner pooling`; reward uses `--task classify` (newer +vLLM flag for sequence-classification heads). + +## Reranker caveat + +Qwen/Qwen3-Reranker-0.6B is a causal-LM checkpoint. The `--hf-overrides` +flag in `compose.yaml` re-maps it to `Qwen3ForSequenceClassification` so +vLLM's `/rerank` and `/score` endpoints work and the model emits only +`no`/`yes` class logits instead of the full 151k-token distribution. + +If that override breaks after a vLLM upgrade, the pre-converted checkpoint +`tomaarsen/Qwen3-Reranker-0.6B-seq-cls` is a drop-in replacement that needs +no overrides — set `RERANK_MODEL=tomaarsen/Qwen3-Reranker-0.6B-seq-cls` in +`.env` and remove the `--hf-overrides` line from the compose. + +## Reward / Skywork specifics + +`vllm-reward` serves `Skywork-Reward-V2-Llama-3.1-8B-AWQ` — an AWQ +quantization produced locally (not pulled from HF). The quant output lives +at `/tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ` on ana-ml2 and +is bind-mounted read-only into the container at `/local-models`. vLLM +loads it as a local-path HF-format model (config.json + safetensors). + +`--max-model-len 16384` is a server-side cap; JudgeClient on the consumer +side also enforces this at dispatch time. Defense-in-depth. + +`--dtype auto` lets vLLM pick the right path for AWQ-quantized weights. + +## Deploy + +```bash +# Sync canonical → ana-ml2 +scripts/deploy-stack.sh ana-ml2 vllm + +# Pre-download Qwen3 models from HF (Skywork is local-path, no pull needed) +scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \ + --var hf_repo=Qwen/Qwen3-Embedding-0.6B +scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \ + --var hf_repo=Qwen/Qwen3-Reranker-0.6B + +# Launch +ssh ana-ml2 'cd /opt/docker/compose/vllm && docker compose up -d && docker compose logs --tail=30' +``` + +First boot compiles CUDA graphs and can take 2–3 minutes per service +(longer for the 8B reward — `start_period: 240s`). + +## Verify + +```bash +# Health (each on its own port) +curl -s http://localhost:8001/health +curl -s http://localhost:8002/health +curl -s http://localhost:8003/health + +# Embedding (OpenAI-compatible) +curl -s http://localhost:8001/v1/embeddings \ + -H "Content-Type: application/json" \ + -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":["hello world"]}' | jq . + +# Reranker +curl -s http://localhost:8002/rerank \ + -H "Content-Type: application/json" \ + -d '{"model":"Qwen/Qwen3-Reranker-0.6B","query":"what is a cat","documents":["cats are mammals","dogs bark"]}' | jq . + +# Reward classify (Skywork) +curl -s http://localhost:8003/classify \ + -H "Content-Type: application/json" \ + -d '{"model":"Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ","input":"User: hello\nAssistant: hi there"}' | jq . + +# Listed models +curl -s http://localhost:8001/v1/models | jq . +curl -s http://localhost:8002/v1/models | jq . +curl -s http://localhost:8003/v1/models | jq . +``` + +## Scaling knobs + +- **`EMBED_GPU_MEM_UTIL` / `RERANK_GPU_MEM_UTIL` / `REWARD_GPU_MEM_UTIL`** — + fractions of **total** GPU VRAM each service reserves (not of free VRAM). + Each profiler runs independently with no awareness of the others, so any + slice that's too small to fit `model + KV cache` will OOM the + second-to-start container with + `Available KV cache memory: -X.XX GiB`. Default 0.20/0.20/0.30 totals + 0.70, leaving ~14 GB headroom on a 48 GB Ada (matches the production + tune-down from the original 0.40/0.40 defaults — embed/rerank fit + comfortably in 0.20 each). Bump REWARD up first if you see OOM — the + 8B AWQ model's KV slice at 16k ctx is the tightest. Drop embed/rerank + further only if you've extended REWARD past 0.40 and still need room. + +- **`EMBED_MAX_MODEL_LEN` / `RERANK_MAX_MODEL_LEN` / `REWARD_MAX_MODEL_LEN`** — + lower to reduce KV-cache allocation if VRAM is tight. Qwen3 supports up + to 32k natively; Skywork's reward cap of 16k is intentional (matches + JudgeClient's dispatch-side cap). + +- **Larger Qwen3 models** — embedding/reranker come in 0.6B / 4B / 8B. + Swap `EMBED_MODEL` / `RERANK_MODEL` and bump the memory fractions + accordingly. + +- **Separate GPUs** — if contention hurts latency, split them. Today all + three share `${GPU_ID}`. Adding `GPU_ID_REWARD`/etc. is a small compose + edit. + +## History + +- **2026-05-13 rename + extend** — stack renamed from `vllm-qwen3` to + `vllm` and gained the `vllm-reward` service (Skywork-Reward-V2 8B AWQ). + No GPU memory rebalance needed in practice — production had already + tuned EMBED/RERANK down from 0.40/0.40 to 0.20/0.20; adding REWARD at + 0.30 fits cleanly with ~14 GB headroom on the 48 GB Ada. + +- **Migrated off Infinity** — Infinity stopped shipping a `transformers` + build that knew Qwen3; this stack is the canonical replacement for both + embed and rerank. Consumers (AIPA agents, LibreChat RAG) point at + `:8001`/`:8002`. diff --git a/stacks/vllm-qwen3/compose.yaml b/stacks/vllm/compose.yaml similarity index 55% rename from stacks/vllm-qwen3/compose.yaml rename to stacks/vllm/compose.yaml index 4233333..5508679 100644 --- a/stacks/vllm-qwen3/compose.yaml +++ b/stacks/vllm/compose.yaml @@ -1,10 +1,16 @@ -# vLLM — Qwen3 Embedding + Reranker (one stack, two services). +# vLLM — Qwen3 Embedding + Reranker + Skywork Reward-V2 classifier. # -# Replaces the unmaintained Infinity stack. vLLM runs one model per process, -# so this stack brings up two containers sharing a single GPU: +# Originally created to replace the unmaintained Infinity stack (embed + +# rerank); generalized 2026-05-13 to host any vLLM-served model on ana-ml2, +# starting with the Skywork-Reward-V2-Llama-3.1-8B reward classifier +# (AWQ-quantized locally, served from /tank/aimodels/llm/). +# +# vLLM runs one model per process, so this stack brings up three containers +# sharing a single GPU: # # vllm-embed — Qwen3-Embedding served as an OpenAI /v1/embeddings server # vllm-rerank — Qwen3-Reranker served as a /rerank + /score server +# vllm-reward — Skywork-Reward-V2-Llama-3.1-8B-AWQ served as a /classify scorer # # The reranker is a causal-LM checkpoint; --hf-overrides re-maps it to # Qwen3ForSequenceClassification so vLLM's reranking endpoints work and the @@ -13,8 +19,14 @@ # All tunables live in .env — edit that, not this file. # # Pre-download models to avoid first-run delay: -# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Embedding-0.6B -# HF_HOME=/tank/aimodels/huggingface hf download Qwen/Qwen3-Reranker-0.6B +# scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \ +# --var hf_repo=Qwen/Qwen3-Embedding-0.6B +# scripts/elway ana-ml2 --playbook playbooks/pull-hf-repo.yaml \ +# --var hf_repo=Qwen/Qwen3-Reranker-0.6B +# +# Skywork-Reward-V2-Llama-3.1-8B-AWQ is a locally-quantized model — lives at +# /tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ on ana-ml2 and is +# bind-mounted into the reward service at /local-models. Not from HF Hub. services: vllm-embed: @@ -127,6 +139,64 @@ services: - homepage.description=Qwen3 Reranker via vLLM (ana-ml2) - homepage.href=http://10.250.50.54:${RERANK_PORT}/docs + vllm-reward: + image: vllm/vllm-openai:${VLLM_VERSION} + container_name: vllm-reward + restart: unless-stopped + ipc: host + ports: + - "${REWARD_PORT}:8000" + volumes: + # AWQ output lives in the legacy llama-swap models tree, not the HF cache + # — bind-mount the LLM models dir read-only so the reward service can + # load it as a local-path HF-format model. + - /tank/aimodels/llm:/local-models:ro + environment: + - VLLM_API_KEY=${API_KEY:-} + command: + - /local-models/Skywork-Reward-V2-Llama-3.1-8B-AWQ + - --served-model-name + - Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ + # vLLM 0.19.1 deprecated --task in favor of --runner. The model's + # config.json declares `LlamaForSequenceClassification` so the + # pooling runner uses it as a classifier (single-label reward score) + # without needing an explicit task flag. + - --runner + - pooling + - --host + - 0.0.0.0 + - --port + - "8000" + - --gpu-memory-utilization + - ${REWARD_GPU_MEM_UTIL} + - --max-model-len + - ${REWARD_MAX_MODEL_LEN} + - --dtype + - auto + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: + - "${GPU_ID}" + capabilities: + - gpu + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 240s + networks: + - tnet + labels: + - homepage.group=AI Systems + - homepage.name=vLLM Reward (Skywork) + - homepage.icon=mdi-scale-balance + - homepage.description=Skywork-Reward-V2 8B classifier via vLLM (ana-ml2) + - homepage.href=http://10.250.50.54:${REWARD_PORT}/docs + networks: tnet: name: traefik-net