Files

T

vh 7e7130172e vllm: rename stack from vllm-qwen3 → vllm + add Skywork reward classifier

Two related changes shipped together. The stack rename is independent
but adding `vllm-reward` to the existing `vllm-qwen3` would have made
that name actively misleading.

**Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo
references (README.md root, servers/ana-ml2/, stacks/llama-swap/,
configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two
intentional history mentions retained (servers/ana-ml2 + stacks/vllm
README).

**Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ
on port 8003. The AWQ output is a locally-quantized model (not from HF),
so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the
shared HF cache. Model config.json declares LlamaForSequenceClassification
which vLLM's pooling runner picks up automatically — produces a single
reward score per input via /classify.

**Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1
deprecated --task in favor of --runner pooling (model architecture in
config.json drives the classification head). Compose uses --runner
pooling with a comment explaining the substitution.

**GPU memory:** no rebalance needed — production had already tuned
EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now
matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB
headroom on the 48 GB Ada.

**Server-side:** brought existing vllm-qwen3 down, mv'd
/opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended
REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed
new compose via scripts/deploy-stack.sh, brought all 3 services up.

**Smoke tests:**
- /health on 8001/8002/8003 → 200
- /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
  with max_model_len 16384
- /classify with a sample conversation → returns LABEL_0 with prob 0.9999
  (single-output regression-style reward score, expected shape for a
  reward model)

2026-05-13 22:00:26 -07:00

2.8 KiB

Raw Blame History

ana-ml2

Primary AI inference host for PFI.

Network

LAN IP: 10.250.50.54 (in-band, OS-side)
BMC (OOB): 10.250.250.50 — Supermicro IPMI web UI at https://10.250.250.50 (homepage card: PFI-ANA-ML2 BMC)
SSH: standard port 22 on 10.250.50.54

Hardware

Chassis: Supermicro mid-range inferencing server (bare metal, NOT Dell / not the same box as sf-r630 / sfsrv-ana)
CPU: AMD EPYC 9254 24-core (96 threads)
RAM: 566 GB
GPUs: 2x NVIDIA RTX 6000 Ada Generation (46 GB VRAM each, GPU 0 and GPU 1)
Storage: ZFS zroot (434 GB root) + tank pool (8.6 TB at /tank)
OS: Debian 13 (trixie), kernel 6.12.x
Docker: 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2

Key paths

Path	Purpose
`/opt/docker/compose/<stack>/`	Compose files
`/opt/docker/conf/<stack>/`	Config bind mounts
`/tank/aimodels/huggingface/`	HF cache (267 GB, pre-downloaded models)
`/tank/aimodels/llm/`	Legacy GGUF models (790 GB, referenced by llama-swap as `/models/`)
`/var/lib/docker/`	Docker data (on zroot)

Running stacks

Stack	Port	Notes
llama-swap	9292	GGUF model server via llama.cpp
vllm-embed (Qwen3)	8001	OpenAI-compatible embeddings; part of the `vllm` stack (GPU 1)
vllm-rerank (Qwen3)	8002	OpenAI-compatible reranker; part of the `vllm` stack (GPU 1)
vllm-reward (Skywork)	8003	Skywork-Reward-V2-8B-AWQ classifier; part of the `vllm` stack (GPU 1)
dockge	5001	Docker stack management UI
dozzle-agent	7007	Log agent; reports to the Dozzle hub on ana-docker
beszel-agent	45876	Metrics agent; reports to the Beszel hub on ana-docker

Retired since last README update:

infinity — replaced by the vllm stack (originally vllm-qwen3, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a transformers build that knew Qwen3.
LibreChat (+ rag_api, vectordb, mongodb, meilisearch) — removed from this host.
searxng — now hosted on ana-docker for the whole fleet.
Residual networks (librechat_default, kokoro-tts-gpu_default) from prior experiments are still present; safe to docker network rm at leisure.

Refresh state

scripts/refresh-server-info.sh ana-ml2

Latest snapshot: system-details.txt (regenerate as needed).

GPU allocation policy

By default, no container is pinned. For predictable performance when multiple GPU workloads run concurrently:

GPU 0: heavy LLM (llama-swap big models).
GPU 1: light services (the three vllm services share this GPU via --gpu-memory-utilization).

Use deploy.resources.reservations.devices[].device_ids: ["<id>"] in compose to pin.

2.8 KiB Raw Blame History