vh
7e7130172e
Two related changes shipped together. The stack rename is independent but adding `vllm-reward` to the existing `vllm-qwen3` would have made that name actively misleading. **Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo references (README.md root, servers/ana-ml2/, stacks/llama-swap/, configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two intentional history mentions retained (servers/ana-ml2 + stacks/vllm README). **Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ on port 8003. The AWQ output is a locally-quantized model (not from HF), so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the shared HF cache. Model config.json declares LlamaForSequenceClassification which vLLM's pooling runner picks up automatically — produces a single reward score per input via /classify. **Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1 deprecated --task in favor of --runner pooling (model architecture in config.json drives the classification head). Compose uses --runner pooling with a comment explaining the substitution. **GPU memory:** no rebalance needed — production had already tuned EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB headroom on the 48 GB Ada. **Server-side:** brought existing vllm-qwen3 down, mv'd /opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed new compose via scripts/deploy-stack.sh, brought all 3 services up. **Smoke tests:** - /health on 8001/8002/8003 → 200 - /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ with max_model_len 16384 - /classify with a sample conversation → returns LABEL_0 with prob 0.9999 (single-output regression-style reward score, expected shape for a reward model)
67 lines
2.8 KiB
Markdown
67 lines
2.8 KiB
Markdown
# ana-ml2
|
|
|
|
Primary AI inference host for PFI.
|
|
|
|
## Network
|
|
|
|
- **LAN IP:** 10.250.50.54 (in-band, OS-side)
|
|
- **BMC (OOB):** 10.250.250.50 — Supermicro IPMI web UI
|
|
at <https://10.250.250.50> (homepage card: *PFI-ANA-ML2 BMC*)
|
|
- **SSH:** standard port 22 on 10.250.50.54
|
|
|
|
## Hardware
|
|
|
|
- **Chassis:** Supermicro mid-range inferencing server (bare metal,
|
|
NOT Dell / not the same box as sf-r630 / sfsrv-ana)
|
|
- **CPU:** AMD EPYC 9254 24-core (96 threads)
|
|
- **RAM:** 566 GB
|
|
- **GPUs:** 2x NVIDIA RTX 6000 Ada Generation (46 GB VRAM each, GPU 0 and GPU 1)
|
|
- **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (8.6 TB at `/tank`)
|
|
- **OS:** Debian 13 (trixie), kernel 6.12.x
|
|
- **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
|
|
|
|
## Key paths
|
|
|
|
| Path | Purpose |
|
|
|------|---------|
|
|
| `/opt/docker/compose/<stack>/` | Compose files |
|
|
| `/opt/docker/conf/<stack>/` | Config bind mounts |
|
|
| `/tank/aimodels/huggingface/` | HF cache (267 GB, pre-downloaded models) |
|
|
| `/tank/aimodels/llm/` | Legacy GGUF models (790 GB, referenced by llama-swap as `/models/`) |
|
|
| `/var/lib/docker/` | Docker data (on zroot) |
|
|
|
|
## Running stacks
|
|
|
|
| Stack | Port | Notes |
|
|
|-------|------|-------|
|
|
| llama-swap | 9292 | GGUF model server via llama.cpp |
|
|
| vllm-embed (Qwen3) | 8001 | OpenAI-compatible embeddings; part of the `vllm` stack (GPU 1) |
|
|
| vllm-rerank (Qwen3) | 8002 | OpenAI-compatible reranker; part of the `vllm` stack (GPU 1) |
|
|
| vllm-reward (Skywork) | 8003 | Skywork-Reward-V2-8B-AWQ classifier; part of the `vllm` stack (GPU 1) |
|
|
| dockge | 5001 | Docker stack management UI |
|
|
| dozzle-agent | 7007 | Log agent; reports to the Dozzle hub on ana-docker |
|
|
| beszel-agent | 45876 | Metrics agent; reports to the Beszel hub on ana-docker |
|
|
|
|
**Retired since last README update:**
|
|
- `infinity` — replaced by the `vllm` stack (originally `vllm-qwen3`, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a `transformers` build that knew Qwen3.
|
|
- `LibreChat (+ rag_api, vectordb, mongodb, meilisearch)` — removed from this host.
|
|
- `searxng` — now hosted on ana-docker for the whole fleet.
|
|
- Residual networks (`librechat_default`, `kokoro-tts-gpu_default`) from prior experiments are still present; safe to `docker network rm` at leisure.
|
|
|
|
## Refresh state
|
|
|
|
```bash
|
|
scripts/refresh-server-info.sh ana-ml2
|
|
```
|
|
|
|
Latest snapshot: `system-details.txt` (regenerate as needed).
|
|
|
|
## GPU allocation policy
|
|
|
|
By default, no container is pinned. For predictable performance when multiple GPU workloads run concurrently:
|
|
|
|
- **GPU 0:** heavy LLM (llama-swap big models).
|
|
- **GPU 1:** light services (the three `vllm` services share this GPU via `--gpu-memory-utilization`).
|
|
|
|
Use `deploy.resources.reservations.devices[].device_ids: ["<id>"]` in compose to pin.
|