vh
966324c5f1
The README's running-stacks table had drifted well behind reality (still listed llama-swap + only the embed/rerank/reward trio). Regenerated system-details.txt and rewrote the stacks + GPU-allocation sections from a live docker ps + nvidia-smi (2026-07-22): - GPU 0 (hot): vllm-aeon-gen (qwen3.6-35b-a3b-heretic, NVFP4) + vllm-charrp- reasoning-nvfp4 (char-rp-reasoning, NVFP4), ~42+45 GB. - GPU 1 (on-demand): granite-4.1-8b, selene-1-mini-8b, Skywork reward, Qwen3 embed/rerank, and the Magidonia-24B char-RP GGUF (llama-charrp), ~91 GB. - Recorded the dormant on-disk stacks and llama-swap's retirement.
105 lines
4.7 KiB
Markdown
105 lines
4.7 KiB
Markdown
# ana-ml2
|
||
|
||
Primary AI inference host for PFI.
|
||
|
||
## Network
|
||
|
||
- **LAN IP:** 10.250.50.54 (in-band, OS-side)
|
||
- **BMC (OOB):** 10.250.250.50 — Supermicro IPMI web UI
|
||
at <https://10.250.250.50> (homepage card: *PFI-ANA-ML2 BMC*)
|
||
- **SSH:** standard port 22 on 10.250.50.54
|
||
|
||
## Hardware
|
||
|
||
- **Chassis:** Supermicro mid-range inferencing server (bare metal,
|
||
NOT Dell / not the same box as sf-r630 / sfsrv-ana)
|
||
- **CPU:** AMD EPYC 9254 24-core (96 threads)
|
||
- **RAM:** 566 GB
|
||
- **GPUs:** 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
|
||
- **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (8.6 TB at `/tank`)
|
||
- **OS:** Debian 13 (trixie), kernel 6.12.x
|
||
- **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
|
||
|
||
## Key paths
|
||
|
||
| Path | Purpose |
|
||
|------|---------|
|
||
| `/opt/docker/compose/<stack>/` | Compose files |
|
||
| `/opt/docker/conf/<stack>/` | Config bind mounts |
|
||
| `/tank/aimodels/huggingface/` | HF cache (267 GB, pre-downloaded models) |
|
||
| `/tank/aimodels/llm/` | Legacy GGUF models (790 GB, referenced by llama-swap as `/models/`) |
|
||
| `/var/lib/docker/` | Docker data (on zroot) |
|
||
|
||
## Running stacks
|
||
|
||
Live inventory as of 2026-07-22. Each model is its own compose stack now
|
||
(container `vllm-<x>` / `llama-<x>`); the `vllm` stack proper is just the
|
||
embed/rerank/reward trio. GPUs are pinned per container via
|
||
`deploy.resources.reservations.devices[].device_ids`.
|
||
|
||
**GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):**
|
||
|
||
| Container | Port | Served model | Quant | Ctx |
|
||
|-----------|------|--------------|-------|-----|
|
||
| `vllm-aeon-gen` | 8015 | `qwen3.6-35b-a3b-heretic` — the "gen" hero seat | NVFP4 (modelopt) | 256k |
|
||
| `vllm-charrp-reasoning-nvfp4` | 8018 | `char-rp-reasoning` (R36 reasoning RP) | NVFP4 (modelopt) | 256k |
|
||
|
||
**GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):**
|
||
|
||
| Container | Port | Served model | Quant | Ctx |
|
||
|-----------|------|--------------|-------|-----|
|
||
| `vllm-granite` | 8004 | `granite-4.1-8b` — fleet summarizer/classifier | FP8 (compressed-tensors) | 131k |
|
||
| `llama-charrp` | 8016 | `Magidonia-24B-v4.3` Q6_K — char-RP (llama.cpp) | GGUF Q6_K | — |
|
||
| `vllm-selene` | 8011 | `selene-1-mini-8b` — Atla LLM-as-judge | FP8 | 32k |
|
||
| `vllm-reward` | 8003 | `Skywork-Reward-V2-Llama-3.1-8B-AWQ` — reward classifier | AWQ | 16k |
|
||
| `vllm-embed` | 8001 | `Qwen3-Embedding-0.6B` | — | 8k |
|
||
| `vllm-rerank` | 8002 | `Qwen3-Reranker-0.6B` | — | 8k |
|
||
|
||
**Infra / non-GPU:**
|
||
|
||
| Container | Port | Notes |
|
||
|-----------|------|-------|
|
||
| `dockge` | 5001 | Docker stack management UI |
|
||
| `dozzle-agent` | 7007 | Log agent → Dozzle hub on ana-docker |
|
||
| `beszel-agent` | 45876 | Metrics agent → Beszel hub on ana-docker |
|
||
|
||
Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing
|
||
it on the card with room or evicting a dormant one first.
|
||
|
||
**Dormant (compose present on disk, containers stopped)** — rollback / audition
|
||
seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`,
|
||
`ms32-24b-angel`, `qwen3.5-122b`, `qwopus3.5-122b`, `qwen35-vl`, `qwen36-vl`,
|
||
`qwen36-27b-aeon`, `qwen-image-bench`, `vibevoice`, `comfyui`, `kokoro`,
|
||
`parakeet`, `vllm-qwen3`.
|
||
|
||
**Retired:**
|
||
- `llama-swap` (former GGUF multiplexer on :9292) — replaced by dedicated
|
||
per-model seats (e.g. `llama-charrp`); no longer running.
|
||
- `infinity` — replaced by the `vllm` stack (originally `vllm-qwen3`, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a `transformers` build that knew Qwen3.
|
||
- `LibreChat (+ rag_api, vectordb, mongodb, meilisearch)`, `searxng` — removed from this host (searxng now on ana-docker fleet-wide).
|
||
|
||
## Refresh state
|
||
|
||
```bash
|
||
scripts/refresh-server-info.sh ana-ml2
|
||
```
|
||
|
||
Latest snapshot: `system-details.txt` (regenerate as needed).
|
||
|
||
## GPU allocation policy
|
||
|
||
Every seat is explicitly pinned via `device_ids` (no unpinned containers), and
|
||
both cards run ~90% full:
|
||
|
||
- **GPU 0:** the two heavy NVFP4 seats — `vllm-aeon-gen` (gen) and
|
||
`vllm-charrp-reasoning-nvfp4`. The live serving path (near-100% util under
|
||
load), ~42 + 45 GB.
|
||
- **GPU 1:** everything else — summarizer (granite), judge (selene), reward,
|
||
embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle
|
||
between calls, ~91 GB resident.
|
||
|
||
Pin with `deploy.resources.reservations.devices[].device_ids: ["<id>"]` in
|
||
compose. Each service caps its share with `--gpu-memory-utilization`; with both
|
||
cards near-full, placing a new seat means freeing room (evict a dormant one) or
|
||
trimming a neighbour's utilization first.
|