vh
ca3c984f93
Benchmarked selene against gen on selene's own job: 24 designed judge items with checkable ground truth, pairwise + absolute modes, 3 repeats, on BOTH a neutral JSON prompt and Selene's native Atla template. 288 calls, all free local. neutral JSON selene 20/24 (83%) gen 23/24 (96%) native Atla selene 21/24 (88%) gen 22/24 (92%) gen won on both templates and selene's BEST sat below gen's WORST. Selene was given its own fine-tuned template as a fairness check; it gained one point, not the three it needed. Decisive defect: selene cannot emit "tie" -- 0/2 on both templates, forcing a winner on every equivalent pair. For eval work that is the case that matters most. gen returned tie correctly on the JSON template. Selene also compressed the 1-5 scale (clustered at 2s and 4s) where gen used it fully. Selene's only win was ~3x latency, unexercised at ~60 calls/day with zero queueing. TWO NAMES, TWO DIFFERENT TREATMENTS, deliberately: - chat-judge -> repointed to gen. It is a ROLE alias and ADR-0012 says consumers bind the capability, not a concrete model. Sampler profile copied from image-judge (temp 0, top_p 1.0, top_k 1, thinking off) so the served config matches the benchmarked condition. - selene-1-mini-8b -> REMOVED. It 404s. It was NOT aliased to gen. A served-name is a contract about what the model IS; answering it with a different model hides a material change behind a stable string. Operator ruling: "never repoint a named model at a different model's endpoint -- that is intentionally misleading." Verified: the gateway now returns HTTP 400 "Invalid model name" for it. Reclaimed 17.2 GiB on ana-ml2 GPU 1 (free 1,818 -> 19,450 MiB) on a card that had under 2 GiB of headroom. gen already runs on GPU 0, so the judge role moved onto an existing seat rather than allocating anything new. Canonical litellm config synced from the host; ana-ml2 README and recommended-model-settings updated. compose.yaml kept for reference, not deployed.
105 lines
4.9 KiB
Markdown
105 lines
4.9 KiB
Markdown
# ana-ml2
|
||
|
||
Primary AI inference host for PFI.
|
||
|
||
## Network
|
||
|
||
- **LAN IP:** 10.250.50.54 (in-band, OS-side)
|
||
- **BMC (OOB):** 10.250.250.50 — Supermicro IPMI web UI
|
||
at <https://10.250.250.50> (homepage card: *PFI-ANA-ML2 BMC*)
|
||
- **SSH:** standard port 22 on 10.250.50.54
|
||
|
||
## Hardware
|
||
|
||
- **Chassis:** Supermicro mid-range inferencing server (bare metal,
|
||
NOT Dell / not the same box as sf-r630 / sfsrv-ana)
|
||
- **CPU:** AMD EPYC 9254 24-core (96 threads)
|
||
- **RAM:** 566 GB
|
||
- **GPUs:** 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
|
||
- **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (8.6 TB at `/tank`)
|
||
- **OS:** Debian 13 (trixie), kernel 6.12.x
|
||
- **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
|
||
|
||
## Key paths
|
||
|
||
| Path | Purpose |
|
||
|------|---------|
|
||
| `/opt/docker/compose/<stack>/` | Compose files |
|
||
| `/opt/docker/conf/<stack>/` | Config bind mounts |
|
||
| `/tank/aimodels/huggingface/` | HF cache (267 GB, pre-downloaded models) |
|
||
| `/tank/aimodels/llm/` | Legacy GGUF models (790 GB, referenced by llama-swap as `/models/`) |
|
||
| `/var/lib/docker/` | Docker data (on zroot) |
|
||
|
||
## Running stacks
|
||
|
||
Live inventory as of 2026-07-22. Each model is its own compose stack now
|
||
(container `vllm-<x>` / `llama-<x>`); the `vllm` stack proper is just the
|
||
embed/rerank/reward trio. GPUs are pinned per container via
|
||
`deploy.resources.reservations.devices[].device_ids`.
|
||
|
||
**GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):**
|
||
|
||
| Container | Port | Served model | Quant | Ctx |
|
||
|-----------|------|--------------|-------|-----|
|
||
| `vllm-gen` (project `gen-seat`) | 8015 | `qwen3.8-27b-uncensored` — the "gen" hero seat (Qwen3.8-27B Heretic-abliterated, in-house NVFP4 W4A16 + grafted MTP) | NVFP4 W4A16 (compressed-tensors) | 262k |
|
||
| `vllm-charrp-reasoning-nvfp4` | 8018 | `char-rp-reasoning` (R36 reasoning RP) | NVFP4 (modelopt) | 256k |
|
||
|
||
**GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):**
|
||
|
||
| Container | Port | Served model | Quant | Ctx |
|
||
|-----------|------|--------------|-------|-----|
|
||
| `vllm-granite` | 8004 | `granite-4.1-8b` — fleet summarizer/classifier | FP8 (compressed-tensors) | 131k |
|
||
| `llama-charrp` | 8016 | `Magidonia-24B-v4.3` Q6_K — char-RP (llama.cpp) | GGUF Q6_K | — |
|
||
| ~~`vllm-selene`~~ | ~~8011~~ | **RETIRED 2026-08-23** — lost a head-to-head against `gen` on its own judge task (see `stacks/selene/README.md`); seat downed to reclaim 17.2 GiB on GPU 1. `selene-1-mini-8b` now 404s by design; use `chat-judge`. | — | — |
|
||
| `vllm-reward` | 8003 | `Skywork-Reward-V2-Llama-3.1-8B-AWQ` — reward classifier | AWQ | 16k |
|
||
| `vllm-embed` | 8001 | `Qwen3-Embedding-0.6B` | — | 8k |
|
||
| `vllm-rerank` | 8002 | `Qwen3-Reranker-0.6B` | — | 8k |
|
||
|
||
**Infra / non-GPU:**
|
||
|
||
| Container | Port | Notes |
|
||
|-----------|------|-------|
|
||
| `dockge` | 5001 | Docker stack management UI |
|
||
| `dozzle-agent` | 7007 | Log agent → Dozzle hub on ana-docker |
|
||
| `beszel-agent` | 45876 | Metrics agent → Beszel hub on ana-docker |
|
||
|
||
Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing
|
||
it on the card with room or evicting a dormant one first.
|
||
|
||
**Dormant (compose present on disk, containers stopped)** — rollback / audition
|
||
seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`,
|
||
`ms32-24b-angel`, `qwen3.5-122b`, `qwopus3.5-122b`, `qwen35-vl`, `qwen36-vl`,
|
||
`qwen36-27b-aeon`, `qwen-image-bench`, `vibevoice`, `comfyui`, `kokoro`,
|
||
`parakeet`, `vllm-qwen3`.
|
||
|
||
**Retired:**
|
||
- `llama-swap` (former GGUF multiplexer on :9292) — replaced by dedicated
|
||
per-model seats (e.g. `llama-charrp`); no longer running.
|
||
- `infinity` — replaced by the `vllm` stack (originally `vllm-qwen3`, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a `transformers` build that knew Qwen3.
|
||
- `LibreChat (+ rag_api, vectordb, mongodb, meilisearch)`, `searxng` — removed from this host (searxng now on ana-docker fleet-wide).
|
||
|
||
## Refresh state
|
||
|
||
```bash
|
||
scripts/refresh-server-info.sh ana-ml2
|
||
```
|
||
|
||
Latest snapshot: `system-details.txt` (regenerate as needed).
|
||
|
||
## GPU allocation policy
|
||
|
||
Every seat is explicitly pinned via `device_ids` (no unpinned containers), and
|
||
both cards run ~90% full:
|
||
|
||
- **GPU 0:** the two heavy NVFP4 seats — `vllm-gen` (gen) and
|
||
`vllm-charrp-reasoning-nvfp4`. The live serving path (near-100% util under
|
||
load), ~42 + 45 GB.
|
||
- **GPU 1:** everything else — reward,
|
||
embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle
|
||
between calls, ~91 GB resident.
|
||
|
||
Pin with `deploy.resources.reservations.devices[].device_ids: ["<id>"]` in
|
||
compose. Each service caps its share with `--gpu-memory-utilization`; with both
|
||
cards near-full, placing a new seat means freeing room (evict a dormant one) or
|
||
trimming a neighbour's utilization first.
|