143 lines
7.5 KiB
Markdown
143 lines
7.5 KiB
Markdown
# ana-ml2
|
||
|
||
Primary AI inference host for PFI.
|
||
|
||
## Network
|
||
|
||
- **LAN IP:** 10.250.50.54 (in-band, OS-side)
|
||
- **BMC (OOB):** 10.250.250.50 — Supermicro IPMI web UI
|
||
at <https://10.250.250.50> (homepage card: *PFI-ANA-ML2 BMC*)
|
||
- **SSH:** standard port 22 on 10.250.50.54
|
||
|
||
## Hardware
|
||
|
||
- **Chassis:** Supermicro mid-range inferencing server (bare metal,
|
||
NOT Dell / not the same box as sf-r630 / sfsrv-ana)
|
||
- **CPU:** AMD EPYC 9254 24-core (96 threads)
|
||
- **RAM:** 566 GB
|
||
- **GPUs:** 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
|
||
- **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (raidz2, 8× NVMe, 8.6 TB at `/tank`) — drive inventory below
|
||
- **OS:** Debian 13 (trixie), kernel 6.12.x
|
||
- **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
|
||
|
||
## NVMe drive inventory (`tank`, raidz2-0) — read 2026-09-09 via `nvme-cli`
|
||
|
||
All eight are Dell Express Flash PM1725b 1.6 TB SFF (Samsung OEM), PCIe 3.0 x4 behind
|
||
a Broadcom PEX switch. Two provenance batches: the `S5CU…` six (fw 1.2.2) and the
|
||
`S47V…` pair (fw 1.2.0 / 1.2.1) with thousands of prior-life power cycles.
|
||
|
||
| dev | PCI | serial | fw | pwr-on h | pwr cycles | unsafe shut. | media err | used |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| nvme0 | 46:00.0 | S5CUNEUMB05672 | 1.2.2 | 33856 | 196 | 175 | 0 | 0% |
|
||
| nvme1 | — | S5CUNEUMB05671 | 1.2.2 | 33856 | 199 | 178 | 0 | 0% |
|
||
| nvme2 | — | S5CUNEUMB05694 | 1.2.2 | 15688 | 90 | 75 | 0 | 0% |
|
||
| nvme3 | — | S5CUNEUMB05667 | 1.2.2 | 33857 | 197 | 176 | 0 | 0% |
|
||
| nvme4 | — | S5CUNEUMB05674 | 1.2.2 | 33856 | 198 | 177 | 0 | 0% |
|
||
| nvme5 | c6:00.0 | S47VNY0K600270 | 1.2.1 | 18823 | 5357 | 5342 | 0 | 1% |
|
||
| nvme6 | — | S5CUNEUMB05697 | 1.2.2 | 15570 | 88 | 73 | 0 | 0% |
|
||
| **nvme7** | 07:00.0 (slot 0-5) | S47VNY0K600221 | 1.2.0 | 19525 | 3093 | 3083 | **2084** | 2% |
|
||
|
||
⚠ **nvme7 was ABSENT from every boot 2026-04-23 → 2026-09-05** (kernel enumerated 7
|
||
NVMes per boot; PCIe downstream port `02:04.0` had nothing on bus 07). It reappeared at
|
||
the 09-05 14:26 cold boot, the pool resilvered 638 GB, and 2 CKSUM errors landed on it
|
||
at import. While it was missing `tank` was DEGRADED, and Debian's `zfsutils-linux` cron
|
||
(`/usr/lib/zfs-linux/{scrub,trim}`) only touches pools whose health is `ONLINE`, so tank
|
||
got **no scrub and no trim from 04-12 to 09-06**. ZED's `ZED_EMAIL_ADDR=root` has no
|
||
MTA behind it, so the 4½-month degradation alerted nobody. `media_errors=2084` on
|
||
nvme7 is a lifetime counter.
|
||
|
||
**Settled by the 2026-09-09 scrub** (00:29–02:02 PT, `scrub repaired 0B in 01:32:44
|
||
with 0 errors`, then `zpool clear tank` → CKSUM 2 → 0): `media_errors` read **2084
|
||
before and 2084 after** a full 6.84 TiB verify, so the counter is prior-life
|
||
history, not an active fault, and the 2 CKSUM were the stale-block artefact of the
|
||
09-05 late resilver. **nvme7 stays in service; watch the counter at every visit and
|
||
replace on growth** (`zpool replace tank nvme7n1 <new>`; any PM1725b 1.6 TB or
|
||
larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on
|
||
visit — a bay that dropped a drive for 4½ months is the likelier fault than the
|
||
drive. Playbook: `playbooks/ana-ml2-pool-health.yaml` (idempotent; rerunning is a
|
||
no-op). ⚠ **Nothing alerts on this** — see the open follow-up in
|
||
`persistent-memory.d/2026-09-09-ana-ml2-pool-actions-done.md`.
|
||
|
||
## Key paths
|
||
|
||
| Path | Purpose |
|
||
|------|---------|
|
||
| `/opt/docker/compose/<stack>/` | Compose files |
|
||
| `/opt/docker/conf/<stack>/` | Config bind mounts |
|
||
| `/tank/aimodels/huggingface/` | HF cache (267 GB, pre-downloaded models) |
|
||
| `/tank/aimodels/llm/` | Legacy GGUF models (790 GB, referenced by llama-swap as `/models/`) |
|
||
| `/var/lib/docker/` | Docker data (on zroot) |
|
||
|
||
## Running stacks
|
||
|
||
Live inventory as of 2026-07-22. Each model is its own compose stack now
|
||
(container `vllm-<x>` / `llama-<x>`); the `vllm` stack proper is just the
|
||
embed/rerank/reward trio. GPUs are pinned per container via
|
||
`deploy.resources.reservations.devices[].device_ids`.
|
||
|
||
**GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):**
|
||
|
||
| Container | Port | Served model | Quant | Ctx |
|
||
|-----------|------|--------------|-------|-----|
|
||
| `vllm-gen` (project `gen-seat`) | 8015 | `qwen3.8-27b-uncensored` — the "gen" hero seat (Qwen3.8-27B Heretic-abliterated, in-house NVFP4 W4A16 + grafted MTP) | NVFP4 W4A16 (compressed-tensors) | 262k |
|
||
| `vllm-charrp-reasoning-nvfp4` | 8018 | `char-rp-reasoning` (R36 reasoning RP) | NVFP4 (modelopt) | 256k |
|
||
|
||
**GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):**
|
||
|
||
| Container | Port | Served model | Quant | Ctx |
|
||
|-----------|------|--------------|-------|-----|
|
||
| `vllm-granite` | 8004 | `granite-4.1-8b` — fleet summarizer/classifier | FP8 (compressed-tensors) | 131k |
|
||
| `llama-charrp` | 8016 | `Magidonia-24B-v4.3` Q6_K — char-RP (llama.cpp) | GGUF Q6_K | — |
|
||
| ~~`vllm-selene`~~ | ~~8011~~ | **RETIRED 2026-08-23** — lost a head-to-head against `gen` on its own judge task (see `stacks/selene/README.md`); seat downed to reclaim 17.2 GiB on GPU 1. `selene-1-mini-8b` now 404s by design; use `chat-judge`. | — | — |
|
||
| `vllm-reward` | 8003 | `Skywork-Reward-V2-Llama-3.1-8B-AWQ` — reward classifier | AWQ | 16k |
|
||
| `vllm-embed` | 8001 | `Qwen3-Embedding-0.6B` | — | 8k |
|
||
| `vllm-rerank` | 8002 | `Qwen3-Reranker-0.6B` | — | 8k |
|
||
|
||
**Infra / non-GPU:**
|
||
|
||
| Container | Port | Notes |
|
||
|-----------|------|-------|
|
||
| `dockge` | 5001 | Docker stack management UI |
|
||
| `dozzle-agent` | 7007 | Log agent → Dozzle hub on ana-docker |
|
||
| `beszel-agent` | 45876 | Metrics agent → Beszel hub on ana-docker |
|
||
|
||
Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing
|
||
it on the card with room or evicting a dormant one first.
|
||
|
||
**Dormant (compose present on disk, containers stopped)** — rollback / audition
|
||
seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`,
|
||
`ms32-24b-angel`, `qwen3.5-122b`, `qwopus3.5-122b`, `qwen35-vl`, `qwen36-vl`,
|
||
`qwen36-27b-aeon`, `qwen-image-bench`, `vibevoice`, `comfyui`, `kokoro`,
|
||
`parakeet`, `vllm-qwen3`.
|
||
|
||
**Retired:**
|
||
- `llama-swap` (former GGUF multiplexer on :9292) — replaced by dedicated
|
||
per-model seats (e.g. `llama-charrp`); no longer running.
|
||
- `infinity` — replaced by the `vllm` stack (originally `vllm-qwen3`, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a `transformers` build that knew Qwen3.
|
||
- `LibreChat (+ rag_api, vectordb, mongodb, meilisearch)`, `searxng` — removed from this host (searxng now on ana-docker fleet-wide).
|
||
|
||
## Refresh state
|
||
|
||
```bash
|
||
scripts/refresh-server-info.sh ana-ml2
|
||
```
|
||
|
||
Latest snapshot: `system-details.txt` (regenerate as needed).
|
||
|
||
## GPU allocation policy
|
||
|
||
Every seat is explicitly pinned via `device_ids` (no unpinned containers), and
|
||
both cards run ~90% full:
|
||
|
||
- **GPU 0:** the two heavy NVFP4 seats — `vllm-gen` (gen) and
|
||
`vllm-charrp-reasoning-nvfp4`. The live serving path (near-100% util under
|
||
load), ~42 + 45 GB.
|
||
- **GPU 1:** everything else — reward,
|
||
embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle
|
||
between calls, ~91 GB resident.
|
||
|
||
Pin with `deploy.resources.reservations.devices[].device_ids: ["<id>"]` in
|
||
compose. Each service caps its share with `--gpu-memory-utilization`; with both
|
||
cards near-full, placing a new seat means freeing room (evict a dormant one) or
|
||
trimming a neighbour's utilization first.
|