vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
19aeaedeba21b3d9f36de8b726d9717358a01883
esh-pfi-infrastructure/servers/ana-ml2
T
History

README.md

ana-ml2

Primary AI inference host for PFI.

Network

  • LAN IP: 10.250.50.54 (in-band, OS-side)
  • BMC (OOB): 10.250.250.50 — Supermicro IPMI web UI at https://10.250.250.50 (homepage card: PFI-ANA-ML2 BMC)
  • SSH: standard port 22 on 10.250.50.54

Hardware

  • Chassis: Supermicro mid-range inferencing server (bare metal, NOT Dell / not the same box as sf-r630 / sfsrv-ana)
  • CPU: AMD EPYC 9254 24-core (96 threads)
  • RAM: 566 GB
  • GPUs: 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
  • Storage: ZFS zroot (434 GB root) + tank pool (raidz2, 8× NVMe, 8.6 TB at /tank) — drive inventory below
  • OS: Debian 13 (trixie), kernel 6.12.x
  • Docker: 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2

NVMe drive inventory (tank, raidz2-0) — read 2026-09-09 via nvme-cli

All eight are Dell Express Flash PM1725b 1.6 TB SFF (Samsung OEM), PCIe 3.0 x4 behind a Broadcom PEX switch. Two provenance batches: the S5CU… six (fw 1.2.2) and the S47V… pair (fw 1.2.0 / 1.2.1) with thousands of prior-life power cycles.

dev PCI serial fw pwr-on h pwr cycles unsafe shut. media err used
nvme0 46:00.0 S5CUNEUMB05672 1.2.2 33856 196 175 0 0%
nvme1 — S5CUNEUMB05671 1.2.2 33856 199 178 0 0%
nvme2 — S5CUNEUMB05694 1.2.2 15688 90 75 0 0%
nvme3 — S5CUNEUMB05667 1.2.2 33857 197 176 0 0%
nvme4 — S5CUNEUMB05674 1.2.2 33856 198 177 0 0%
nvme5 c6:00.0 S47VNY0K600270 1.2.1 18823 5357 5342 0 1%
nvme6 — S5CUNEUMB05697 1.2.2 15570 88 73 0 0%
nvme7 07:00.0 (slot 0-5) S47VNY0K600221 1.2.0 19525 3093 3083 2084 2%

⚠ nvme7 was ABSENT from every boot 2026-04-23 → 2026-09-05 (kernel enumerated 7 NVMes per boot; PCIe downstream port 02:04.0 had nothing on bus 07). It reappeared at the 09-05 14:26 cold boot, the pool resilvered 638 GB, and 2 CKSUM errors landed on it at import. While it was missing tank was DEGRADED, and Debian's zfsutils-linux cron (/usr/lib/zfs-linux/{scrub,trim}) only touches pools whose health is ONLINE, so tank got no scrub and no trim from 04-12 to 09-06. ZED's ZED_EMAIL_ADDR=root has no MTA behind it, so the 4½-month degradation alerted nobody. media_errors=2084 on nvme7 is a lifetime counter.

Settled by the 2026-09-09 scrub (00:29–02:02 PT, scrub repaired 0B in 01:32:44 with 0 errors, then zpool clear tank → CKSUM 2 → 0): media_errors read 2084 before and 2084 after a full 6.84 TiB verify, so the counter is prior-life history, not an active fault, and the 2 CKSUM were the stale-block artefact of the 09-05 late resilver. nvme7 stays in service; watch the counter at every visit and replace on growth (zpool replace tank nvme7n1 <new>; any PM1725b 1.6 TB or larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on visit — a bay that dropped a drive for 4½ months is the likelier fault than the drive. Playbook: playbooks/ana-ml2-pool-health.yaml (idempotent; rerunning is a no-op). ⚠ Nothing alerts on this — see the open follow-up in persistent-memory.d/2026-09-09-ana-ml2-pool-actions-done.md.

Key paths

Path Purpose
/opt/docker/compose/<stack>/ Compose files
/opt/docker/conf/<stack>/ Config bind mounts
/tank/aimodels/huggingface/ HF cache (267 GB, pre-downloaded models)
/tank/aimodels/llm/ Legacy GGUF models (790 GB, referenced by llama-swap as /models/)
/var/lib/docker/ Docker data (on zroot)

Running stacks

Live inventory as of 2026-07-22. Each model is its own compose stack now (container vllm-<x> / llama-<x>); the vllm stack proper is just the embed/rerank/reward trio. GPUs are pinned per container via deploy.resources.reservations.devices[].device_ids.

GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):

Container Port Served model Quant Ctx
vllm-gen (project gen-seat) 8015 qwen3.8-27b-uncensored — the "gen" hero seat (Qwen3.8-27B Heretic-abliterated, in-house NVFP4 W4A16 + grafted MTP) NVFP4 W4A16 (compressed-tensors) 262k
vllm-charrp-reasoning-nvfp4 8018 char-rp-reasoning (R36 reasoning RP) NVFP4 (modelopt) 256k

GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):

Container Port Served model Quant Ctx
vllm-granite 8004 granite-4.1-8b — fleet summarizer/classifier FP8 (compressed-tensors) 131k
llama-charrp 8016 Magidonia-24B-v4.3 Q6_K — char-RP (llama.cpp) GGUF Q6_K —
vllm-selene 8011 RETIRED 2026-08-23 — lost a head-to-head against gen on its own judge task (see stacks/selene/README.md); seat downed to reclaim 17.2 GiB on GPU 1. selene-1-mini-8b now 404s by design; use chat-judge. — —
vllm-reward 8003 Skywork-Reward-V2-Llama-3.1-8B-AWQ — reward classifier AWQ 16k
vllm-embed 8001 Qwen3-Embedding-0.6B — 8k
vllm-rerank 8002 Qwen3-Reranker-0.6B — 8k

Infra / non-GPU:

Container Port Notes
dockge 5001 Docker stack management UI
dozzle-agent 7007 Log agent → Dozzle hub on ana-docker
beszel-agent 45876 Metrics agent → Beszel hub on ana-docker

Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing it on the card with room or evicting a dormant one first.

Dormant (compose present on disk, containers stopped) — rollback / audition seats, safe to leave: mistral-medium-3.5, mistral-small-4(-heretic), ms32-24b-angel, qwen3.5-122b, qwopus3.5-122b, qwen35-vl, qwen36-vl, qwen36-27b-aeon, qwen-image-bench, vibevoice, comfyui, kokoro, parakeet, vllm-qwen3.

Retired:

  • llama-swap (former GGUF multiplexer on :9292) — replaced by dedicated per-model seats (e.g. llama-charrp); no longer running.
  • infinity — replaced by the vllm stack (originally vllm-qwen3, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a transformers build that knew Qwen3.
  • LibreChat (+ rag_api, vectordb, mongodb, meilisearch), searxng — removed from this host (searxng now on ana-docker fleet-wide).

Refresh state

scripts/refresh-server-info.sh ana-ml2

Latest snapshot: system-details.txt (regenerate as needed).

GPU allocation policy

Every seat is explicitly pinned via device_ids (no unpinned containers), and both cards run ~90% full:

  • GPU 0: the two heavy NVFP4 seats — vllm-gen (gen) and vllm-charrp-reasoning-nvfp4. The live serving path (near-100% util under load), ~42 + 45 GB.
  • GPU 1: everything else — reward, embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle between calls, ~91 GB resident.

Pin with deploy.resources.reservations.devices[].device_ids: ["<id>"] in compose. Each service caps its share with --gpu-memory-utilization; with both cards near-full, placing a new seat means freeing room (evict a dormant one) or trimming a neighbour's utilization first.

Reference in New Issue View Git Blame Copy Permalink