esh-pfi-infrastructure/servers/ana-ml2

ana-ml2

Primary AI inference host for PFI.

Network

• LAN IP: 10.250.50.54 (in-band, OS-side)
• BMC (OOB): 10.250.250.50 — Supermicro IPMI web UI at https://10.250.250.50 (homepage card: PFI-ANA-ML2 BMC)
• SSH: standard port 22 on 10.250.50.54

Hardware

• Chassis: Supermicro mid-range inferencing server (bare metal, NOT Dell / not the same box as sf-r630 / sfsrv-ana)
• CPU: AMD EPYC 9254 24-core (96 threads)
• RAM: 566 GB
• GPUs: 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
• Storage: ZFS zroot (434 GB root) + tank pool (8.6 TB at /tank)
• OS: Debian 13 (trixie), kernel 6.12.x
• Docker: 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2

Key paths

Path	Purpose
/opt/docker/compose/<stack>/	Compose files
/opt/docker/conf/<stack>/	Config bind mounts
/tank/aimodels/huggingface/	HF cache (267 GB, pre-downloaded models)
/tank/aimodels/llm/	Legacy GGUF models (790 GB, referenced by llama-swap as /models/)
/var/lib/docker/	Docker data (on zroot)

Running stacks

Live inventory as of 2026-07-22. Each model is its own compose stack now (container vllm-<x> / llama-<x>); the vllm stack proper is just the embed/rerank/reward trio. GPUs are pinned per container via deploy.resources.reservations.devices[].device_ids.

GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):

Container	Port	Served model	Quant	Ctx
vllm-aeon-gen	8015	qwen3.6-35b-a3b-heretic — the "gen" hero seat	NVFP4 (modelopt)	256k
vllm-charrp-reasoning-nvfp4	8018	char-rp-reasoning (R36 reasoning RP)	NVFP4 (modelopt)	256k

GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):

Container	Port	Served model	Quant	Ctx
vllm-granite	8004	granite-4.1-8b — fleet summarizer/classifier	FP8 (compressed-tensors)	131k
llama-charrp	8016	Magidonia-24B-v4.3 Q6_K — char-RP (llama.cpp)	GGUF Q6_K	—
vllm-selene	8011	selene-1-mini-8b — Atla LLM-as-judge	FP8	32k
vllm-reward	8003	Skywork-Reward-V2-Llama-3.1-8B-AWQ — reward classifier	AWQ	16k
vllm-embed	8001	Qwen3-Embedding-0.6B	—	8k
vllm-rerank	8002	Qwen3-Reranker-0.6B	—	8k

Infra / non-GPU:

Container	Port	Notes
dockge	5001	Docker stack management UI
dozzle-agent	7007	Log agent → Dozzle hub on ana-docker
beszel-agent	45876	Metrics agent → Beszel hub on ana-docker

Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing it on the card with room or evicting a dormant one first.

Dormant (compose present on disk, containers stopped) — rollback / audition seats, safe to leave: mistral-medium-3.5, mistral-small-4(-heretic), ms32-24b-angel, qwen3.5-122b, qwopus3.5-122b, qwen35-vl, qwen36-vl, qwen36-27b-aeon, qwen-image-bench, vibevoice, comfyui, kokoro, parakeet, vllm-qwen3.

Retired:

• llama-swap (former GGUF multiplexer on :9292) — replaced by dedicated per-model seats (e.g. llama-charrp); no longer running.
• infinity — replaced by the vllm stack (originally vllm-qwen3, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a transformers build that knew Qwen3.
• LibreChat (+ rag_api, vectordb, mongodb, meilisearch), searxng — removed from this host (searxng now on ana-docker fleet-wide).

Refresh state

scripts/refresh-server-info.sh ana-ml2

Latest snapshot: system-details.txt (regenerate as needed).

GPU allocation policy

Every seat is explicitly pinned via device_ids (no unpinned containers), and both cards run ~90% full:

• GPU 0: the two heavy NVFP4 seats — vllm-aeon-gen (gen) and vllm-charrp-reasoning-nvfp4. The live serving path (near-100% util under load), ~42 + 45 GB.
• GPU 1: everything else — summarizer (granite), judge (selene), reward, embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle between calls, ~91 GB resident.

Pin with deploy.resources.reservations.devices[].device_ids: ["<id>"] in compose. Each service caps its share with --gpu-memory-utilization; with both cards near-full, placing a new seat means freeing room (evict a dormant one) or trimming a neighbour's utilization first.