esh-pfi-infrastructure/servers/fv-ml1/README.md

fv-ml1

Primary AI inference host for PFI.

Network

• LAN IP: 10.251.50.54 (in-band, OS-side)
• BMC (OOB): 10.251.250.50 — Supermicro IPMI web UI at https://10.251.250.50 (homepage card: PFI-ANA-ML2 BMC)
• SSH: standard port 22 on 10.251.50.54

Hardware

• Chassis: Supermicro mid-range inferencing server (bare metal, NOT Dell / not the same box as sf-r630 / sfsrv-ana)
• CPU: AMD EPYC 9254 24-core (96 threads)
• RAM: 566 GB
• GPUs: 4x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (95.6 GB VRAM each = 382 GB total, cc 12.0 / sm_120, GPU 0-3) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores. ⚠ This line read "2x" until 2026-09-19; nvidia-smi reports four. Read the host, not the doc.
• PSUs: four present (PS1-PS4, all ok). ⚠ See § Power — on a single circuit that redundancy does not protect against the failure most likely to happen.
• Storage: ZFS zroot (434 GB root) + tank pool (raidz2, 8× NVMe, 8.6 TB at /tank) — drive inventory below
• OS: Debian 13 (trixie), kernel 6.12.x
• Docker: 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2

NVMe drive inventory (tank, raidz2-0) — read 2026-09-09 via nvme-cli

All eight are Dell Express Flash PM1725b 1.6 TB SFF (Samsung OEM), PCIe 3.0 x4 behind a Broadcom PEX switch. Two provenance batches: the S5CU… six (fw 1.2.2) and the S47V… pair (fw 1.2.0 / 1.2.1) with thousands of prior-life power cycles.

dev	PCI	serial	fw	pwr-on h	pwr cycles	unsafe shut.	media err	used
nvme0	46:00.0	S5CUNEUMB05672	1.2.2	33856	196	175	0	0%
nvme1	—	S5CUNEUMB05671	1.2.2	33856	199	178	0	0%
nvme2	—	S5CUNEUMB05694	1.2.2	15688	90	75	0	0%
nvme3	—	S5CUNEUMB05667	1.2.2	33857	197	176	0	0%
nvme4	—	S5CUNEUMB05674	1.2.2	33856	198	177	0	0%
nvme5	c6:00.0	S47VNY0K600270	1.2.1	18823	5357	5342	0	1%
nvme6	—	S5CUNEUMB05697	1.2.2	15570	88	73	0	0%
nvme7	07:00.0 (slot 0-5)	S47VNY0K600221	1.2.0	19525	3093	3083	2084	2%

⚠ nvme7 was ABSENT from every boot 2026-04-23 → 2026-09-05 (kernel enumerated 7 NVMes per boot; PCIe downstream port 02:04.0 had nothing on bus 07). It reappeared at the 09-05 14:26 cold boot, the pool resilvered 638 GB, and 2 CKSUM errors landed on it at import. While it was missing tank was DEGRADED, and Debian's zfsutils-linux cron (/usr/lib/zfs-linux/{scrub,trim}) only touches pools whose health is ONLINE, so tank got no scrub and no trim from 04-12 to 09-06. ZED's ZED_EMAIL_ADDR=root has no MTA behind it, so the 4½-month degradation alerted nobody. media_errors=2084 on nvme7 is a lifetime counter.

Settled by the 2026-09-09 scrub (00:29–02:02 PT, scrub repaired 0B in 01:32:44 with 0 errors, then zpool clear tank → CKSUM 2 → 0): media_errors read 2084 before and 2084 after a full 6.84 TiB verify, so the counter is prior-life history, not an active fault, and the 2 CKSUM were the stale-block artefact of the 09-05 late resilver. nvme7 stays in service; watch the counter at every visit and replace on growth (zpool replace tank nvme7n1 <new>; any PM1725b 1.6 TB or larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on visit — a bay that dropped a drive for 4½ months is the likelier fault than the drive. Playbook: playbooks/fv-ml1-pool-health.yaml (idempotent; rerunning is a no-op). ⚠ Nothing alerts on this — see the open follow-up in persistent-memory.d/2026-09-09-fv-ml1-pool-actions-done.md.

Power — a single 20 A circuit, shared with the FV edge router

Confirmed by the operator 2026-09-19: fv-ml1 and the R420 running OPNsense are the ONLY loads on a dedicated 20 A circuit.

The budget

Circuit	20 A @ 120 V = 2400 VA absolute
Continuous limit (NEC 80%)	1920 W

A GPU inference host running for hours is a continuous load by definition, so 1920 W is the real ceiling, not 2400.

Measured (BMC, 2026-09-19, GPUs at idle)

ipmitool dcmi power reading
  instantaneous  390 W      min 386 W    max 461 W    avg 412 W
  sampling period 2423 s
nvidia-smi  per GPU, all four identical:
  power.min_limit      250 W
  power.limit          275 W   <- currently ENFORCED
  power.default_limit  300 W   <- the card's stock Max-Q TGP
  power.max_limit      325 W   <- firmware ceiling, NOT an operating point
GPU draw at time of reading: 3.6 / 3.7 / 62.7 / 7.0 W  ≈ 77 W total

⚠ Compare the cap against 300 W, not 325 W. The meaningful number is the stock TGP the cards would otherwise run at; 325 W is an overclock ceiling nobody should pick. So the 275 W cap is a 100 W saving across four cards (4 × 25 W) — not the 200 W you get by measuring against the firmware max. This file said 200 W until the operator corrected it on 2026-09-19.

So the non-GPU baseline is ~313 W (EPYC 9254 24C/96T, 5+ drives, fans, board).

Derived worst case

Load	capped 275 W	stock 300 W
4 GPUs	1100	1200
CPU + board + drives under load	~400	~400
fv-ml1 subtotal	~1500	~1600
R420 / OPNsense (estimate)	~125	~125
Total	~1625	~1725
% of the 1920 W continuous budget	~85%	~90%
Headroom	~295 W	~195 W

So the cap buys about 5 points of margin — 85% instead of 90%.

⚠ What this forbids

• Keep the 275 W caps. Stock 300 W is not itself illegal — it lands near 90% of continuous — but 90% leaves nothing for the R420 being heavier than estimated, for PSU efficiency if the BMC reports DC, or for a warm day. The cap costs ~8% of GPU power headroom and buys back ~100 W of circuit margin; on a shared breaker feeding the site's router, that is a good trade. Same posture as feedback_idle_vram_is_reserved_not_waste: the margin is the point, not waste waiting to be reclaimed.
• Never go to 325 W. That is a firmware ceiling, not an operating point, and it puts the circuit around 95% of continuous.
• Do not add a fifth GPU, or another box, on this circuit.
• Anything new here needs a load calculation first, against 1920 W, not 2400.

⚠ The coupling risk, which is worse than the trip

OPNsense on the R420 is the Fountain Valley edge. It shares the breaker with the thing most likely to trip it. So a GPU overload does not just reboot the inference host — it takes the site's router with it, and with the router gone there is no remote path in to diagnose or power-cycle anything. The failure is correlated and it locks you out of its own recovery.

Four PSUs on fv-ml1 do not help: PSU redundancy protects against a PSU dying, not against the circuit going away, and all four are downstream of one breaker.

Breaker trips are not hypothetical on this fleet — the ANA colo has 2026 incident history for exactly this (docs/pfi/headscale-mesh-plan.md: "breaker, PSU1, WAN admin closed").

⚠ What is NOT measured

Stated so nobody reads the table above as more solid than it is:

1. fv-ml1 has never been measured under real 4-GPU load. The 461 W max above is a 40-minute idle-ish sample. The ~1500 W figure is derived from the caps, not observed.
2. Unknown whether the BMC reports AC input or DC output. If DC, add ~8-10% for PSU efficiency — about 150 W at full load, which would take the circuit from 86% to ~94%.
3. The R420's draw is an estimate, not a reading.

The cheap way to close 1 and 2 together: run all four GPUs at cap (a saturating load), read ipmitool dcmi power reading at the top, and compare against a clamp meter on the circuit. Until then, treat 1650 W as a floor.

Key paths

Path	Purpose
/opt/docker/compose/<stack>/	Compose files
/opt/docker/conf/<stack>/	Config bind mounts
/tank/aimodels/huggingface/	HF cache (267 GB, pre-downloaded models)
/tank/aimodels/llm/	Legacy GGUF models (790 GB, referenced by llama-swap as /models/)
/var/lib/docker/	Docker data (on zroot)

Running stacks

Live inventory as of 2026-07-22. Each model is its own compose stack now (container vllm-<x> / llama-<x>); the vllm stack proper is just the embed/rerank/reward trio. GPUs are pinned per container via deploy.resources.reservations.devices[].device_ids.

GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):

Container	Port	Served model	Quant	Ctx
vllm-gen (project gen-seat)	8015	qwen3.8-27b-uncensored — the "gen" hero seat (Qwen3.8-27B Heretic-abliterated, in-house NVFP4 W4A16 + grafted MTP)	NVFP4 W4A16 (compressed-tensors)	262k
vllm-charrp-reasoning-nvfp4	8018	char-rp-reasoning (R36 reasoning RP)	NVFP4 (modelopt)	256k

GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):

Container	Port	Served model	Quant	Ctx
vllm-granite	8004	granite-4.1-8b — fleet summarizer/classifier	FP8 (compressed-tensors)	131k
llama-charrp	8016	Magidonia-24B-v4.3 Q6_K — char-RP (llama.cpp)	GGUF Q6_K	—
vllm-selene	8011	RETIRED 2026-08-23 — lost a head-to-head against gen on its own judge task (see stacks/selene/README.md); seat downed to reclaim 17.2 GiB on GPU 1. selene-1-mini-8b now 404s by design; use chat-judge.	—	—
vllm-reward	8003	Skywork-Reward-V2-Llama-3.1-8B-AWQ — reward classifier	AWQ	16k
vllm-embed	8001	Qwen3-Embedding-0.6B	—	8k
vllm-rerank	8002	Qwen3-Reranker-0.6B	—	8k

Infra / non-GPU:

Container	Port	Notes
dockge	5001	Docker stack management UI
dozzle-agent	7007	Log agent → Dozzle hub on ana-docker
beszel-agent	45876	Metrics agent → Beszel hub on ana-docker

Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing it on the card with room or evicting a dormant one first.

Dormant (compose present on disk, containers stopped) — rollback / audition seats, safe to leave: mistral-medium-3.5, mistral-small-4(-heretic), ms32-24b-angel, qwen3.5-122b, qwopus3.5-122b, qwen35-vl, qwen36-vl, qwen36-27b-aeon, qwen-image-bench, vibevoice, comfyui, kokoro, vllm-qwen3.

Also on GPU 0 (non-vLLM):

Container	Port	Serves	Notes
parakeet	8300	Parakeet-TDT 0.6B v3 int8 (25 languages)	ASR via sherpa-onnx, LiteLLM ext-stt / whisper-1. Relocated from irv-ml1 2026-09-15. ~800 MiB. stacks/parakeet/.

⚠ GPU 3 is deliberately kept EMPTY (2 MiB). It is the only card that can still take a full-size seat — flash-next needs 93 GiB of 96 — and vLLM sizes its KV cache against total VRAM rather than free VRAM, so even a sub-1 GB tenant there eats into a future big seat's profiling margin. Small seats go on GPU 0, which has the most uncommitted headroom (its seats commit util 0.88; GPU 1 is at 0.975 and GPU 2 at 0.96).

Retired:

• llama-swap (former GGUF multiplexer on :9292) — replaced by dedicated per-model seats (e.g. llama-charrp); no longer running.
• infinity — replaced by the vllm stack (originally vllm-qwen3, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a transformers build that knew Qwen3.
• LibreChat (+ rag_api, vectordb, mongodb, meilisearch), searxng — removed from this host (searxng now on ana-docker fleet-wide).

Refresh state

scripts/refresh-server-info.sh fv-ml1

Latest snapshot: system-details.txt (regenerate as needed).

GPU allocation policy

Every seat is explicitly pinned via device_ids (no unpinned containers), and both cards run ~90% full:

• GPU 0: the two heavy NVFP4 seats — vllm-gen (gen) and vllm-charrp-reasoning-nvfp4. The live serving path (near-100% util under load), ~42 + 45 GB.
• GPU 1: everything else — reward, embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle between calls, ~91 GB resident.

Pin with deploy.resources.reservations.devices[].device_ids: ["<id>"] in compose. Each service caps its share with --gpu-memory-utilization; with both cards near-full, placing a new seat means freeing room (evict a dormant one) or trimming a neighbour's utilization first.