LAN IP: 10.251.50.54 (in-band, OS-side)
BMC (OOB): 10.251.250.50 — Supermicro IPMI web UI at https://10.251.250.50 (homepage card: PFI-ANA-ML2 BMC)
SSH: standard port 22 on 10.251.50.54

Hardware

Chassis: Supermicro mid-range inferencing server (bare metal, NOT Dell / not the same box as sf-r630 / sfsrv-ana)
CPU: AMD EPYC 9254 24-core (96 threads)
RAM: 566 GB
GPUs: 4x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (95.6 GB VRAM each = 382 GB total, cc 12.0 / sm_120, GPU 0-3) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores. ⚠ This line read "2x" until 2026-09-19; nvidia-smi reports four. Read the host, not the doc.
PSUs: four present (PS1-PS4, all ok). ⚠ See § Power — on a single circuit that redundancy does not protect against the failure most likely to happen.
Storage: ZFS zroot (434 GB root) + tank pool (raidz2, 8× NVMe, 8.6 TB at /tank) — drive inventory below
Root-pool hygiene (2026-10-04, Prime): HF_HOME=/tank/aimodels/huggingface system-wide (/etc/environment + /etc/profile.d/hf-home.sh), the same tree the model containers mount as /hf; group llm (lkraven, llmuser, infra-ops) writes it and every dir carries setgid so new files stay group llm. All user HF caches were merged into it (rsync --ignore-existing, nothing lost) and cleared, as were the uv and pip caches: zroot went from 124 G free to 194 G. Left on purpose: lkraven's models--Cloudflare--clef-flash (18 G), in use by the 2026-10-04 bench; it is already on tank, so delete it once that bench is done. Docker moved to tank on 2026-10-05 (see the Docker bullet below). sudo session recordings are archived weekly by services/sudo-io-archive/.
OS: Debian 13 (trixie), kernel 6.12.x
Docker: 29.8.2 (since 2026-10-05), runtimes: runc (default), nvidia, io.containerd.runc.v2
⚠ /var/lib/docker is the zvol tank/docker (sparse 1 TiB, ext4), not zroot, since 2026-10-05 0233 (Prime, park 99). It is mounted by /etc/fstab with nofail after zfs-volume-wait.service, and docker.service.d/10-requires-docker-mount.conf (RequiresMountsFor=/var/lib/docker) stops Docker from ever starting on an empty root directory. It is a zvol, not a dataset, because Docker's overlay2 refuses a ZFS backing filesystem (an unpinned daemon would pick the zfs graph driver and lose every image). Freed blocks reach the zvol only through TRIM: fstrim.timer is enabled weekly (the first trim gave back 108 G). Root went from 49% to 8% used. Move script: scripts/oneoff/fv-ml1-docker-to-tank-20261005.sh. ⚠ Lesson from run 1, which aborted on its own guard: a live rsync of /var/lib/docker without -x walks INTO mounted overlay2/*/merged views and copies whole container root filesystems.
Patched 2026-10-08 0025–0042 (Prime's go): kernel 6.12.111 (metas reinstalled; 6.12.41/.73 purged; 6.12.74 kept as the fallback, selectable in ZFSBootMenu), ZFS 2.3.9 (kmod + userland), NVIDIA 580.178.04 (from 580.65.06; same R580 branch, CUDA 13.0), nvidia-container-toolkit 1.20.1. One reboot: down 0030:42, ssh back 0034:23, all 13 containers healthy at 0041:39. Every fv-ml1 gateway alias answered; the STT round-trip passed.
- ⚠ Boot is ZFSBootMenu (EFI \EFI\ZBM\VMLINUZ.EFI, backup entry too), not GRUB: there is no /etc/default/grub. ZBM boots the newest kernel in /boot (org.zfsbootmenu:kernel is -). To fall back, pick the kernel or a snapshot in the ZBM menu over SOL or iKVM. BMC credentials are in the vault as ana-ml2/bmc-infra-ops (user infra-ops, OPERATOR); SOL is enabled; ipmitool is on nh3-dev.
- ⚠ Where the driver comes from: the installed apt source is the CUDA local repo (cuda-debian12-13-0-local, 580.65.06 only). NVIDIA's network debian12 repo has the R580 updates, but trixie's apt (sqv) rejects its signing key (SHA1 binding signature, refused since 2026-02-01), and its debian13 repo carries only 590+. So 580.178.04 was installed from 45 .debs fetched directly: InRelease checked with gpgv against NVIDIA's keyring → Packages.gz sha256 → each deb's sha256 (~infra-ops/maint-20261008/fetch-nvidia.py, amd64/all only, the proprietary nvidia-kernel-dkms excluded because this host runs the open modules). The network repo is not left configured, so apt update stays clean. Do the same for the next driver bump.
- Holds re-applied afterwards: kernel metas, the ZFS stack, nvidia-container-toolkit. Bump them deliberately, checking dkms status for the new kernel before any reboot.
- Rollback snapshot zroot/ROOT/debian@pre-patch-20261008 (1.3 G); destroy after ~10-15 if nothing regressed.

NVMe drive inventory (`tank`, raidz2-0) — read 2026-09-09 via `nvme-cli`

All eight are Dell Express Flash PM1725b 1.6 TB SFF (Samsung OEM), PCIe 3.0 x4 behind a Broadcom PEX switch. Two provenance batches: the S5CU… six (fw 1.2.2) and the S47V… pair (fw 1.2.0 / 1.2.1) with thousands of prior-life power cycles.

dev	PCI	serial	fw	pwr-on h	pwr cycles	unsafe shut.	media err	used
nvme0	46:00.0	S5CUNEUMB05672	1.2.2	33856	196	175	0	0%
nvme1	—	S5CUNEUMB05671	1.2.2	33856	199	178	0	0%
nvme2	—	S5CUNEUMB05694	1.2.2	15688	90	75	0	0%
nvme3	—	S5CUNEUMB05667	1.2.2	33857	197	176	0	0%
nvme4	—	S5CUNEUMB05674	1.2.2	33856	198	177	0	0%
nvme5	c6:00.0	S47VNY0K600270	1.2.1	18823	5357	5342	0	1%
nvme6	—	S5CUNEUMB05697	1.2.2	15570	88	73	0	0%
nvme7	07:00.0 (slot 0-5)	S47VNY0K600221	1.2.0	19525	3093	3083	2084	2%

⚠ nvme7 was ABSENT from every boot 2026-04-23 → 2026-09-05 (kernel enumerated 7 NVMes per boot; PCIe downstream port 02:04.0 had nothing on bus 07). It reappeared at the 09-05 14:26 cold boot, the pool resilvered 638 GB, and 2 CKSUM errors landed on it at import. While it was missing tank was DEGRADED, and Debian's zfsutils-linux cron (/usr/lib/zfs-linux/{scrub,trim}) only touches pools whose health is ONLINE, so tank got no scrub and no trim from 04-12 to 09-06. ZED's ZED_EMAIL_ADDR=root has no MTA behind it, so the 4½-month degradation alerted nobody. media_errors=2084 on nvme7 is a lifetime counter.

Settled by the 2026-09-09 scrub (00:29–02:02 PT, scrub repaired 0B in 01:32:44 with 0 errors, then zpool clear tank → CKSUM 2 → 0): media_errors read 2084 before and 2084 after a full 6.84 TiB verify, so the counter is prior-life history, not an active fault, and the 2 CKSUM were the stale-block artefact of the 09-05 late resilver. nvme7 stays in service; watch the counter at every visit and replace on growth (zpool replace tank nvme7n1 <new>; any PM1725b 1.6 TB or larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on visit — a bay that dropped a drive for 4½ months is the likelier fault than the drive. Playbook: playbooks/fv-ml1-pool-health.yaml (idempotent; rerunning is a no-op). ⚠ Nothing alerts on this — see the open follow-up in persistent-memory.d/2026-09-09-fv-ml1-pool-actions-done.md.

Power — a single 20 A circuit, shared with the FV edge router

Confirmed by the operator 2026-09-19: fv-ml1 and the R420 running OPNsense are the ONLY loads on a dedicated 20 A circuit.

The budget


Circuit	20 A @ 120 V = 2400 VA absolute
Continuous limit (NEC 80%)	1920 W

A GPU inference host running for hours is a continuous load by definition, so 1920 W is the real ceiling, not 2400.

Measured (BMC, 2026-09-19, GPUs at idle)

ipmitool dcmi power reading
  instantaneous  390 W      min 386 W    max 461 W    avg 412 W
  sampling period 2423 s
nvidia-smi  per GPU, all four identical:
  power.min_limit      250 W
  power.limit          275 W   <- currently ENFORCED
  power.default_limit  300 W   <- the card's stock Max-Q TGP
  power.max_limit      325 W   <- firmware ceiling, NOT an operating point
GPU draw at time of reading: 3.6 / 3.7 / 62.7 / 7.0 W  ≈ 77 W total

⚠ Compare the cap against 300 W, not 325 W. The meaningful number is the stock TGP the cards would otherwise run at; 325 W is an overclock ceiling nobody should pick. So the 275 W cap is a 100 W saving across four cards (4 × 25 W) — not the 200 W you get by measuring against the firmware max. This file said 200 W until the operator corrected it on 2026-09-19.

So the non-GPU baseline is ~313 W (EPYC 9254 24C/96T, 5+ drives, fans, board).

Derived worst case

Load	capped 275 W	stock 300 W
4 GPUs	1100	1200
CPU + board + drives under load	~400	~400
fv-ml1 subtotal	~1500	~1600
R420 / OPNsense (estimate)	~125	~125
Total	~1625	~1725
% of the 1920 W continuous budget	~85%	~90%
Headroom	~295 W	~195 W

So the cap buys about 5 points of margin — 85% instead of 90%.

⚠ What this forbids

Keep the 275 W caps. Stock 300 W is not itself illegal — it lands near 90% of continuous — but 90% leaves nothing for the R420 being heavier than estimated, for PSU efficiency if the BMC reports DC, or for a warm day. The cap costs ~8% of GPU power headroom and buys back ~100 W of circuit margin; on a shared breaker feeding the site's router, that is a good trade. Same posture as feedback_idle_vram_is_reserved_not_waste: the margin is the point, not waste waiting to be reclaimed.
Never go to 325 W. That is a firmware ceiling, not an operating point, and it puts the circuit around 95% of continuous.
Do not add a fifth GPU, or another box, on this circuit.
Anything new here needs a load calculation first, against 1920 W, not 2400.

⚠ The coupling risk, which is worse than the trip

OPNsense on the R420 is the Fountain Valley edge. It shares the breaker with the thing most likely to trip it. So a GPU overload does not just reboot the inference host — it takes the site's router with it, and with the router gone there is no remote path in to diagnose or power-cycle anything. The failure is correlated and it locks you out of its own recovery.

Four PSUs on fv-ml1 do not help: PSU redundancy protects against a PSU dying, not against the circuit going away, and all four are downstream of one breaker.

Breaker trips are not hypothetical on this fleet — the ANA colo has 2026 incident history for exactly this (docs/pfi/headscale-mesh-plan.md: "breaker, PSU1, WAN admin closed").

⚠ What is NOT measured

Stated so nobody reads the table above as more solid than it is:

fv-ml1 has never been measured under real 4-GPU load. The 461 W max above is a 40-minute idle-ish sample. The ~1500 W figure is derived from the caps, not observed.
Unknown whether the BMC reports AC input or DC output. If DC, add ~8-10% for PSU efficiency — about 150 W at full load, which would take the circuit from 86% to ~94%.
The R420's draw is an estimate, not a reading.

The cheap way to close 1 and 2 together: run all four GPUs at cap (a saturating load), read ipmitool dcmi power reading at the top, and compare against a clamp meter on the circuit. Until then, treat 1650 W as a floor.

Key paths

Path	Purpose
`/opt/docker/compose/<stack>/`	Compose files
`/opt/docker/conf/<stack>/`	Config bind mounts
`/tank/aimodels/huggingface/`	HF cache (267 GB, pre-downloaded models)
`/tank/aimodels/llm/`	Legacy GGUF models (790 GB, referenced by llama-swap as `/models/`)
`/tank/aimodels/mlx/`	MLX packs (Apple-silicon formats, kept for acquisition only; nothing on this host serves them). Created 2026-09-28 for Ternary-Bonsai-2-27B
`/tank/aimodels/<model>.PROVENANCE.txt`	Provenance sits BESIDE each model dir, not inside it
`/var/lib/docker/`	Docker data (zvol `tank/docker`, ext4, since 2026-10-05)

Running stacks

Live inventory as of 2026-07-22. Each model is its own compose stack now (container vllm-<x> / llama-<x>); the vllm stack proper is just the embed/rerank/reward trio. GPUs are pinned per container via deploy.resources.reservations.devices[].device_ids.

GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):

Container	Port	Served model	Quant	Ctx
`vllm-gen` (project `gen-seat`)	8015	`qwen3.8-27b-uncensored` — the "gen" hero seat (Qwen3.8-27B Heretic-abliterated, in-house NVFP4 W4A16 + grafted MTP)	NVFP4 W4A16 (compressed-tensors)	262k
`vllm-charrp-reasoning-nvfp4`	8018	`char-rp-reasoning` (R36 reasoning RP)	NVFP4 (modelopt)	256k

GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):

⚠ This table is stale (checked 2026-09-27). Live GPU 1 residents are vllm-coder, vllm-erp-seat and vllm-meromero-rp (scriberr moved to GPU 3 on 2026-09-30 1322). intern-decision joined them on 2026-09-30, 0941 PT: :8033, 8,812 MiB at rest, 9,866 MiB peak at first; since 1330 it is capped at 14.4 GiB with MAX_TOKENS 32,768 (card peak 15,220 MiB at the limit), because scriberr left this card. See stacks/intern-decision. It replaced semif (:8032), whose container was removed and is kept as the rollback, per Prime's ruling of 2026-09-30. GPU 1 budget: nvidia-smi Free reads 6,625 MiB with intern-decision at rest. All of it is intern-decision's headroom for 32k calls (217 MiB spare at its peak). Nothing else fits on this card now. Measure nvidia-smi Free (the driver reserves 640 MiB per card) before adding anything to this card. Read the host (docker inspect … DeviceRequests), not this table. (A fixtures-only augaman instance ran here for about an hour on 2026-09-27 for a speed bench, and was then removed on Prime's call.)

Container	Port	Served model	Quant	Ctx
`vllm-granite`	8004	`granite-4.1-8b` — fleet summarizer/classifier	FP8 (compressed-tensors)	131k
`llama-charrp`	8016	`Magidonia-24B-v4.3` Q6_K — char-RP (llama.cpp)	GGUF Q6_K	—
~~`vllm-selene`~~	~~8011~~	RETIRED 2026-08-23 — lost a head-to-head against `gen` on its own judge task (see `stacks/selene/README.md`); seat downed to reclaim 17.2 GiB on GPU 1. `selene-1-mini-8b` now 404s by design; use `chat-judge`.	—	—
`vllm-reward`	8003	`Skywork-Reward-V2-Llama-3.1-8B-AWQ` — reward classifier	AWQ	16k
`vllm-embed`	8001	`Qwen3-Embedding-0.6B`	—	8k
`vllm-rerank`	8002	`Qwen3-Reranker-0.6B`	—	8k

Infra / non-GPU:

Container	Port	Notes
`dockge`	5001	Docker stack management UI
`dozzle-agent`	7007	Log agent → Dozzle hub on ana-docker
`beszel-agent`	45876	Metrics agent → Beszel hub on ana-docker

Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing it on the card with room or evicting a dormant one first.

Dormant (compose present on disk, containers stopped) — rollback / audition seats, safe to leave: mistral-medium-3.5, mistral-small-4(-heretic), ms32-24b-angel, qwen3.5-122b, qwopus3.5-122b, qwen35-vl, qwen36-vl, qwen36-27b-aeon, qwen-image-bench, vibevoice, comfyui, kokoro, vllm-qwen3.

Also on GPU 0 (non-vLLM):

Container	Port	Serves	Notes
`parakeet`	8300	Parakeet-TDT 0.6B v3 int8 (25 languages)	ASR via sherpa-onnx, LiteLLM `ext-stt` / `whisper-1`. Relocated from irv-ml1 2026-09-15. ~800 MiB. `stacks/parakeet/`.

⚠ GPU 3 is deliberately kept EMPTY (2 MiB). It is the only card that can still take a full-size seat — flash-next needs 93 GiB of 96 — and vLLM sizes its KV cache against total VRAM rather than free VRAM, so even a sub-1 GB tenant there eats into a future big seat's profiling margin. Small seats go on GPU 0, which has the most uncommitted headroom (its seats commit util 0.88; GPU 1 is at 0.975 and GPU 2 at 0.96).

GPU 3 on-demand tenant (Prime, 2026-09-27): blender (stacks/blender/). It is up only while in use (restart: "no", about 270 MiB when idle with the desktop running, 0 when down). The reserve still stands: whenever a full-size seat takes GPU 3, Blender stays down.

GPU 3 on-demand tenant (Prime, 2026-09-30 1322): scriberr (stacks/scriberr/). It holds 0 VRAM when idle and peaks at ~5.5 GB per job. Same rule as Blender: when a full-size seat claims GPU 3, Scriberr steps aside, and its planned landing spot is irv-ml1's A6000 (~32 GB free on 2026-09-30, but shared with bursty ComfyUI work; check the peaks first). It does NOT go back to GPU 1, because GPU 1's headroom now funds intern-decision's 32k-token calls (cap 14.4 GiB).

Retired:

llama-swap (former GGUF multiplexer on :9292) — replaced by dedicated per-model seats (e.g. llama-charrp); no longer running.
infinity — replaced by the vllm stack (originally vllm-qwen3, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a transformers build that knew Qwen3.
LibreChat (+ rag_api, vectordb, mongodb, meilisearch), searxng — removed from this host (searxng now on ana-docker fleet-wide).

Refresh state

scripts/refresh-server-info.sh fv-ml1

Latest snapshot: system-details.txt (regenerate as needed).

GPU allocation policy

Every seat is explicitly pinned via device_ids (no unpinned containers), and both cards run ~90% full:

GPU 0: the two heavy NVFP4 seats — vllm-gen (gen) and vllm-charrp-reasoning-nvfp4. The live serving path (near-100% util under load), ~42 + 45 GB.
GPU 1: everything else — reward, embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle between calls, ~91 GB resident.

Pin with deploy.resources.reservations.devices[].device_ids: ["<id>"] in compose. Each service caps its share with --gpu-memory-utilization; with both cards near-full, placing a new seat means freeing room (evict a dormant one) or trimming a neighbour's utilization first.

README.md Unescape Escape

fv-ml1

Network