# fv-ml1

Primary AI inference host for PFI.

## Network

- **LAN IP:** 10.251.50.54 (in-band, OS-side)
- **BMC (OOB):** 10.251.250.50 — Supermicro IPMI web UI
  at <https://10.251.250.50> (homepage card: *PFI-ANA-ML2 BMC*)
- **SSH:** standard port 22 on 10.251.50.54

## Hardware

- **Chassis:** Supermicro mid-range inferencing server (bare metal,
  NOT Dell / not the same box as sf-r630 / sfsrv-ana)
- **CPU:** AMD EPYC 9254 24-core (96 threads)
- **RAM:** 566 GB
- **GPUs:** **4x** NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (95.6 GB VRAM each = 382 GB total, cc 12.0 / sm_120, GPU 0-3) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores. ⚠ This line read "2x" until 2026-09-19; `nvidia-smi` reports four. Read the host, not the doc.
- **PSUs:** four present (PS1-PS4, all `ok`). ⚠ See § Power — on a single circuit that redundancy does not protect against the failure most likely to happen.
- **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (raidz2, 8× NVMe, 8.6 TB at `/tank`) — drive inventory below
- **OS:** Debian 13 (trixie), kernel 6.12.x
- **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2

## NVMe drive inventory (`tank`, raidz2-0) — read 2026-09-09 via `nvme-cli`

All eight are Dell Express Flash PM1725b 1.6 TB SFF (Samsung OEM), PCIe 3.0 x4 behind
a Broadcom PEX switch. Two provenance batches: the `S5CU…` six (fw 1.2.2) and the
`S47V…` pair (fw 1.2.0 / 1.2.1) with thousands of prior-life power cycles.

| dev | PCI | serial | fw | pwr-on h | pwr cycles | unsafe shut. | media err | used |
|---|---|---|---|---|---|---|---|---|
| nvme0 | 46:00.0 | S5CUNEUMB05672 | 1.2.2 | 33856 | 196 | 175 | 0 | 0% |
| nvme1 | — | S5CUNEUMB05671 | 1.2.2 | 33856 | 199 | 178 | 0 | 0% |
| nvme2 | — | S5CUNEUMB05694 | 1.2.2 | 15688 | 90 | 75 | 0 | 0% |
| nvme3 | — | S5CUNEUMB05667 | 1.2.2 | 33857 | 197 | 176 | 0 | 0% |
| nvme4 | — | S5CUNEUMB05674 | 1.2.2 | 33856 | 198 | 177 | 0 | 0% |
| nvme5 | c6:00.0 | S47VNY0K600270 | 1.2.1 | 18823 | 5357 | 5342 | 0 | 1% |
| nvme6 | — | S5CUNEUMB05697 | 1.2.2 | 15570 | 88 | 73 | 0 | 0% |
| **nvme7** | 07:00.0 (slot 0-5) | S47VNY0K600221 | 1.2.0 | 19525 | 3093 | 3083 | **2084** | 2% |

⚠ **nvme7 was ABSENT from every boot 2026-04-23 → 2026-09-05** (kernel enumerated 7
NVMes per boot; PCIe downstream port `02:04.0` had nothing on bus 07). It reappeared at
the 09-05 14:26 cold boot, the pool resilvered 638 GB, and 2 CKSUM errors landed on it
at import. While it was missing `tank` was DEGRADED, and Debian's `zfsutils-linux` cron
(`/usr/lib/zfs-linux/{scrub,trim}`) only touches pools whose health is `ONLINE`, so tank
got **no scrub and no trim from 04-12 to 09-06**. ZED's `ZED_EMAIL_ADDR=root` has no
MTA behind it, so the 4½-month degradation alerted nobody. `media_errors=2084` on
nvme7 is a lifetime counter.

**Settled by the 2026-09-09 scrub** (00:29–02:02 PT, `scrub repaired 0B in 01:32:44
with 0 errors`, then `zpool clear tank` → CKSUM 2 → 0): `media_errors` read **2084
before and 2084 after** a full 6.84 TiB verify, so the counter is prior-life
history, not an active fault, and the 2 CKSUM were the stale-block artefact of the
09-05 late resilver. **nvme7 stays in service; watch the counter at every visit and
replace on growth** (`zpool replace tank nvme7n1 <new>`; any PM1725b 1.6 TB or
larger). Slot 0-5 itself deserves a reseat / cable check at the next hands-on
visit — a bay that dropped a drive for 4½ months is the likelier fault than the
drive. Playbook: `playbooks/fv-ml1-pool-health.yaml` (idempotent; rerunning is a
no-op). ⚠ **Nothing alerts on this** — see the open follow-up in
`persistent-memory.d/2026-09-09-fv-ml1-pool-actions-done.md`.

## Power — a single 20 A circuit, shared with the FV edge router

**Confirmed by the operator 2026-09-19: fv-ml1 and the R420 running OPNsense are
the ONLY loads on a dedicated 20 A circuit.**

### The budget

| | |
|---|---|
| Circuit | 20 A @ 120 V = 2400 VA absolute |
| **Continuous limit (NEC 80%)** | **1920 W** |

A GPU inference host running for hours is a continuous load by definition, so
1920 W is the real ceiling, not 2400.

### Measured (BMC, 2026-09-19, GPUs at idle)

    ipmitool dcmi power reading
      instantaneous  390 W      min 386 W    max 461 W    avg 412 W
      sampling period 2423 s
    nvidia-smi  per GPU, all four identical:
      power.min_limit      250 W
      power.limit          275 W   <- currently ENFORCED
      power.default_limit  300 W   <- the card's stock Max-Q TGP
      power.max_limit      325 W   <- firmware ceiling, NOT an operating point
    GPU draw at time of reading: 3.6 / 3.7 / 62.7 / 7.0 W  ≈ 77 W total

⚠ **Compare the cap against 300 W, not 325 W.** The meaningful number is the
stock TGP the cards would otherwise run at; 325 W is an overclock ceiling nobody
should pick. So the 275 W cap is a **100 W** saving across four cards
(4 × 25 W) — not the 200 W you get by measuring against the firmware max. This
file said 200 W until the operator corrected it on 2026-09-19.

So the **non-GPU baseline is ~313 W** (EPYC 9254 24C/96T, 5+ drives, fans, board).

### Derived worst case

| Load | capped 275 W | stock 300 W |
|---|---|---|
| 4 GPUs | 1100 | 1200 |
| CPU + board + drives under load | ~400 | ~400 |
| **fv-ml1 subtotal** | **~1500** | **~1600** |
| R420 / OPNsense (estimate) | ~125 | ~125 |
| **Total** | **~1625** | **~1725** |
| **% of the 1920 W continuous budget** | **~85%** | **~90%** |
| Headroom | ~295 W | ~195 W |

**So the cap buys about 5 points of margin — 85% instead of 90%.**

### ⚠ What this forbids

- **Keep the 275 W caps.** Stock 300 W is not itself illegal — it lands near 90%
  of continuous — but 90% leaves nothing for the R420 being heavier than
  estimated, for PSU efficiency if the BMC reports DC, or for a warm day. The
  cap costs ~8% of GPU power headroom and buys back ~100 W of circuit margin;
  on a shared breaker feeding the site's router, that is a good trade. Same
  posture as `feedback_idle_vram_is_reserved_not_waste`: the margin is the point,
  not waste waiting to be reclaimed.
- **Never go to 325 W.** That is a firmware ceiling, not an operating point, and
  it puts the circuit around 95% of continuous.
- **Do not add a fifth GPU, or another box, on this circuit.**
- Anything new here needs a load calculation first, against 1920 W, not 2400.

### ⚠ The coupling risk, which is worse than the trip

OPNsense on the R420 **is the Fountain Valley edge**. It shares the breaker with
the thing most likely to trip it. So a GPU overload does not just reboot the
inference host — it takes the site's router with it, and with the router gone
there is no remote path in to diagnose or power-cycle anything. The failure is
correlated and it locks you out of its own recovery.

Four PSUs on fv-ml1 do not help: PSU redundancy protects against a PSU dying,
not against the circuit going away, and all four are downstream of one breaker.

Breaker trips are not hypothetical on this fleet — the ANA colo has 2026
incident history for exactly this (`docs/pfi/headscale-mesh-plan.md`: "breaker,
PSU1, WAN admin closed").

### ⚠ What is NOT measured

Stated so nobody reads the table above as more solid than it is:

1. **fv-ml1 has never been measured under real 4-GPU load.** The 461 W max above
   is a 40-minute idle-ish sample. The ~1500 W figure is derived from the caps,
   not observed.
2. **Unknown whether the BMC reports AC input or DC output.** If DC, add ~8-10%
   for PSU efficiency — about 150 W at full load, which would take the circuit
   from 86% to ~94%.
3. **The R420's draw is an estimate**, not a reading.

The cheap way to close 1 and 2 together: run all four GPUs at cap (a saturating
load), read `ipmitool dcmi power reading` at the top, and compare against a clamp
meter on the circuit. Until then, treat 1650 W as a floor.

## Key paths

| Path | Purpose |
|------|---------|
| `/opt/docker/compose/<stack>/` | Compose files |
| `/opt/docker/conf/<stack>/` | Config bind mounts |
| `/tank/aimodels/huggingface/` | HF cache (267 GB, pre-downloaded models) |
| `/tank/aimodels/llm/` | Legacy GGUF models (790 GB, referenced by llama-swap as `/models/`) |
| `/var/lib/docker/` | Docker data (on zroot) |

## Running stacks

Live inventory as of 2026-07-22. Each model is its own compose stack now
(container `vllm-<x>` / `llama-<x>`); the `vllm` stack proper is just the
embed/rerank/reward trio. GPUs are pinned per container via
`deploy.resources.reservations.devices[].device_ids`.

**GPU 0 — heavy RP / reasoning seats (~88/98 GB, hot serving path):**

| Container | Port | Served model | Quant | Ctx |
|-----------|------|--------------|-------|-----|
| `vllm-gen` (project `gen-seat`) | 8015 | `qwen3.8-27b-uncensored` — the "gen" hero seat (Qwen3.8-27B Heretic-abliterated, in-house NVFP4 W4A16 + grafted MTP) | NVFP4 W4A16 (compressed-tensors) | 262k |
| `vllm-charrp-reasoning-nvfp4` | 8018 | `char-rp-reasoning` (R36 reasoning RP) | NVFP4 (modelopt) | 256k |

**GPU 1 — light / eval / retrieval + char-RP GGUF (~91/98 GB, on-demand):**

| Container | Port | Served model | Quant | Ctx |
|-----------|------|--------------|-------|-----|
| `vllm-granite` | 8004 | `granite-4.1-8b` — fleet summarizer/classifier | FP8 (compressed-tensors) | 131k |
| `llama-charrp` | 8016 | `Magidonia-24B-v4.3` Q6_K — char-RP (llama.cpp) | GGUF Q6_K | — |
| ~~`vllm-selene`~~ | ~~8011~~ | **RETIRED 2026-08-23** — lost a head-to-head against `gen` on its own judge task (see `stacks/selene/README.md`); seat downed to reclaim 17.2 GiB on GPU 1. `selene-1-mini-8b` now 404s by design; use `chat-judge`. | — | — |
| `vllm-reward` | 8003 | `Skywork-Reward-V2-Llama-3.1-8B-AWQ` — reward classifier | AWQ | 16k |
| `vllm-embed` | 8001 | `Qwen3-Embedding-0.6B` | — | 8k |
| `vllm-rerank` | 8002 | `Qwen3-Reranker-0.6B` | — | 8k |

**Infra / non-GPU:**

| Container | Port | Notes |
|-----------|------|-------|
| `dockge` | 5001 | Docker stack management UI |
| `dozzle-agent` | 7007 | Log agent → Dozzle hub on ana-docker |
| `beszel-agent` | 45876 | Metrics agent → Beszel hub on ana-docker |

Both cards run near-full (~7–10 GB headroom each) — adding a seat means placing
it on the card with room or evicting a dormant one first.

**Dormant (compose present on disk, containers stopped)** — rollback / audition
seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`,
`ms32-24b-angel`, `qwen3.5-122b`, `qwopus3.5-122b`, `qwen35-vl`, `qwen36-vl`,
`qwen36-27b-aeon`, `qwen-image-bench`, `vibevoice`, `comfyui`, `kokoro`,
`vllm-qwen3`.

**Also on GPU 0 (non-vLLM):**

| Container | Port | Serves | Notes |
|-----------|------|--------|-------|
| `parakeet` | 8300 | Parakeet-TDT 0.6B v3 int8 (25 languages) | ASR via sherpa-onnx, LiteLLM `ext-stt` / `whisper-1`. Relocated from irv-ml1 2026-09-15. ~800 MiB. `stacks/parakeet/`. |

⚠ **GPU 3 is deliberately kept EMPTY (2 MiB).** It is the only card that can still
take a full-size seat — `flash-next` needs 93 GiB of 96 — and vLLM sizes its KV
cache against *total* VRAM rather than free VRAM, so even a sub-1 GB tenant there
eats into a future big seat's profiling margin. Small seats go on GPU 0, which has
the most uncommitted headroom (its seats commit util 0.88; GPU 1 is at 0.975 and
GPU 2 at 0.96).

**Retired:**
- `llama-swap` (former GGUF multiplexer on :9292) — replaced by dedicated
  per-model seats (e.g. `llama-charrp`); no longer running.
- `infinity` — replaced by the `vllm` stack (originally `vllm-qwen3`, renamed 2026-05-13 when the stack expanded beyond Qwen3) after the upstream Infinity image stopped shipping a `transformers` build that knew Qwen3.
- `LibreChat (+ rag_api, vectordb, mongodb, meilisearch)`, `searxng` — removed from this host (searxng now on ana-docker fleet-wide).

## Refresh state

```bash
scripts/refresh-server-info.sh fv-ml1
```

Latest snapshot: `system-details.txt` (regenerate as needed).

## GPU allocation policy

Every seat is explicitly pinned via `device_ids` (no unpinned containers), and
both cards run ~90% full:

- **GPU 0:** the two heavy NVFP4 seats — `vllm-gen` (gen) and
  `vllm-charrp-reasoning-nvfp4`. The live serving path (near-100% util under
  load), ~42 + 45 GB.
- **GPU 1:** everything else — reward,
  embed, rerank, and the Magidonia char-RP GGUF seat. Bursty/on-demand, idle
  between calls, ~91 GB resident.

Pin with `deploy.resources.reservations.devices[].device_ids: ["<id>"]` in
compose. Each service caps its share with `--gpu-memory-utilization`; with both
cards near-full, placing a new seat means freeing room (evict a dormant one) or
trimming a neighbour's utilization first.