docs(fv-ml1): record the 20 A circuit, its real ceiling, and what it forbids
Operator confirmed 2026-09-19 that fv-ml1 and the R420 running OPNsense are the only loads on a dedicated 20 A circuit. The governing number is 1920 W, not 2400: a GPU inference host running for hours is a continuous load, so NEC's 80% rule applies. Worst case lands at ~1625 W with the current caps -- about 85% of budget. Measured via the BMC rather than assumed: 390 W instantaneous, 461 W max over a 2423 s sample, with GPUs at idle, giving a ~313 W non-GPU baseline. Compare the GPU caps against the 300 W stock TGP, NOT the 325 W firmware ceiling. The operator corrected this: 275 W across four cards saves 100 W, not the 200 W you get by measuring against a number nobody would ever run at. Stock 300 W would put the circuit near 90%, which is not illegal but leaves nothing for a heavier R420, PSU efficiency, or a warm day. Keep the caps. The coupling matters more than the trip. OPNsense IS the Fountain Valley edge and shares the breaker with the thing most likely to trip it, so an overload takes the router with it and removes the remote path needed to diagnose or power-cycle anything. fv-ml1's four PSUs do not help -- PSU redundancy protects against a PSU dying, not against the circuit going away. Three things are explicitly NOT measured and the file says so: fv-ml1 under real 4-GPU load, whether the BMC reports AC input or DC output, and the R420's actual draw. Treat 1625 W as a floor. Also corrects the hardware section, which claimed 2x GPUs. nvidia-smi reports four.
This commit is contained in:
@@ -15,7 +15,8 @@ Primary AI inference host for PFI.
|
||||
NOT Dell / not the same box as sf-r630 / sfsrv-ana)
|
||||
- **CPU:** AMD EPYC 9254 24-core (96 threads)
|
||||
- **RAM:** 566 GB
|
||||
- **GPUs:** 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
|
||||
- **GPUs:** **4x** NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (95.6 GB VRAM each = 382 GB total, cc 12.0 / sm_120, GPU 0-3) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores. ⚠ This line read "2x" until 2026-09-19; `nvidia-smi` reports four. Read the host, not the doc.
|
||||
- **PSUs:** four present (PS1-PS4, all `ok`). ⚠ See § Power — on a single circuit that redundancy does not protect against the failure most likely to happen.
|
||||
- **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (raidz2, 8× NVMe, 8.6 TB at `/tank`) — drive inventory below
|
||||
- **OS:** Debian 13 (trixie), kernel 6.12.x
|
||||
- **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
|
||||
@@ -58,6 +59,100 @@ drive. Playbook: `playbooks/fv-ml1-pool-health.yaml` (idempotent; rerunning is a
|
||||
no-op). ⚠ **Nothing alerts on this** — see the open follow-up in
|
||||
`persistent-memory.d/2026-09-09-fv-ml1-pool-actions-done.md`.
|
||||
|
||||
## Power — a single 20 A circuit, shared with the FV edge router
|
||||
|
||||
**Confirmed by the operator 2026-09-19: fv-ml1 and the R420 running OPNsense are
|
||||
the ONLY loads on a dedicated 20 A circuit.**
|
||||
|
||||
### The budget
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Circuit | 20 A @ 120 V = 2400 VA absolute |
|
||||
| **Continuous limit (NEC 80%)** | **1920 W** |
|
||||
|
||||
A GPU inference host running for hours is a continuous load by definition, so
|
||||
1920 W is the real ceiling, not 2400.
|
||||
|
||||
### Measured (BMC, 2026-09-19, GPUs at idle)
|
||||
|
||||
ipmitool dcmi power reading
|
||||
instantaneous 390 W min 386 W max 461 W avg 412 W
|
||||
sampling period 2423 s
|
||||
nvidia-smi per GPU, all four identical:
|
||||
power.min_limit 250 W
|
||||
power.limit 275 W <- currently ENFORCED
|
||||
power.default_limit 300 W <- the card's stock Max-Q TGP
|
||||
power.max_limit 325 W <- firmware ceiling, NOT an operating point
|
||||
GPU draw at time of reading: 3.6 / 3.7 / 62.7 / 7.0 W ≈ 77 W total
|
||||
|
||||
⚠ **Compare the cap against 300 W, not 325 W.** The meaningful number is the
|
||||
stock TGP the cards would otherwise run at; 325 W is an overclock ceiling nobody
|
||||
should pick. So the 275 W cap is a **100 W** saving across four cards
|
||||
(4 × 25 W) — not the 200 W you get by measuring against the firmware max. This
|
||||
file said 200 W until the operator corrected it on 2026-09-19.
|
||||
|
||||
So the **non-GPU baseline is ~313 W** (EPYC 9254 24C/96T, 5+ drives, fans, board).
|
||||
|
||||
### Derived worst case
|
||||
|
||||
| Load | capped 275 W | stock 300 W |
|
||||
|---|---|---|
|
||||
| 4 GPUs | 1100 | 1200 |
|
||||
| CPU + board + drives under load | ~400 | ~400 |
|
||||
| **fv-ml1 subtotal** | **~1500** | **~1600** |
|
||||
| R420 / OPNsense (estimate) | ~125 | ~125 |
|
||||
| **Total** | **~1625** | **~1725** |
|
||||
| **% of the 1920 W continuous budget** | **~85%** | **~90%** |
|
||||
| Headroom | ~295 W | ~195 W |
|
||||
|
||||
**So the cap buys about 5 points of margin — 85% instead of 90%.**
|
||||
|
||||
### ⚠ What this forbids
|
||||
|
||||
- **Keep the 275 W caps.** Stock 300 W is not itself illegal — it lands near 90%
|
||||
of continuous — but 90% leaves nothing for the R420 being heavier than
|
||||
estimated, for PSU efficiency if the BMC reports DC, or for a warm day. The
|
||||
cap costs ~8% of GPU power headroom and buys back ~100 W of circuit margin;
|
||||
on a shared breaker feeding the site's router, that is a good trade. Same
|
||||
posture as `feedback_idle_vram_is_reserved_not_waste`: the margin is the point,
|
||||
not waste waiting to be reclaimed.
|
||||
- **Never go to 325 W.** That is a firmware ceiling, not an operating point, and
|
||||
it puts the circuit around 95% of continuous.
|
||||
- **Do not add a fifth GPU, or another box, on this circuit.**
|
||||
- Anything new here needs a load calculation first, against 1920 W, not 2400.
|
||||
|
||||
### ⚠ The coupling risk, which is worse than the trip
|
||||
|
||||
OPNsense on the R420 **is the Fountain Valley edge**. It shares the breaker with
|
||||
the thing most likely to trip it. So a GPU overload does not just reboot the
|
||||
inference host — it takes the site's router with it, and with the router gone
|
||||
there is no remote path in to diagnose or power-cycle anything. The failure is
|
||||
correlated and it locks you out of its own recovery.
|
||||
|
||||
Four PSUs on fv-ml1 do not help: PSU redundancy protects against a PSU dying,
|
||||
not against the circuit going away, and all four are downstream of one breaker.
|
||||
|
||||
Breaker trips are not hypothetical on this fleet — the ANA colo has 2026
|
||||
incident history for exactly this (`docs/pfi/headscale-mesh-plan.md`: "breaker,
|
||||
PSU1, WAN admin closed").
|
||||
|
||||
### ⚠ What is NOT measured
|
||||
|
||||
Stated so nobody reads the table above as more solid than it is:
|
||||
|
||||
1. **fv-ml1 has never been measured under real 4-GPU load.** The 461 W max above
|
||||
is a 40-minute idle-ish sample. The ~1500 W figure is derived from the caps,
|
||||
not observed.
|
||||
2. **Unknown whether the BMC reports AC input or DC output.** If DC, add ~8-10%
|
||||
for PSU efficiency — about 150 W at full load, which would take the circuit
|
||||
from 86% to ~94%.
|
||||
3. **The R420's draw is an estimate**, not a reading.
|
||||
|
||||
The cheap way to close 1 and 2 together: run all four GPUs at cap (a saturating
|
||||
load), read `ipmitool dcmi power reading` at the top, and compare against a clamp
|
||||
meter on the circuit. Until then, treat 1650 W as a floor.
|
||||
|
||||
## Key paths
|
||||
|
||||
| Path | Purpose |
|
||||
|
||||
Reference in New Issue
Block a user