docs(fv-ml1): record the 20 A circuit, its real ceiling, and what it forbids
Operator confirmed 2026-09-19 that fv-ml1 and the R420 running OPNsense are the only loads on a dedicated 20 A circuit. The governing number is 1920 W, not 2400: a GPU inference host running for hours is a continuous load, so NEC's 80% rule applies. Worst case lands at ~1625 W with the current caps -- about 85% of budget. Measured via the BMC rather than assumed: 390 W instantaneous, 461 W max over a 2423 s sample, with GPUs at idle, giving a ~313 W non-GPU baseline. Compare the GPU caps against the 300 W stock TGP, NOT the 325 W firmware ceiling. The operator corrected this: 275 W across four cards saves 100 W, not the 200 W you get by measuring against a number nobody would ever run at. Stock 300 W would put the circuit near 90%, which is not illegal but leaves nothing for a heavier R420, PSU efficiency, or a warm day. Keep the caps. The coupling matters more than the trip. OPNsense IS the Fountain Valley edge and shares the breaker with the thing most likely to trip it, so an overload takes the router with it and removes the remote path needed to diagnose or power-cycle anything. fv-ml1's four PSUs do not help -- PSU redundancy protects against a PSU dying, not against the circuit going away. Three things are explicitly NOT measured and the file says so: fv-ml1 under real 4-GPU load, whether the BMC reports AC input or DC output, and the R420's actual draw. Treat 1625 W as a floor. Also corrects the hardware section, which claimed 2x GPUs. nvidia-smi reports four.
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-19 ~07:15 PT (⭐ **the ops log is BUILT** and caught its own tooling's failure during a live bus upgrade. ⭐ althing **3.7.0** deployed to the post office + nh3-extdev, forseti-verified. ⭐ The backup alarm had been unable to notify ANYONE since the 2026-08-28 v3 cutover — fixed, plus coverage-awareness and vzdump task-status checking. ⭐ **elway** bug: root-requiring `when:`/`creates:` evaluated unprivileged → silent SKIP → playbooks report OK while changing nothing. ⭐ Commit attribution across three agents now recorded by handle. ⚠ lv-mccarthy's run outcome STILL UNVERIFIED.)_
|
||||
_Last updated: 2026-09-19 ~08:55 PT (⭐ **FV power budget recorded** — dedicated 20 A circuit shared by fv-ml1 + the OPNsense R420, ~85% of the 1920 W continuous ceiling with the 275 W GPU caps in place; KEEP THEM, and note the router shares the breaker with the thing most likely to trip it. ⭐ The ops log is BUILT and caught its own tooling's failure during a live bus upgrade. ⭐ althing **3.7.0** deployed, forseti-verified. ⭐ The backup alarm had been unable to notify ANYONE since 2026-08-28 — fixed, plus coverage-awareness and vzdump task-status checking. ⚠ lv-mccarthy's run outcome STILL UNVERIFIED.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -221,6 +221,8 @@ nothing touched. Full context in the 09-17 Recent decisions entries.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-19]` ⭐⭐ **FV is a dedicated 20 A circuit carrying fv-ml1 AND the R420 running OPNsense, and nothing else (operator-confirmed).** The ceiling is **1920 W**, not 2400 — a GPU host running for hours is a continuous load, so NEC's 80% rule governs. Measured: BMC `dcmi power reading` 390 W instantaneous / 461 W max over 2423 s **at idle GPUs**, so the non-GPU baseline is ~313 W (EPYC 9254 24C/96T, 5+ drives, 4 PSUs). GPUs are capped **275 W** each against a **300 W** stock TGP (`power.default_limit`) — a 100 W saving across four cards, NOT the 200 W you get by measuring against the 325 W firmware ceiling; the operator corrected me on exactly that. Derived worst case **~1625 W capped (85%)** vs **~1725 W at stock (90%)**. ⚠ **Keep the caps** — 90% leaves nothing for a heavier-than-estimated R420, PSU efficiency, or a warm day. ⚠⚠ **The coupling is worse than the trip:** OPNsense IS the FV edge and shares the breaker with the thing most likely to trip it, so an overload takes the router with it and removes the remote path needed to recover. Four PSUs do not help — they are all downstream of one breaker. ⚠ **NOT measured:** fv-ml1 under real 4-GPU load (the 461 W max is idle-ish), whether the BMC reports AC or DC (±10% ≈ 150 W at load), and the R420's actual draw. Recorded in `servers/fv-ml1/README.md` § Power. ⚠ Also fixed there: the README claimed **2x** GPUs; `nvidia-smi` reports **four**.
|
||||
|
||||
- `[2026-09-19]` ⭐⭐ **althing 3.7.0 rolled out to both infra-ops surfaces — and the rollout broke the claim tooling built that morning.** Post office on nh3-docker (built from `althing@6db955f`, manifest `sha256:df0709b3`, **10.9 s** recreate, volume preserved) and the nh3-extdev system wheel + herald. forseti independently verified both. ⚠ Everything CONTENT-verified, never tag-verified: `postbox --version` inside the image before the push and inside the running container after; data continuity proven by reading forseti's own message back out of the running 3.7.0 store. ⚠ **`deploy-stack.sh` rsyncs with `--delete`, so the host-side `.bak-<version>` compose convention is GONE** — it converges rather than accretes; rollback is git history + the retained 3.6.3 registry digest. ⚠ **The claim bug:** a 45-minute operation claim was refreshed *and then released* by deploy-stack.sh's exit trap, silently dropping the protection mid-rollout. Fixed 3e7d3a3 — `ops-log claim` exits **10** when the claim is already the caller's and leaves the holder file UNTOUCHED (a refresh would overwrite the reason and TTL the original claimant chose).
|
||||
|
||||
- `[2026-09-19]` ⭐⭐⭐ **Three agents commit as one git author, and closing that gap took three instruments to get right.** An unattributable commit (`e43e262`) appeared in the push set between two of mine — unidentifiable from git (all commits are Vuong Hoang by convention), from the ops log (host changes only) or from the bus. Turned out to be infra-hermes; forseti disclaimed it and the files' own `~/.hermes/hermes-agent` imports gave it away. Three fixes, each exposing the next: **(1)** `0dc8e90` a post-commit hook recording the althing handle; **(2)** `4e778ae` its fallback said `lkraven@nh3-dev`, which *reads like an answer while being the absence of one* — now `unattributed(login)`, and handles never contain `@` or parens; **(3)** `9141a41` the hook was **appended** behind graphify's **eight `exit 0` paths**, so a `graphify-out/`-only or empty commit could NEVER be recorded. ⚠ **Root cause was control flow, not the lock race proposed** — see `feedback_control_flow_before_concurrency`. Attribution must run FIRST, never behind another hook's interestingness filter. ⚠ Also: `ALTHING_HANDLE` lived only in `althing-infra-hermes-seat-run.sh`, not in `hermes-gateway.service` / the pump unit — hermes fixed it with drop-ins. ⚠ **`systemctl show -p Environment` reports the CONFIGURED env, not the running process's** — it read correct while `/proc/<pid>/environ` did not have the variable at all. Check the process, never the config.
|
||||
|
||||
@@ -15,7 +15,8 @@ Primary AI inference host for PFI.
|
||||
NOT Dell / not the same box as sf-r630 / sfsrv-ana)
|
||||
- **CPU:** AMD EPYC 9254 24-core (96 threads)
|
||||
- **RAM:** 566 GB
|
||||
- **GPUs:** 2x NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (96 GB VRAM each, cc 12.0 / sm_120, GPU 0 and GPU 1) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores and doubles VRAM.
|
||||
- **GPUs:** **4x** NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition (95.6 GB VRAM each = 382 GB total, cc 12.0 / sm_120, GPU 0-3) — upgraded 2026-06 from 2x RTX 6000 Ada (46 GB, cc 8.9). Blackwell adds native FP4 (NVFP4) tensor cores. ⚠ This line read "2x" until 2026-09-19; `nvidia-smi` reports four. Read the host, not the doc.
|
||||
- **PSUs:** four present (PS1-PS4, all `ok`). ⚠ See § Power — on a single circuit that redundancy does not protect against the failure most likely to happen.
|
||||
- **Storage:** ZFS `zroot` (434 GB root) + `tank` pool (raidz2, 8× NVMe, 8.6 TB at `/tank`) — drive inventory below
|
||||
- **OS:** Debian 13 (trixie), kernel 6.12.x
|
||||
- **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
|
||||
@@ -58,6 +59,100 @@ drive. Playbook: `playbooks/fv-ml1-pool-health.yaml` (idempotent; rerunning is a
|
||||
no-op). ⚠ **Nothing alerts on this** — see the open follow-up in
|
||||
`persistent-memory.d/2026-09-09-fv-ml1-pool-actions-done.md`.
|
||||
|
||||
## Power — a single 20 A circuit, shared with the FV edge router
|
||||
|
||||
**Confirmed by the operator 2026-09-19: fv-ml1 and the R420 running OPNsense are
|
||||
the ONLY loads on a dedicated 20 A circuit.**
|
||||
|
||||
### The budget
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Circuit | 20 A @ 120 V = 2400 VA absolute |
|
||||
| **Continuous limit (NEC 80%)** | **1920 W** |
|
||||
|
||||
A GPU inference host running for hours is a continuous load by definition, so
|
||||
1920 W is the real ceiling, not 2400.
|
||||
|
||||
### Measured (BMC, 2026-09-19, GPUs at idle)
|
||||
|
||||
ipmitool dcmi power reading
|
||||
instantaneous 390 W min 386 W max 461 W avg 412 W
|
||||
sampling period 2423 s
|
||||
nvidia-smi per GPU, all four identical:
|
||||
power.min_limit 250 W
|
||||
power.limit 275 W <- currently ENFORCED
|
||||
power.default_limit 300 W <- the card's stock Max-Q TGP
|
||||
power.max_limit 325 W <- firmware ceiling, NOT an operating point
|
||||
GPU draw at time of reading: 3.6 / 3.7 / 62.7 / 7.0 W ≈ 77 W total
|
||||
|
||||
⚠ **Compare the cap against 300 W, not 325 W.** The meaningful number is the
|
||||
stock TGP the cards would otherwise run at; 325 W is an overclock ceiling nobody
|
||||
should pick. So the 275 W cap is a **100 W** saving across four cards
|
||||
(4 × 25 W) — not the 200 W you get by measuring against the firmware max. This
|
||||
file said 200 W until the operator corrected it on 2026-09-19.
|
||||
|
||||
So the **non-GPU baseline is ~313 W** (EPYC 9254 24C/96T, 5+ drives, fans, board).
|
||||
|
||||
### Derived worst case
|
||||
|
||||
| Load | capped 275 W | stock 300 W |
|
||||
|---|---|---|
|
||||
| 4 GPUs | 1100 | 1200 |
|
||||
| CPU + board + drives under load | ~400 | ~400 |
|
||||
| **fv-ml1 subtotal** | **~1500** | **~1600** |
|
||||
| R420 / OPNsense (estimate) | ~125 | ~125 |
|
||||
| **Total** | **~1625** | **~1725** |
|
||||
| **% of the 1920 W continuous budget** | **~85%** | **~90%** |
|
||||
| Headroom | ~295 W | ~195 W |
|
||||
|
||||
**So the cap buys about 5 points of margin — 85% instead of 90%.**
|
||||
|
||||
### ⚠ What this forbids
|
||||
|
||||
- **Keep the 275 W caps.** Stock 300 W is not itself illegal — it lands near 90%
|
||||
of continuous — but 90% leaves nothing for the R420 being heavier than
|
||||
estimated, for PSU efficiency if the BMC reports DC, or for a warm day. The
|
||||
cap costs ~8% of GPU power headroom and buys back ~100 W of circuit margin;
|
||||
on a shared breaker feeding the site's router, that is a good trade. Same
|
||||
posture as `feedback_idle_vram_is_reserved_not_waste`: the margin is the point,
|
||||
not waste waiting to be reclaimed.
|
||||
- **Never go to 325 W.** That is a firmware ceiling, not an operating point, and
|
||||
it puts the circuit around 95% of continuous.
|
||||
- **Do not add a fifth GPU, or another box, on this circuit.**
|
||||
- Anything new here needs a load calculation first, against 1920 W, not 2400.
|
||||
|
||||
### ⚠ The coupling risk, which is worse than the trip
|
||||
|
||||
OPNsense on the R420 **is the Fountain Valley edge**. It shares the breaker with
|
||||
the thing most likely to trip it. So a GPU overload does not just reboot the
|
||||
inference host — it takes the site's router with it, and with the router gone
|
||||
there is no remote path in to diagnose or power-cycle anything. The failure is
|
||||
correlated and it locks you out of its own recovery.
|
||||
|
||||
Four PSUs on fv-ml1 do not help: PSU redundancy protects against a PSU dying,
|
||||
not against the circuit going away, and all four are downstream of one breaker.
|
||||
|
||||
Breaker trips are not hypothetical on this fleet — the ANA colo has 2026
|
||||
incident history for exactly this (`docs/pfi/headscale-mesh-plan.md`: "breaker,
|
||||
PSU1, WAN admin closed").
|
||||
|
||||
### ⚠ What is NOT measured
|
||||
|
||||
Stated so nobody reads the table above as more solid than it is:
|
||||
|
||||
1. **fv-ml1 has never been measured under real 4-GPU load.** The 461 W max above
|
||||
is a 40-minute idle-ish sample. The ~1500 W figure is derived from the caps,
|
||||
not observed.
|
||||
2. **Unknown whether the BMC reports AC input or DC output.** If DC, add ~8-10%
|
||||
for PSU efficiency — about 150 W at full load, which would take the circuit
|
||||
from 86% to ~94%.
|
||||
3. **The R420's draw is an estimate**, not a reading.
|
||||
|
||||
The cheap way to close 1 and 2 together: run all four GPUs at cap (a saturating
|
||||
load), read `ipmitool dcmi power reading` at the top, and compare against a clamp
|
||||
meter on the circuit. Until then, treat 1650 W as a floor.
|
||||
|
||||
## Key paths
|
||||
|
||||
| Path | Purpose |
|
||||
|
||||
Reference in New Issue
Block a user