docs(fv-ml1): record the 20 A circuit, its real ceiling, and what it forbids

Operator confirmed 2026-09-19 that fv-ml1 and the R420 running OPNsense are the
only loads on a dedicated 20 A circuit.

The governing number is 1920 W, not 2400: a GPU inference host running for hours
is a continuous load, so NEC's 80% rule applies. Worst case lands at ~1625 W
with the current caps -- about 85% of budget.

Measured via the BMC rather than assumed: 390 W instantaneous, 461 W max over a
2423 s sample, with GPUs at idle, giving a ~313 W non-GPU baseline.

Compare the GPU caps against the 300 W stock TGP, NOT the 325 W firmware
ceiling. The operator corrected this: 275 W across four cards saves 100 W, not
the 200 W you get by measuring against a number nobody would ever run at. Stock
300 W would put the circuit near 90%, which is not illegal but leaves nothing
for a heavier R420, PSU efficiency, or a warm day. Keep the caps.

The coupling matters more than the trip. OPNsense IS the Fountain Valley edge
and shares the breaker with the thing most likely to trip it, so an overload
takes the router with it and removes the remote path needed to diagnose or
power-cycle anything. fv-ml1's four PSUs do not help -- PSU redundancy protects
against a PSU dying, not against the circuit going away.

Three things are explicitly NOT measured and the file says so: fv-ml1 under real
4-GPU load, whether the BMC reports AC input or DC output, and the R420's actual
draw. Treat 1625 W as a floor.

Also corrects the hardware section, which claimed 2x GPUs. nvidia-smi reports
four.
This commit is contained in:
vh
2026-09-19 15:37:24 -07:00
parent cebffae56d
commit 8305145ce1
2 changed files with 99 additions and 2 deletions
+3 -1
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-19 ~07:15 PT (⭐ **the ops log is BUILT** and caught its own tooling's failure during a live bus upgrade. ⭐ althing **3.7.0** deployed to the post office + nh3-extdev, forseti-verified. ⭐ The backup alarm had been unable to notify ANYONE since the 2026-08-28 v3 cutover — fixed, plus coverage-awareness and vzdump task-status checking. ⭐ **elway** bug: root-requiring `when:`/`creates:` evaluated unprivileged → silent SKIP → playbooks report OK while changing nothing. ⭐ Commit attribution across three agents now recorded by handle. ⚠ lv-mccarthy's run outcome STILL UNVERIFIED.)_
_Last updated: 2026-09-19 ~08:55 PT (⭐ **FV power budget recorded** — dedicated 20 A circuit shared by fv-ml1 + the OPNsense R420, ~85% of the 1920 W continuous ceiling with the 275 W GPU caps in place; KEEP THEM, and note the router shares the breaker with the thing most likely to trip it. ⭐ The ops log is BUILT and caught its own tooling's failure during a live bus upgrade. ⭐ althing **3.7.0** deployed, forseti-verified. ⭐ The backup alarm had been unable to notify ANYONE since 2026-08-28 — fixed, plus coverage-awareness and vzdump task-status checking. ⚠ lv-mccarthy's run outcome STILL UNVERIFIED.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -221,6 +221,8 @@ nothing touched. Full context in the 09-17 Recent decisions entries.
## Recent decisions
- `[2026-09-19]` ⭐⭐ **FV is a dedicated 20 A circuit carrying fv-ml1 AND the R420 running OPNsense, and nothing else (operator-confirmed).** The ceiling is **1920 W**, not 2400 — a GPU host running for hours is a continuous load, so NEC's 80% rule governs. Measured: BMC `dcmi power reading` 390 W instantaneous / 461 W max over 2423 s **at idle GPUs**, so the non-GPU baseline is ~313 W (EPYC 9254 24C/96T, 5+ drives, 4 PSUs). GPUs are capped **275 W** each against a **300 W** stock TGP (`power.default_limit`) — a 100 W saving across four cards, NOT the 200 W you get by measuring against the 325 W firmware ceiling; the operator corrected me on exactly that. Derived worst case **~1625 W capped (85%)** vs **~1725 W at stock (90%)**. ⚠ **Keep the caps** — 90% leaves nothing for a heavier-than-estimated R420, PSU efficiency, or a warm day. ⚠⚠ **The coupling is worse than the trip:** OPNsense IS the FV edge and shares the breaker with the thing most likely to trip it, so an overload takes the router with it and removes the remote path needed to recover. Four PSUs do not help — they are all downstream of one breaker. ⚠ **NOT measured:** fv-ml1 under real 4-GPU load (the 461 W max is idle-ish), whether the BMC reports AC or DC (±10% ≈ 150 W at load), and the R420's actual draw. Recorded in `servers/fv-ml1/README.md` § Power. ⚠ Also fixed there: the README claimed **2x** GPUs; `nvidia-smi` reports **four**.
- `[2026-09-19]` ⭐⭐ **althing 3.7.0 rolled out to both infra-ops surfaces — and the rollout broke the claim tooling built that morning.** Post office on nh3-docker (built from `althing@6db955f`, manifest `sha256:df0709b3`, **10.9 s** recreate, volume preserved) and the nh3-extdev system wheel + herald. forseti independently verified both. ⚠ Everything CONTENT-verified, never tag-verified: `postbox --version` inside the image before the push and inside the running container after; data continuity proven by reading forseti's own message back out of the running 3.7.0 store. ⚠ **`deploy-stack.sh` rsyncs with `--delete`, so the host-side `.bak-<version>` compose convention is GONE** — it converges rather than accretes; rollback is git history + the retained 3.6.3 registry digest. ⚠ **The claim bug:** a 45-minute operation claim was refreshed *and then released* by deploy-stack.sh's exit trap, silently dropping the protection mid-rollout. Fixed 3e7d3a3 — `ops-log claim` exits **10** when the claim is already the caller's and leaves the holder file UNTOUCHED (a refresh would overwrite the reason and TTL the original claimant chose).
- `[2026-09-19]` ⭐⭐⭐ **Three agents commit as one git author, and closing that gap took three instruments to get right.** An unattributable commit (`e43e262`) appeared in the push set between two of mine — unidentifiable from git (all commits are Vuong Hoang by convention), from the ops log (host changes only) or from the bus. Turned out to be infra-hermes; forseti disclaimed it and the files' own `~/.hermes/hermes-agent` imports gave it away. Three fixes, each exposing the next: **(1)** `0dc8e90` a post-commit hook recording the althing handle; **(2)** `4e778ae` its fallback said `lkraven@nh3-dev`, which *reads like an answer while being the absence of one* — now `unattributed(login)`, and handles never contain `@` or parens; **(3)** `9141a41` the hook was **appended** behind graphify's **eight `exit 0` paths**, so a `graphify-out/`-only or empty commit could NEVER be recorded. ⚠ **Root cause was control flow, not the lock race proposed** — see `feedback_control_flow_before_concurrency`. Attribution must run FIRST, never behind another hook's interestingness filter. ⚠ Also: `ALTHING_HANDLE` lived only in `althing-infra-hermes-seat-run.sh`, not in `hermes-gateway.service` / the pump unit — hermes fixed it with drop-ins. ⚠ **`systemctl show -p Environment` reports the CONFIGURED env, not the running process's** — it read correct while `/proc/<pid>/environ` did not have the variable at all. Check the process, never the config.