caps: fv-ml1 250W / ana-ml3 200W — and nvidia-smi -pl caps BOARD power, not wall power

Operator set fv-ml1 at 250 W per card (83% of TGP, ~5% throughput) and ana-ml3 at 200 W
(67%, ~10-15%). Records the term that decides whether 250 W actually clears a 15 A feed,
because it is easy to drop: a power limit bounds BOARD power, and the wall sees that
divided by PSU efficiency.

Four cards at 250 W is 1000 W of board; add 180-300 W of host components and divide by
~0.90 and the plug sees ~1310-1445 W, against a 15 A circuit's 1440 W NEC continuous
derating -- an inference box serving all day being a continuous load. So 250 W lands ON
the limit rather than under it, where 200 W would give ~1090-1220 W with real margin.
The deciding term is the host draw, which is still an estimate, so the procedure is: set
250 W, verify at the plug under four-card load, fall back to 200 W if it reads near
1440 W. A cap is a claim; the ammeter is the verification.

Two consequences recorded alongside. Caps bound sustained draw and not transients -- the
enforcement window is short but not instantaneous -- and while a breaker's
thermal-magnetic curve forgives brief overload, a UPS's overload protection does not. So
250 W implicitly commits the fv-ml1 chassis to the PDU rather than behind the 1500 VA
unit, which it exceeds even capped. And ana-ml3's 200 W across only two cards is
deliberately conservative at 400 W total, relaxable if Anaheim's measured headroom beats
its trip history.

docs/runbooks/fv-site-dark-20260913.md

@@ -358,8 +358,10 @@ probably a 20 A circuit -- but size it from `power.log`, not from a spec sheet.
 > | **fv-ml1** | 4x **Blackwell** RTX PRO 6000 **Max-Q** | **300 W** (Max-Q is the reduced-TGP SKU; the Workstation Edition is 600 W) | 4x96 = 391 GB | deployed, currently dark |
 > | **ana-ml3** | 2x **Ada Generation** RTX 6000 | **300 W** | 2x48 = 96 GB | **NOT YET DEPLOYED** |
 >
-> The 200 W cap directive applies to **ana-ml3**. It very likely wants applying to
-> fv-ml1 as well — see the four-card circuit arithmetic below.
+> **CAPS (operator, 2026-09-13):** **fv-ml1 → 250 W** per card (83% of TGP, ~5%
+> throughput). **ana-ml3 → 200 W** per card (67% of TGP, ~10-15%). See the plug-side
+> arithmetic below — 250 W is marginal on a 15 A circuit once PSU efficiency is counted,
+> and must be verified with the ammeter rather than assumed.
 
 The generalised lesson from this outage: **decide the power envelope first and size the
 cards into it**, rather than installing cards and discovering the constraint by tripping
@@ -408,10 +410,45 @@ guessed at, and idle-at-the-plug with all seats down measures it directly.
         4 x 300 W card  +  ~300 W host     ~  1500 W
         15 A circuit, 80% continuous       =  1440 W
 
-**Four cards uncapped is marginal on a 15 A circuit with no UPS in the path at all.** So
-capping belongs at **fv-ml1 too**, not only ana-ml3. If the four-card ammeter reading
-confirms it, fv-ml1 needs a per-card cap or a 20 A feed before anyone loads all four
-again — and note that today's incident only ever had TWO cards working.
+**Four cards uncapped is marginal on a 15 A circuit with no UPS in the path at all**, so
+capping belongs at fv-ml1 too. Note today's incident only ever had TWO of the four cards
+working; nobody has loaded all four.
+
+### ⚠⚠⚠ AND `nvidia-smi -pl` CAPS BOARD POWER, NOT WALL POWER
+
+This term is easy to drop and it is the one that decides whether 250 W clears a 15 A
+feed:
+
+        4 x 250 W board                           = 1000 W
+        host components (board, 566 GB RAM,
+             drives, fans)                        ~  180-300 W
+                                                     -----------
+        component total                           ~ 1180-1300 W
+        / PSU efficiency (~0.90)  -> AT THE PLUG  ~ 1310-1445 W
+
+        15 A circuit, NEC 80% continuous derating =  1440 W
+
+**250 W lands ON the limit, not under it.** An inference box serving all day is a
+continuous load, so 1440 W is the design figure, not 1800.
+
+At **200 W** the same arithmetic gives **~1090-1220 W at the plug** — comfortable, 200+ W
+of margin.
+
+⚠ **So 250 W is PROBABLY fine and POSSIBLY not, and the deciding term is the host draw,
+which is still an estimate.** Procedure: set 250 W, then **verify at the plug under
+four-card load** before calling it done; fall back to 200 W if the reading comes in near
+1440 W. A cap is a claim; the ammeter is the verification.
+
+⚠ **Power limits cap SUSTAINED draw, not transients.** The enforcement window is short
+but not instantaneous, so four cards at 250 W can momentarily exceed 1000 W of board.
+A breaker tolerates that (thermal-magnetic curves are forgiving of brief overload); a
+UPS's overload protection is not. Which means **250 W implicitly commits the chassis to
+the PDU rather than behind the 1500 VA unit** — even capped, 4 x 250 W + host exceeds
+that UPS's real rating.
+
+⚠ **200 W on ana-ml3's TWO cards is deliberately conservative** (2 x 200 = 400 W is
+trivial on any circuit). Relaxable later if Anaheim's measured headroom beats its trip
+history; not a permanent figure.
 
 3. ⭐ **Decode tolerates a cap far better than training does**, which is lucky given what
    this fleet mostly does. Decode is memory-bandwidth-bound; the perf/watt curve is