power: answer the bank-budget question — DCGM has the concept, the dynamic part is a control loop, and 4x250 already is 1000 W

DCGM_CONFIG_POWER_BUDGET_GROUP ('the power budget for the entire group') exists
alongside DCGM_CONFIG_POWER_CAP_INDIVIDUAL, so the concept is first-class. The docs do
not state how a group budget is distributed, and the deduction is that it cannot be
anything exotic: the only enforcement primitive underneath is NVML's per-GPU
nvmlDeviceSetPowerManagementLimit and there is no bank-level register, so any group
budget resolves to N per-GPU writes. Static even division is one write each; 'each card
free until they are all loaded' requires continuous re-writing, which is a control loop
rather than a hardware feature. Worth a ten-minute test when the box returns, in case
NVIDIA already runs that loop.

Records the design constraint that matters more than the logic: power readings lag and
-pl application takes tens of milliseconds, so a reactive daemon overshoots during a load
ramp -- and the ramp is the dangerous moment, being the same all-cards-at-once shape as
this box's ten restart:unless-stopped containers starting together. So any such loop must
be safe-by-default and opportunistic upward: boot at budget/N, only ever raise after
observing idle neighbours. Inverted, it works for weeks and then fails on precisely the
event it existed to prevent.

And the reason to defer it: 4x250 W is already 1000 W, so the static cap is the
conservative floor of the dynamic scheme rather than an alternative. The daemon's entire
contribution is the one-card-busy case, worth perhaps 5% throughput, which is rare for a
serving fleet that puts one seat per card and common only for a training window.

docs/runbooks/fv-site-dark-20260913.md

@@ -475,3 +475,59 @@ history; not a permanent figure.
 stops holding — the worst possible failure shape, because the thing that reboots the box
 is likely to be the power event the cap existed to prevent. Systemd unit, persistence
 mode, ordered before Docker starts.
+
+## Can the whole BANK be capped at 1000 W instead of per-card? (operator question)
+
+> "is it possible to cap the ENTIRE bank to 1000w? meaning that each card can go to max
+> until they're all loaded?"
+
+**The concept is first-class in DCGM, the dynamic behaviour is not free, and the static
+cap already equals the bank budget.**
+
+### What exists
+
+`dcgmConfigPowerLimitType_enum` (DCGM API) carries exactly this distinction:
+
+    DCGM_CONFIG_POWER_CAP_INDIVIDUAL  "the power cap to be applied for each member of the group"
+    DCGM_CONFIG_POWER_BUDGET_GROUP    "the power budget for the entire group"
+
+⚠ **The documentation does not state how a group budget is distributed.** Deduction, not
+a quote: the only enforcement primitive underneath is NVML's per-GPU
+`nvmlDeviceSetPowerManagementLimit` — **there is no bank-level register** — so any group
+budget ultimately resolves to N per-GPU writes. Static even division needs one write
+each; "each card free until they are all loaded" needs **continuous re-writing**, i.e. a
+control loop rather than a hardware feature.
+
+⭐ **Worth a 10-minute test when the box is back:** set `DCGM_CONFIG_POWER_BUDGET_GROUP`
+to 1000 W and watch whether per-GPU limits move as load shifts. If NVIDIA already runs
+the loop, take it for free.
+
+### ⚠⚠ If a loop is written, its failure direction matters more than its logic
+
+Power readings lag and `-pl` application takes tens of ms, so a reactive daemon
+overshoots during a load RAMP — and the ramp is exactly the dangerous moment, because it
+is the all-four-cards-loading-at-once case (the same shape as this box's ten
+`restart: unless-stopped` containers starting together).
+
+**Therefore: safe-by-default, opportunistic upward.** Boot every card at budget/N and
+only ever RAISE a card's cap after observing idle neighbours. Never boot high and react
+down. Failure mode then becomes "slower than it could have been" instead of "tripped the
+breaker." Inverted, it is a thing that works for weeks and then fails on precisely the
+event it existed to prevent.
+
+### Why not yet
+
+**4 x 250 W = 1000 W — the static cap IS the bank budget**, and it is the conservative
+floor of the dynamic scheme rather than an alternative to it. The daemon's entire
+contribution is the one-card-busy case: ~300 W instead of ~250 W on a single card, ~17%
+more board power, which on a concave perf/watt curve is perhaps ~5% throughput. That is
+the whole prize, against a control loop whose failure mode points at a breaker.
+
+And whether that case is even common depends on workload mix. A **serving** fleet spreads
+across cards by construction — one seat per card, gateway traffic split across aliases —
+so single-card-busy is rare. A **training window** is the opposite: one card hammered,
+three idle, which is where the dynamic scheme actually pays.
+
+**Recommendation: static 250 W now, measure at the plug, build the loop only if the
+measurements show single-card-busy is the common case.** It is a pure optimization on
+top; adding it later re-architects nothing and would be built against real numbers.