power: what DCGM is, and why not to plan around it on workstation-SKU cards

DCGM is NVIDIA's own Data Center GPU Manager -- first-party, Apache-2.0, packaged as
datacenter-gpu-manager -- and it layers above NVML rather than beside it: nvidia-smi is a
thin CLI over NVML's per-GPU primitives, and DCGM is a daemon plus dcgmi adding health,
diagnostics, config enforcement, policy and group abstractions on top. Which is why its
group notion still resolves to N per-GPU writes underneath.

The caveat that matters, and it undercuts the experiment suggested in the previous
commit: DCGM is datacenter-oriented and parts of it are gated to datacenter SKUs of the
Tesla/A100/H100 class. Our cards are professional/workstation parts -- RTX PRO 6000
Blackwell Max-Q and RTX 6000 Ada -- and several DCGM capabilities are unsupported or
degraded outside that line, plausibly including config enforcement, which is precisely
the power path. So DCGM_CONFIG_POWER_BUDGET_GROUP may return 'unsupported on this
device'. Downgraded from 'worth testing' to five minutes of curiosity after the real
work, and explicitly not a planning assumption.

None of which touches the plan: nvidia-smi -pl 250 is plain NVML and works on these
cards. DCGM would only have bought the group-budget experiment and nicer telemetry, and
is probably not installed anyway since beszel-agent-nvidia shells out to nvidia-smi.

docs/runbooks/fv-site-dark-20260913.md

@@ -498,9 +498,29 @@ budget ultimately resolves to N per-GPU writes. Static even division needs one w
 each; "each card free until they are all loaded" needs **continuous re-writing**, i.e. a
 control loop rather than a hardware feature.
 
-⭐ **Worth a 10-minute test when the box is back:** set `DCGM_CONFIG_POWER_BUDGET_GROUP`
-to 1000 W and watch whether per-GPU limits move as load shifts. If NVIDIA already runs
-the loop, take it for free.
+**What DCGM is:** NVIDIA's own **Data Center GPU Manager** — first-party, open source
+(Apache 2.0, `NVIDIA/DCGM`), packaged as `datacenter-gpu-manager`. It layers above NVML:
+
+    nvidia-smi  CLI, thin wrapper over NVML
+    NVML        low-level C library, PER-GPU primitives (what -pl actually calls)
+    DCGM        daemon (nv-hostengine) + dcgmi, ABOVE NVML — health, diagnostics,
+                config enforcement, policy, group abstractions; dcgm-exporter is its
+                Prometheus sidecar
+
+Its "group" notion is therefore a management-layer abstraction over per-GPU NVML calls,
+which is why the bank budget still resolves to N per-GPU writes underneath.
+
+⚠⚠ **DO NOT PLAN AROUND DCGM HERE. It is datacenter-oriented and parts of it are gated
+to datacenter SKUs** (Tesla / A100 / H100 class). Our cards are **professional /
+workstation** parts — RTX PRO 6000 Blackwell Max-Q and RTX 6000 Ada — and several DCGM
+capabilities are unsupported or degraded outside the datacenter line, plausibly including
+**config enforcement, which is exactly the power path**. The
+`DCGM_CONFIG_POWER_BUDGET_GROUP` experiment may simply return "unsupported on this
+device": five minutes of curiosity after the real work, not a planning assumption.
+
+✅ **The static cap needs none of this.** `nvidia-smi -pl 250` is plain NVML and works on
+these cards. DCGM would only buy the group-budget experiment and richer telemetry, and is
+probably not even installed — `beszel-agent-nvidia` shells out to `nvidia-smi`.
 
 ### ⚠⚠ If a loop is written, its failure direction matters more than its logic