runbook(fv-outage): the circuit case — split power survives a trip on battery, but only if the colo handoff does

Operator: 'unless of course the thing trips the circuit anyway.' Correct, and it splits
into two halves with different answers.

A breaker trip is the event the split-power proposal survives: firewall + BMC is 25-40 W
on a 1500 VA unit, which is hours of battery, and on a trip the UPS stops being a
load-bearing supply and goes back to being what it is for. What it does NOT cover is the
colo's own handoff -- their switch, ONT or demarc. If that sits on the circuit we just
tripped, the outcome is a firewall running on battery with nothing upstream to talk to
and the drive happens anyway. Added as a question for the facility, because it decides
whether split power delivers remote diagnosis or merely feels like it does.

Records the case where none of it matters: removing an undersized UPS does not remove
the constraint, it promotes the next one -- UPS ~900-1200 W to circuit ~1800 W at 15 A
or ~2400 W at 20 A. Which side the four-card figure lands on decides everything, which
is what makes that single ammeter reading the load-bearing measurement of the visit.

Surfaces the lever that may avoid an electrician entirely: nvidia-smi -pl caps per-card
TGP, so the box can be made to fit its feed at a throughput cost rather than a rewiring
cost. Read nvidia-smi -q -d POWER for the enforced range before assuming how much room
the dial has, and persist any cap -- one that evaporates on reboot will hold right up
until the next power event and then silently stop holding.

docs/runbooks/fv-site-dark-20260913.md

@@ -253,6 +253,55 @@ Expect `[]`. Before the outage that query returned 16 entries.
 **Do not reboot the OPNsense firewall** (standing operator directive; its reboot API
 403s anyway).
 
+## ⚠ THE CIRCUIT CASE — what split power does and does not buy (operator, 2026-09-13)
+
+> "unless of course the thing trips the circuit anyway."
+
+**It still helps, but only halfway, and the halfway matters.**
+
+- ✅ **A breaker trip is exactly what the split survives.** Firewall + BMC on the UPS is
+  ~25-40 W of load on a 1500 VA unit — hours of battery, not minutes. On a trip the UPS
+  stops being a load-bearing supply and goes back to being what it is for.
+- ❌ **A live firewall is useless if the path OUT of the site is dead.** Our UPS covers
+  our gear; it does not cover the **colo's handoff** — their switch, ONT or demarc. If
+  that sits on the circuit we just tripped, the result is a firewall running happily on
+  battery with nothing upstream to talk to, and the drive happens anyway.
+  ⭐ **ASK THE FACILITY: is the network handoff on our circuit or theirs, and is theirs
+  on facility UPS?** This is the question that decides whether split power actually
+  delivers remote diagnosis or merely feels like it does.
+
+### ⚠⚠ And the case where none of the above matters
+
+**If four cards plus host exceeds the circuit, no UPS arrangement helps** — the box does
+not fit its feed. Removing an undersized UPS does not remove the constraint, it promotes
+the next one:
+
+    UPS        ~900-1200 W   (the one that just gave way)
+    circuit    ~1800 W @ 15 A  /  ~2400 W @ 20 A
+
+Which side of those the four-card figure lands on decides everything, which is why that
+single ammeter reading is the load-bearing measurement of the visit.
+
+### ⭐ The lever that may avoid an electrician: per-card power limits
+
+`nvidia-smi -pl <watts>` caps TGP per card. The box can be made to fit whatever the feed
+turns out to be, at a **throughput** cost rather than a **rewiring** cost — four capped
+cards on a 15 A circuit is a dial we control today, where a 20 A drop is a ticket and a
+site visit.
+
+- Read `nvidia-smi -q -d POWER` first for the enforced min/max range per card; do not
+  assume how much room the dial has.
+- ⚠ **If capping is the answer it MUST be persisted** (systemd unit, or an `if-up`
+  equivalent). A limit that evaporates on reboot is worse than no limit, because it will
+  hold right up until the next power event and then silently stop holding.
+
+### Three questions for the site visit
+
+1. What is the **breaker rating** on that circuit?
+2. Is the circuit **dedicated** to us, or shared with other racks/tenants?
+3. Is the **network handoff** on our circuit or the facility's, and is the facility's on
+   their UPS?
+
 ## ⚠ PROPOSED, NOT RATIFIED — split the power so the management plane survives
 
 **This is a recommendation awaiting the operator's call, not settled intent.** Written

persistent-memory.md

@@ -132,6 +132,17 @@ cross-check on the GPU share. Measure idle / 1-card / **2-card** / **4-card**, a
 overload; an average-only meter's figure is a FLOOR, not the draw). ⭐ The 4-card number
 goes into `servers/fv-ml1/README.md` permanently — it closes the cutover's open
 "circuit specced against half the real draw" question with a measurement.
+⚠⚠ **Removing an undersized UPS does not remove the constraint, it promotes the next
+one** (UPS ~900-1200 W → circuit ~1800 W @15 A / ~2400 W @20 A). If 4 cards + host
+exceeds the circuit, NO UPS arrangement helps. ⭐ **The lever is `nvidia-smi -pl`** —
+cap per-card TGP so the box fits its feed at a throughput cost instead of a rewiring
+cost; read `nvidia-smi -q -d POWER` for the enforced range first, and **persist the cap**
+(a limit that evaporates on reboot holds until the next power event and then does not).
+⭐ Split power (firewall+BMC on UPS, chassis on PDU) survives a *breaker* trip on
+battery — 25-40 W on a 1500 VA unit is hours — **but only delivers remote access if the
+COLO'S HANDOFF survives too**; ask the facility whether the handoff is on our circuit or
+theirs. Three questions for the visit: breaker rating, is the circuit dedicated, whose
+gear is the handoff on.
 **It will not self-recover; do not poll FV addresses.** 19 of 30 gateway aliases are
 down with no local fallback (every free local model was on fv-ml1; irv-ml1 runs no
 chat seat). On recovery: recover `power.log` first, mask Docker before the network,