diff --git a/docs/runbooks/fv-site-dark-20260913.md b/docs/runbooks/fv-site-dark-20260913.md index 3780d56..9cb4bd2 100644 --- a/docs/runbooks/fv-site-dark-20260913.md +++ b/docs/runbooks/fv-site-dark-20260913.md @@ -253,6 +253,55 @@ Expect `[]`. Before the outage that query returned 16 entries. **Do not reboot the OPNsense firewall** (standing operator directive; its reboot API 403s anyway). +## ⚠ THE CIRCUIT CASE — what split power does and does not buy (operator, 2026-09-13) + +> "unless of course the thing trips the circuit anyway." + +**It still helps, but only halfway, and the halfway matters.** + +- ✅ **A breaker trip is exactly what the split survives.** Firewall + BMC on the UPS is + ~25-40 W of load on a 1500 VA unit — hours of battery, not minutes. On a trip the UPS + stops being a load-bearing supply and goes back to being what it is for. +- ❌ **A live firewall is useless if the path OUT of the site is dead.** Our UPS covers + our gear; it does not cover the **colo's handoff** — their switch, ONT or demarc. If + that sits on the circuit we just tripped, the result is a firewall running happily on + battery with nothing upstream to talk to, and the drive happens anyway. + ⭐ **ASK THE FACILITY: is the network handoff on our circuit or theirs, and is theirs + on facility UPS?** This is the question that decides whether split power actually + delivers remote diagnosis or merely feels like it does. + +### ⚠⚠ And the case where none of the above matters + +**If four cards plus host exceeds the circuit, no UPS arrangement helps** — the box does +not fit its feed. Removing an undersized UPS does not remove the constraint, it promotes +the next one: + + UPS ~900-1200 W (the one that just gave way) + circuit ~1800 W @ 15 A / ~2400 W @ 20 A + +Which side of those the four-card figure lands on decides everything, which is why that +single ammeter reading is the load-bearing measurement of the visit. + +### ⭐ The lever that may avoid an electrician: per-card power limits + +`nvidia-smi -pl ` caps TGP per card. The box can be made to fit whatever the feed +turns out to be, at a **throughput** cost rather than a **rewiring** cost — four capped +cards on a 15 A circuit is a dial we control today, where a 20 A drop is a ticket and a +site visit. + +- Read `nvidia-smi -q -d POWER` first for the enforced min/max range per card; do not + assume how much room the dial has. +- ⚠ **If capping is the answer it MUST be persisted** (systemd unit, or an `if-up` + equivalent). A limit that evaporates on reboot is worse than no limit, because it will + hold right up until the next power event and then silently stop holding. + +### Three questions for the site visit + +1. What is the **breaker rating** on that circuit? +2. Is the circuit **dedicated** to us, or shared with other racks/tenants? +3. Is the **network handoff** on our circuit or the facility's, and is the facility's on + their UPS? + ## ⚠ PROPOSED, NOT RATIFIED — split the power so the management plane survives **This is a recommendation awaiting the operator's call, not settled intent.** Written diff --git a/persistent-memory.md b/persistent-memory.md index 086c42e..f0d5ee6 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -132,6 +132,17 @@ cross-check on the GPU share. Measure idle / 1-card / **2-card** / **4-card**, a overload; an average-only meter's figure is a FLOOR, not the draw). ⭐ The 4-card number goes into `servers/fv-ml1/README.md` permanently — it closes the cutover's open "circuit specced against half the real draw" question with a measurement. +⚠⚠ **Removing an undersized UPS does not remove the constraint, it promotes the next +one** (UPS ~900-1200 W → circuit ~1800 W @15 A / ~2400 W @20 A). If 4 cards + host +exceeds the circuit, NO UPS arrangement helps. ⭐ **The lever is `nvidia-smi -pl`** — +cap per-card TGP so the box fits its feed at a throughput cost instead of a rewiring +cost; read `nvidia-smi -q -d POWER` for the enforced range first, and **persist the cap** +(a limit that evaporates on reboot holds until the next power event and then does not). +⭐ Split power (firewall+BMC on UPS, chassis on PDU) survives a *breaker* trip on +battery — 25-40 W on a 1500 VA unit is hours — **but only delivers remote access if the +COLO'S HANDOFF survives too**; ask the facility whether the handoff is on our circuit or +theirs. Three questions for the visit: breaker rating, is the circuit dedicated, whose +gear is the handoff on. **It will not self-recover; do not poll FV addresses.** 19 of 30 gateway aliases are down with no local fallback (every free local model was on fv-ml1; irv-ml1 runs no chat seat). On recovery: recover `power.log` first, mask Docker before the network,