From 3e61d7d4e0c6f8e10542bc5d2e9599d18b5d4140 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 13 Sep 2026 00:23:20 -0700 Subject: [PATCH] =?UTF-8?q?policy:=20cap=20GPU=20power=20limits=20at=20bui?= =?UTF-8?q?ld=20time=20=E2=80=94=20200=20W=20for=20the=20other=20colo's=20?= =?UTF-8?q?cards?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator directive, and the right generalisation of the FV outage: decide the power envelope first and size the cards into it, rather than installing cards and discovering the constraint by tripping it. Four cards at 200 W is 800 W, which fits a real circuit with a real UPS and headroom. Records three things to settle before it is a plan. First, 200 W may sit below the card's enforceable floor: nvidia-smi -pl is bounded by Min Power Limit, often around half of TGP on a high-TGP part, and a sub-floor request is refused -- quietly, depending on how it is scripted. Run nvidia-smi -q -d POWER before any build planning depends on the number. Second, the 600 W figure wants confirming against the actual SKU. The Ada parts do not land there -- RTX 6000 Ada is 300 W, L40/L40S 300/350 W, 4090 450 W -- while 600 W is Blackwell RTX PRO 6000 Workstation territory, so these may be Blackwell or the figure may be a two-card total. Read it off the device rather than a spec sheet. Third, the workload asymmetry is in this fleet's favour: decode is memory-bandwidth-bound and tolerates a cap far better than training does, with a concave perf/watt curve where 60-70% of TGP costs roughly 10-15% of throughput. A cap to a third of TGP is deeper into the steep region; measure it on the first card rather than predicting, and expect prefill-heavy and training work to pay more than a serving seat. And persist the cap. A hand-set limit holds until the next reboot and then silently stops holding, which is the worst shape available given that the thing rebooting the box is likely to be the power event the cap existed to prevent. --- docs/runbooks/fv-site-dark-20260913.md | 39 ++++++++++++++++++++++++++ persistent-memory.md | 2 ++ 2 files changed, 41 insertions(+) diff --git a/docs/runbooks/fv-site-dark-20260913.md b/docs/runbooks/fv-site-dark-20260913.md index 9cb4bd2..bf3cbb8 100644 --- a/docs/runbooks/fv-site-dark-20260913.md +++ b/docs/runbooks/fv-site-dark-20260913.md @@ -345,3 +345,42 @@ probably a 20 A circuit -- but size it from `power.log`, not from a spec sheet. - `services/flash-next-mtp-bench/power.log` on the box holds the per-card draw right up to the cut. Recover it after boot — it is the only measurement of what the load actually drew, and it survives on `/tank`, not in the container. + +## ⭐ POLICY, forward-looking (operator, 2026-09-13): cap new cards at build time + +> "i believe our ada cards for the other colo are rated 600w each, we'll want them +> power limited to 200w" + +The generalised lesson from this outage: **decide the power envelope first and size the +cards into it**, rather than installing cards and discovering the constraint by tripping +it. 4 x 200 W = 800 W of card, which fits a real circuit with a real UPS and headroom. + +Three things to settle before that is a plan: + +1. ⚠ **200 W may be below the card's ENFORCEABLE FLOOR.** `nvidia-smi -pl` is bounded by + the part's own `Min Power Limit`, which on a high-TGP card is often around half the + rating. If the floor is 300 W, a 200 W request is refused — and, depending on how it + is scripted, refused quietly. **First command on the new hardware:** + + nvidia-smi -q -d POWER | grep -iE 'power limit|default' + + If the floor lands above 200 W, the envelope has to come from fewer cards or a bigger + feed, not from the cap. + +2. ⚠ **The 600 W figure wants confirming against the actual SKU.** The Ada parts do not + land there — RTX 6000 Ada is 300 W, L40/L40S 300/350 W, RTX 4090 450 W. 600 W is + *Blackwell* RTX PRO 6000 Workstation Edition territory. So either these are Blackwell + rather than Ada, or 600 W is a two-card/total figure. Read it off the device + (`nvidia-smi -q -d POWER`), not off a spec sheet or a recollection. + +3. ⭐ **Decode tolerates a cap far better than training does**, which is lucky given what + this fleet mostly does. Decode is memory-bandwidth-bound; the perf/watt curve is + strongly concave, so ~60-70% of TGP typically costs ~10-15% of throughput. A cap to + 33% of TGP is deeper into the steep region — the cost is real and should be measured + on the first card rather than predicted, and it will hurt a prefill-heavy or training + workload considerably more than a serving seat. + +⚠ **PERSIST THE CAP.** A hand-set limit holds until the next reboot and then silently +stops holding — the worst possible failure shape, because the thing that reboots the box +is likely to be the power event the cap existed to prevent. Systemd unit, persistence +mode, ordered before Docker starts. diff --git a/persistent-memory.md b/persistent-memory.md index f0d5ee6..1ecdb58 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -207,6 +207,8 @@ last, and do NOT restart the MTP campaign. ## Recent decisions +- `[2026-09-13]` ⭐ **STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint.** Cards for the **other colo** — operator believes Ada, rated ~600 W each — are to be **power-limited to 200 W**. 4x200 W = 800 W of card, which fits a real circuit with a real UPS. This is the generalised lesson of the FV outage: decide the power envelope first and size the cards into it. ⚠ **200 W may be below the card's enforceable floor** — `nvidia-smi -pl` is bounded by `Min Power Limit`, often ~half of TGP on a high-TGP part, and a sub-floor request is refused (possibly quietly); run `nvidia-smi -q -d POWER | grep -iE 'power limit|default'` FIRST. ⚠ **The 600 W figure needs confirming against the real SKU** — Ada parts do not land there (RTX 6000 Ada 300 W, L40/L40S 300/350 W, 4090 450 W); 600 W is Blackwell RTX PRO 6000 Workstation territory, so these may be Blackwell or 600 W may be a two-card total. Read it off the device, not a spec sheet. ⭐ **Decode tolerates caps far better than training** (memory-bandwidth-bound; ~60-70% TGP ≈ 10-15% throughput cost, concave curve) — a 33%-of-TGP cap is deeper into the steep region and should be measured on the first card. ⚠ **PERSIST the cap** (systemd unit + persistence mode, ordered before Docker): a hand-set limit stops holding at the next reboot, which is likely to be the very power event it existed to prevent. → `docs/runbooks/fv-site-dark-20260913.md` + - `[2026-09-13]` ⚠⚠⚠ **FV SITE DARK — every Fountain Valley address including the BMC went unreachable ~2.5 min into a two-card load test; all other sites healthy. Operator's leading hypothesis: the 1500 VA Eaton UPS overloaded and DIED.** It fits better than a breaker trip because a UPS's output rating sits far below the circuit's, making it the first protective device to give — which explains why the site let go at **two** cards loaded rather than four, and why the ~25 W OPNsense box died with it. ⚠ **Will not self-recover** (tripped needs a human, dead needs replacing) — do NOT poll FV. ⚠ **Do NOT use surge-only outlets to exceed a UPS rating**: both banks share one NEMA 5-15P inlet rated 12 A total; the surge bank bypasses the inverter, not the current limit. ⭐ **Recover `/tank/aimodels/flash-next-mtp-bench/power.log` FIRST** — all four cards every 10 s to the cut, on `/tank` not in a container, and the ONLY load measurement that exists. ⚠ **19 of 30 gateway aliases dark and NO local fallback** — every free local model was on fv-ml1, irv-ml1 runs no chat seat at all; the only non-fv chat backends are paid, and any coverage must be a NEW opt-in alias, never a silent repoint. ⭐ **OOB design gap**: OPNsense-as-subnet-router covers box-down/gateway-up and nothing for a site-wide loss, since the BMC's only route out is that gateway. ⚠ Recovery hazard: ten `restart: unless-stopped` vLLM containers will all load at once on power-up — mask Docker first, then `compose up -d` seat by seat (which also finishes the stale-homepage-label fix, since labels attach only at creation). → `docs/runbooks/fv-site-dark-20260913.md`, `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md` - `[2026-09-13]` ⭐⭐ **Qwen3.8-Flash-Next serving on ONE card with its 51B n-gram table in host RAM — the first seat whose weights do not fit its GPU.** `stacks/flash-next-seat/`, fv-ml1 GPU 2 `:8022`, plus a `gen-large` LiteLLM alias. Measured: 74.36 GiB weights resident, 14.00 GiB KV = 560,654 tokens at the full 262,144 context, 67 GiB host RSS, 75.5/212.3/387.8 tok/s at conc 1/4/8 (⚠ n=1). ⭐ The offload is vLLM **#54371 (UVA, merged 2026-09-09)** which **supersedes the paused #53899** — it has no worker process, so #53899's whole bug family (TP=1 deadlock #53960, `pidfd_getfd`/ptrace gate, stale-output-under-graphs) is designed out; in `v0.29.1rc0`, **not** `v0.29.0`. ⚠ **`text_config.ple_embedding_dtype` is the load-or-fail discriminator** for any community build. ⚠⚠ **`--kv-cache-memory` makes vLLM SKIP MEMORY PROFILING and ignore `--gpu-memory-utilization`** — 16 GiB nearly OOM'd on a 155K prefill with no visible failure; 14 GiB is the measured-safe value and vLLM's own "17.46 GiB to fully utilize" is 3.5 GiB too high. ⚠ MTP is off **pending measurement here, not written off** — the recipe's number is cross-harness and tested k=3 only, while the head is ONE layer run autoregressively, so k=1 is unpublished and may win (`services/flash-next-mtp-bench/`, one `off_A` rep banked before the outage). ⚠ A container once ran `(healthy)` with `PORTS=[]` — verify `docker port`, not the healthcheck. → `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md` - `[2026-09-13]` **Finished the ana-ml2→fv-ml1 renumber the cutover missed: 16 live Homepage entries pointed at the dead 10.250.50.54 and zero at the live IP.** The sweep allowlist was built from files that mention the HOST and a `homepage.href` mentions only an IP, so every label-only stack fell outside it by construction; 24 files fixed plus host copies, allowlist extended with how to derive it next time. Two bugs fell out: `deploy-stack.sh` **rejected any stack name containing a dot** (so `qwen3.5-122b`/`qwopus3.5-122b`/`mistral-medium-3.5` could not be deployed at all), and scriberr's CORS allowlist held only the dead IP and the dead `scriberr.ana.internal`. ⚠ Incomplete: the 10 running containers were never recreated and a plain power-on will NOT apply labels — the staged `compose up -d` recovery does. Eight stacks deliberately not pushed (real host drift); three of those are untracked host-only stacks. Commits `3132a16`, `969a1b6`, `d79f104`.