diff --git a/docs/runbooks/fv-site-dark-20260913.md b/docs/runbooks/fv-site-dark-20260913.md index 9cb4bd2..bf3cbb8 100644 --- a/docs/runbooks/fv-site-dark-20260913.md +++ b/docs/runbooks/fv-site-dark-20260913.md @@ -345,3 +345,42 @@ probably a 20 A circuit -- but size it from `power.log`, not from a spec sheet. - `services/flash-next-mtp-bench/power.log` on the box holds the per-card draw right up to the cut. Recover it after boot — it is the only measurement of what the load actually drew, and it survives on `/tank`, not in the container. + +## ⭐ POLICY, forward-looking (operator, 2026-09-13): cap new cards at build time + +> "i believe our ada cards for the other colo are rated 600w each, we'll want them +> power limited to 200w" + +The generalised lesson from this outage: **decide the power envelope first and size the +cards into it**, rather than installing cards and discovering the constraint by tripping +it. 4 x 200 W = 800 W of card, which fits a real circuit with a real UPS and headroom. + +Three things to settle before that is a plan: + +1. ⚠ **200 W may be below the card's ENFORCEABLE FLOOR.** `nvidia-smi -pl` is bounded by + the part's own `Min Power Limit`, which on a high-TGP card is often around half the + rating. If the floor is 300 W, a 200 W request is refused — and, depending on how it + is scripted, refused quietly. **First command on the new hardware:** + + nvidia-smi -q -d POWER | grep -iE 'power limit|default' + + If the floor lands above 200 W, the envelope has to come from fewer cards or a bigger + feed, not from the cap. + +2. ⚠ **The 600 W figure wants confirming against the actual SKU.** The Ada parts do not + land there — RTX 6000 Ada is 300 W, L40/L40S 300/350 W, RTX 4090 450 W. 600 W is + *Blackwell* RTX PRO 6000 Workstation Edition territory. So either these are Blackwell + rather than Ada, or 600 W is a two-card/total figure. Read it off the device + (`nvidia-smi -q -d POWER`), not off a spec sheet or a recollection. + +3. ⭐ **Decode tolerates a cap far better than training does**, which is lucky given what + this fleet mostly does. Decode is memory-bandwidth-bound; the perf/watt curve is + strongly concave, so ~60-70% of TGP typically costs ~10-15% of throughput. A cap to + 33% of TGP is deeper into the steep region — the cost is real and should be measured + on the first card rather than predicted, and it will hurt a prefill-heavy or training + workload considerably more than a serving seat. + +⚠ **PERSIST THE CAP.** A hand-set limit holds until the next reboot and then silently +stops holding — the worst possible failure shape, because the thing that reboots the box +is likely to be the power event the cap existed to prevent. Systemd unit, persistence +mode, ordered before Docker starts. diff --git a/persistent-memory.md b/persistent-memory.md index f0d5ee6..1ecdb58 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -207,6 +207,8 @@ last, and do NOT restart the MTP campaign. ## Recent decisions +- `[2026-09-13]` ⭐ **STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint.** Cards for the **other colo** — operator believes Ada, rated ~600 W each — are to be **power-limited to 200 W**. 4x200 W = 800 W of card, which fits a real circuit with a real UPS. This is the generalised lesson of the FV outage: decide the power envelope first and size the cards into it. ⚠ **200 W may be below the card's enforceable floor** — `nvidia-smi -pl` is bounded by `Min Power Limit`, often ~half of TGP on a high-TGP part, and a sub-floor request is refused (possibly quietly); run `nvidia-smi -q -d POWER | grep -iE 'power limit|default'` FIRST. ⚠ **The 600 W figure needs confirming against the real SKU** — Ada parts do not land there (RTX 6000 Ada 300 W, L40/L40S 300/350 W, 4090 450 W); 600 W is Blackwell RTX PRO 6000 Workstation territory, so these may be Blackwell or 600 W may be a two-card total. Read it off the device, not a spec sheet. ⭐ **Decode tolerates caps far better than training** (memory-bandwidth-bound; ~60-70% TGP ≈ 10-15% throughput cost, concave curve) — a 33%-of-TGP cap is deeper into the steep region and should be measured on the first card. ⚠ **PERSIST the cap** (systemd unit + persistence mode, ordered before Docker): a hand-set limit stops holding at the next reboot, which is likely to be the very power event it existed to prevent. → `docs/runbooks/fv-site-dark-20260913.md` + - `[2026-09-13]` ⚠⚠⚠ **FV SITE DARK — every Fountain Valley address including the BMC went unreachable ~2.5 min into a two-card load test; all other sites healthy. Operator's leading hypothesis: the 1500 VA Eaton UPS overloaded and DIED.** It fits better than a breaker trip because a UPS's output rating sits far below the circuit's, making it the first protective device to give — which explains why the site let go at **two** cards loaded rather than four, and why the ~25 W OPNsense box died with it. ⚠ **Will not self-recover** (tripped needs a human, dead needs replacing) — do NOT poll FV. ⚠ **Do NOT use surge-only outlets to exceed a UPS rating**: both banks share one NEMA 5-15P inlet rated 12 A total; the surge bank bypasses the inverter, not the current limit. ⭐ **Recover `/tank/aimodels/flash-next-mtp-bench/power.log` FIRST** — all four cards every 10 s to the cut, on `/tank` not in a container, and the ONLY load measurement that exists. ⚠ **19 of 30 gateway aliases dark and NO local fallback** — every free local model was on fv-ml1, irv-ml1 runs no chat seat at all; the only non-fv chat backends are paid, and any coverage must be a NEW opt-in alias, never a silent repoint. ⭐ **OOB design gap**: OPNsense-as-subnet-router covers box-down/gateway-up and nothing for a site-wide loss, since the BMC's only route out is that gateway. ⚠ Recovery hazard: ten `restart: unless-stopped` vLLM containers will all load at once on power-up — mask Docker first, then `compose up -d` seat by seat (which also finishes the stale-homepage-label fix, since labels attach only at creation). → `docs/runbooks/fv-site-dark-20260913.md`, `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md` - `[2026-09-13]` ⭐⭐ **Qwen3.8-Flash-Next serving on ONE card with its 51B n-gram table in host RAM — the first seat whose weights do not fit its GPU.** `stacks/flash-next-seat/`, fv-ml1 GPU 2 `:8022`, plus a `gen-large` LiteLLM alias. Measured: 74.36 GiB weights resident, 14.00 GiB KV = 560,654 tokens at the full 262,144 context, 67 GiB host RSS, 75.5/212.3/387.8 tok/s at conc 1/4/8 (⚠ n=1). ⭐ The offload is vLLM **#54371 (UVA, merged 2026-09-09)** which **supersedes the paused #53899** — it has no worker process, so #53899's whole bug family (TP=1 deadlock #53960, `pidfd_getfd`/ptrace gate, stale-output-under-graphs) is designed out; in `v0.29.1rc0`, **not** `v0.29.0`. ⚠ **`text_config.ple_embedding_dtype` is the load-or-fail discriminator** for any community build. ⚠⚠ **`--kv-cache-memory` makes vLLM SKIP MEMORY PROFILING and ignore `--gpu-memory-utilization`** — 16 GiB nearly OOM'd on a 155K prefill with no visible failure; 14 GiB is the measured-safe value and vLLM's own "17.46 GiB to fully utilize" is 3.5 GiB too high. ⚠ MTP is off **pending measurement here, not written off** — the recipe's number is cross-harness and tested k=3 only, while the head is ONE layer run autoregressively, so k=1 is unpublished and may win (`services/flash-next-mtp-bench/`, one `off_A` rep banked before the outage). ⚠ A container once ran `(healthy)` with `PORTS=[]` — verify `docker port`, not the healthcheck. → `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md` - `[2026-09-13]` **Finished the ana-ml2→fv-ml1 renumber the cutover missed: 16 live Homepage entries pointed at the dead 10.250.50.54 and zero at the live IP.** The sweep allowlist was built from files that mention the HOST and a `homepage.href` mentions only an IP, so every label-only stack fell outside it by construction; 24 files fixed plus host copies, allowlist extended with how to derive it next time. Two bugs fell out: `deploy-stack.sh` **rejected any stack name containing a dot** (so `qwen3.5-122b`/`qwopus3.5-122b`/`mistral-medium-3.5` could not be deployed at all), and scriberr's CORS allowlist held only the dead IP and the dead `scriberr.ana.internal`. ⚠ Incomplete: the 10 running containers were never recreated and a plain power-on will NOT apply labels — the staged `compose up -d` recovery does. Eight stacks deliberately not pushed (real host drift); three of those are untracked host-only stacks. Commits `3132a16`, `969a1b6`, `d79f104`.