From 8fcc26e2c9872888673597987a20f7f8849d3091 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 13 Sep 2026 00:24:25 -0700 Subject: [PATCH] =?UTF-8?q?policy(gpu-power):=20cards=20are=20RTX=206000?= =?UTF-8?q?=20Ada=20at=20300=20W=20=E2=80=94=20200=20W=20is=20a=20mild=20c?= =?UTF-8?q?ap,=20plus=20two=20sm=5F89=20placement=20consequences?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Corrects the SKU: RTX 6000 Ada, 300 W, not the ~600 W initially recalled. That makes 200 W a cap to 67% of TGP -- the favourable part of the concave perf/watt curve, roughly 10-15% of throughput -- rather than the severe 33% cap a 600 W part would have implied, and it very likely sits above the card's enforceable floor, so the check becomes a formality rather than a gate. The protective value is worth stating: four cards at 300 W uncapped is ~1200 W, which is roughly the neighbourhood that overwhelmed a 1500 VA unit at FV with only TWO Blackwell cards drawing. Capping to 800 W makes a repeat of today a non-event. Two consequences that follow from Ada independent of power, and both are placement constraints rather than details. sm_89 has native FP8 but NOT NVFP4, which is Blackwell-only -- so the in-house NVFP4 quants that most of this fleet runs will not be accelerated on that colo's cards, and its seats want FP8 W8A8 builds or the NVFP4 checkpoints stay at FV. And it unparks the triton-backend item, which is a hard no on Ampere because fp8e4nv is unsupported on sm_86 and was explicitly deferred to Ada; sm_89 has what it needs. VRAM is 4x48 = 192 GB against fv-ml1's 391 GB, so big-model placement stays at FV. The Flash-Next seat needs 74 GiB resident on one card and would not fit a 48 GB Ada card even with the n-gram table offloaded -- the offload moves the table, not the experts. --- docs/runbooks/fv-site-dark-20260913.md | 30 +++++++++++++++++++++----- persistent-memory.md | 2 +- 2 files changed, 26 insertions(+), 6 deletions(-) diff --git a/docs/runbooks/fv-site-dark-20260913.md b/docs/runbooks/fv-site-dark-20260913.md index bf3cbb8..8fa1070 100644 --- a/docs/runbooks/fv-site-dark-20260913.md +++ b/docs/runbooks/fv-site-dark-20260913.md @@ -350,6 +350,8 @@ probably a 20 A circuit -- but size it from `power.log`, not from a spec sheet. > "i believe our ada cards for the other colo are rated 600w each, we'll want them > power limited to 200w" +> +> **CONFIRMED 2026-09-13: they are RTX 6000 Ada — 300 W, not 600 W.** The generalised lesson from this outage: **decide the power envelope first and size the cards into it**, rather than installing cards and discovering the constraint by tripping @@ -367,11 +369,15 @@ Three things to settle before that is a plan: If the floor lands above 200 W, the envelope has to come from fewer cards or a bigger feed, not from the cap. -2. ⚠ **The 600 W figure wants confirming against the actual SKU.** The Ada parts do not - land there — RTX 6000 Ada is 300 W, L40/L40S 300/350 W, RTX 4090 450 W. 600 W is - *Blackwell* RTX PRO 6000 Workstation Edition territory. So either these are Blackwell - rather than Ada, or 600 W is a two-card/total figure. Read it off the device - (`nvidia-smi -q -d POWER`), not off a spec sheet or a recollection. +2. ✅ **RESOLVED — RTX 6000 Ada, 300 W.** So 200 W is a cap to **67% of TGP**, which is + the favourable part of the curve, not the severe 33% cap a 600 W part would have + implied. The floor concern largely goes away too: 200 W was borderline against a + 600 W card's minimum and is very unlikely to sit below a 300 W card's. Still worth the + one command, but expect it to take. + + ⭐ **The protective value is real**: 4 x 300 W uncapped is ~1200 W of card, which is + roughly the neighbourhood that just overwhelmed a 1500 VA unit at FV with only TWO + Blackwell cards drawing. Capping to 800 W makes a repeat a non-event. 3. ⭐ **Decode tolerates a cap far better than training does**, which is lucky given what this fleet mostly does. Decode is memory-bandwidth-bound; the perf/watt curve is @@ -380,6 +386,20 @@ Three things to settle before that is a plan: on the first card rather than predicted, and it will hurt a prefill-heavy or training workload considerably more than a serving seat. +### ⚠ Two placement consequences of Ada, independent of power + +- **sm_89 has native FP8 but NOT NVFP4** (Blackwell-only, sm_100/sm_120). Most of our + in-house quants are NVFP4, so **they will not run accelerated on that colo's cards.** + Its seats want FP8 W8A8 builds, or the NVFP4 checkpoints stay on fv-ml1. Same class of + constraint as the Ampere finding for irv-ml1, one generation up. +- ⭐ **It unparks the triton-backend item.** That is a hard no on Ampere — crashes every + render on the A6000, `fp8e4nv` unsupported on sm_86 — and was explicitly deferred TO + Ada. sm_89 has the FP8 support it needs, so it becomes testable on this hardware. +- **VRAM:** 4 x 48 GB = 192 GB, against fv-ml1's 4 x 96 = 391 GB. Big-model placement + stays at FV. The Flash-Next seat needs 74 GiB resident on ONE card and would not fit a + 48 GB Ada card even with the n-gram table offloaded — the offload moves the *table*, + not the experts. + ⚠ **PERSIST THE CAP.** A hand-set limit holds until the next reboot and then silently stops holding — the worst possible failure shape, because the thing that reboots the box is likely to be the power event the cap existed to prevent. Systemd unit, persistence diff --git a/persistent-memory.md b/persistent-memory.md index 1ecdb58..741d979 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -207,7 +207,7 @@ last, and do NOT restart the MTP campaign. ## Recent decisions -- `[2026-09-13]` ⭐ **STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint.** Cards for the **other colo** — operator believes Ada, rated ~600 W each — are to be **power-limited to 200 W**. 4x200 W = 800 W of card, which fits a real circuit with a real UPS. This is the generalised lesson of the FV outage: decide the power envelope first and size the cards into it. ⚠ **200 W may be below the card's enforceable floor** — `nvidia-smi -pl` is bounded by `Min Power Limit`, often ~half of TGP on a high-TGP part, and a sub-floor request is refused (possibly quietly); run `nvidia-smi -q -d POWER | grep -iE 'power limit|default'` FIRST. ⚠ **The 600 W figure needs confirming against the real SKU** — Ada parts do not land there (RTX 6000 Ada 300 W, L40/L40S 300/350 W, 4090 450 W); 600 W is Blackwell RTX PRO 6000 Workstation territory, so these may be Blackwell or 600 W may be a two-card total. Read it off the device, not a spec sheet. ⭐ **Decode tolerates caps far better than training** (memory-bandwidth-bound; ~60-70% TGP ≈ 10-15% throughput cost, concave curve) — a 33%-of-TGP cap is deeper into the steep region and should be measured on the first card. ⚠ **PERSIST the cap** (systemd unit + persistence mode, ordered before Docker): a hand-set limit stops holding at the next reboot, which is likely to be the very power event it existed to prevent. → `docs/runbooks/fv-site-dark-20260913.md` +- `[2026-09-13]` ⭐ **STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint.** Cards for the **other colo** are **RTX 6000 Ada, 300 W** (confirmed 2026-09-13; an initial ~600 W recollection was wrong), to be **power-limited to 200 W**. 4x200 W = 800 W of card, which fits a real circuit with a real UPS. This is the generalised lesson of the FV outage: decide the power envelope first and size the cards into it. At 300 W TGP, 200 W is a **67% cap — the favourable part of the concave perf/watt curve, ~10-15% throughput cost**, not the severe 33% cap a 600 W part would have meant; and 200 W is very unlikely to sit below a 300 W card's enforceable floor (still confirm with `nvidia-smi -q -d POWER | grep -iE 'power limit|default'`). ⭐ 4x300 W uncapped ≈ 1200 W of card — roughly what overwhelmed a 1500 VA unit at FV with only TWO Blackwell cards drawing; capping to 800 W makes a repeat a non-event. ⭐ **Decode tolerates caps far better than training** (memory-bandwidth-bound, concave curve). ⚠⚠ **Ada is sm_89: native FP8 but NO NVFP4** (Blackwell-only) — most of our in-house quants are NVFP4 and will NOT run accelerated there; that colo's seats want FP8 W8A8, or NVFP4 checkpoints stay on fv-ml1. ⭐ **This unparks [[parked_triton_backend_ampere_fp8]]** — a hard no on Ampere (fp8e4nv unsupported sm_86), explicitly deferred TO Ada, and sm_89 has the FP8 support it needs. **VRAM 4x48 = 192 GB** vs fv-ml1's 391 GB, so big-model placement stays at FV (Flash-Next needs 74 GiB resident on ONE card — the offload moves the table, not the experts). ⚠ **PERSIST the cap** (systemd unit + persistence mode, ordered before Docker): a hand-set limit stops holding at the next reboot, which is likely to be the very power event it existed to prevent. → `docs/runbooks/fv-site-dark-20260913.md` - `[2026-09-13]` ⚠⚠⚠ **FV SITE DARK — every Fountain Valley address including the BMC went unreachable ~2.5 min into a two-card load test; all other sites healthy. Operator's leading hypothesis: the 1500 VA Eaton UPS overloaded and DIED.** It fits better than a breaker trip because a UPS's output rating sits far below the circuit's, making it the first protective device to give — which explains why the site let go at **two** cards loaded rather than four, and why the ~25 W OPNsense box died with it. ⚠ **Will not self-recover** (tripped needs a human, dead needs replacing) — do NOT poll FV. ⚠ **Do NOT use surge-only outlets to exceed a UPS rating**: both banks share one NEMA 5-15P inlet rated 12 A total; the surge bank bypasses the inverter, not the current limit. ⭐ **Recover `/tank/aimodels/flash-next-mtp-bench/power.log` FIRST** — all four cards every 10 s to the cut, on `/tank` not in a container, and the ONLY load measurement that exists. ⚠ **19 of 30 gateway aliases dark and NO local fallback** — every free local model was on fv-ml1, irv-ml1 runs no chat seat at all; the only non-fv chat backends are paid, and any coverage must be a NEW opt-in alias, never a silent repoint. ⭐ **OOB design gap**: OPNsense-as-subnet-router covers box-down/gateway-up and nothing for a site-wide loss, since the BMC's only route out is that gateway. ⚠ Recovery hazard: ten `restart: unless-stopped` vLLM containers will all load at once on power-up — mask Docker first, then `compose up -d` seat by seat (which also finishes the stale-homepage-label fix, since labels attach only at creation). → `docs/runbooks/fv-site-dark-20260913.md`, `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md` - `[2026-09-13]` ⭐⭐ **Qwen3.8-Flash-Next serving on ONE card with its 51B n-gram table in host RAM — the first seat whose weights do not fit its GPU.** `stacks/flash-next-seat/`, fv-ml1 GPU 2 `:8022`, plus a `gen-large` LiteLLM alias. Measured: 74.36 GiB weights resident, 14.00 GiB KV = 560,654 tokens at the full 262,144 context, 67 GiB host RSS, 75.5/212.3/387.8 tok/s at conc 1/4/8 (⚠ n=1). ⭐ The offload is vLLM **#54371 (UVA, merged 2026-09-09)** which **supersedes the paused #53899** — it has no worker process, so #53899's whole bug family (TP=1 deadlock #53960, `pidfd_getfd`/ptrace gate, stale-output-under-graphs) is designed out; in `v0.29.1rc0`, **not** `v0.29.0`. ⚠ **`text_config.ple_embedding_dtype` is the load-or-fail discriminator** for any community build. ⚠⚠ **`--kv-cache-memory` makes vLLM SKIP MEMORY PROFILING and ignore `--gpu-memory-utilization`** — 16 GiB nearly OOM'd on a 155K prefill with no visible failure; 14 GiB is the measured-safe value and vLLM's own "17.46 GiB to fully utilize" is 3.5 GiB too high. ⚠ MTP is off **pending measurement here, not written off** — the recipe's number is cross-harness and tested k=3 only, while the head is ONE layer run autoregressively, so k=1 is unpublished and may win (`services/flash-next-mtp-bench/`, one `off_A` rep banked before the outage). ⚠ A container once ran `(healthy)` with `PORTS=[]` — verify `docker port`, not the healthcheck. → `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md`