From 2da0c76d991a1644727753f4ae0834f0f3248471 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 13 Sep 2026 00:26:29 -0700 Subject: [PATCH] =?UTF-8?q?correct=20the=20hardware:=20fv-ml1=20is=204x=20?= =?UTF-8?q?Blackwell=20Max-Q=20300W,=20ana-ml3=20is=202x=20Ada=20RTX=20600?= =?UTF-8?q?0=20=E2=80=94=20and=20four=20cards=20is=20a=20breaker=20problem?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator clarification, and it separates two boxes I had been conflating. fv-ml1 is 4x Blackwell RTX PRO 6000 Max-Q at 300 W each (Max-Q being the reduced-TGP SKU; the Workstation Edition is the 600 W part), 391 GB VRAM, deployed and currently dark. ana-ml3 is 2x Ada Generation RTX 6000 at 300 W, 96 GB VRAM, not yet deployed. The 200 W cap directive is ana-ml3's. With the TGP known, the outage stops being a vague 'undersized' and acquires a mechanism: two Max-Q cards at 300 W is ~600 W of card, plus a host carrying 566 GB of RAM, drives, fans and PSU conversion loss at perhaps 200-350 W, against an Eaton 1500 VA's real ~900-1200 W. That lands at or just over the rating, which is precisely what explains a full day of service on one card and failure minutes into the second. The host term is the only one being guessed; idle-at-the-plug measures it directly. It also surfaces something that is not a UPS question at all. Four cards at 300 W plus ~300 W of host is ~1500 W against a 15 A circuit's 1440 W continuous derating, so four cards uncapped is marginal on the breaker with no UPS in the path. Capping therefore belongs at fv-ml1 as well as ana-ml3, or fv-ml1 needs a 20 A feed -- and worth noting today's incident only ever had two of the four cards working. ana-ml3's placement constraints sharpen too: sm_89 has native FP8 but no NVFP4, so the in-house NVFP4 quants stay at FV, and at 96 GB total it cannot host the Flash-Next seat at all -- that needs 74 GiB resident on a single card, and the offload moves the n-gram table rather than the experts. --- docs/runbooks/fv-site-dark-20260913.md | 52 +++++++++++++++++++++----- persistent-memory.md | 2 +- 2 files changed, 44 insertions(+), 10 deletions(-) diff --git a/docs/runbooks/fv-site-dark-20260913.md b/docs/runbooks/fv-site-dark-20260913.md index 8fa1070..338b3cb 100644 --- a/docs/runbooks/fv-site-dark-20260913.md +++ b/docs/runbooks/fv-site-dark-20260913.md @@ -351,7 +351,15 @@ probably a 20 A circuit -- but size it from `power.log`, not from a spec sheet. > "i believe our ada cards for the other colo are rated 600w each, we'll want them > power limited to 200w" > -> **CONFIRMED 2026-09-13: they are RTX 6000 Ada — 300 W, not 600 W.** +> **CLARIFIED BY OPERATOR 2026-09-13 — the two boxes are different hardware:** +> +> | box | cards | TGP each | VRAM total | status | +> |---|---|---|---|---| +> | **fv-ml1** | 4x **Blackwell** RTX PRO 6000 **Max-Q** | **300 W** (Max-Q is the reduced-TGP SKU; the Workstation Edition is 600 W) | 4x96 = 391 GB | deployed, currently dark | +> | **ana-ml3** | 2x **Ada Generation** RTX 6000 | **300 W** | 2x48 = 96 GB | **NOT YET DEPLOYED** | +> +> The 200 W cap directive applies to **ana-ml3**. It very likely wants applying to +> fv-ml1 as well — see the four-card circuit arithmetic below. The generalised lesson from this outage: **decide the power envelope first and size the cards into it**, rather than installing cards and discovering the constraint by tripping @@ -369,15 +377,41 @@ Three things to settle before that is a plan: If the floor lands above 200 W, the envelope has to come from fewer cards or a bigger feed, not from the cap. -2. ✅ **RESOLVED — RTX 6000 Ada, 300 W.** So 200 W is a cap to **67% of TGP**, which is - the favourable part of the curve, not the severe 33% cap a 600 W part would have - implied. The floor concern largely goes away too: 200 W was borderline against a - 600 W card's minimum and is very unlikely to sit below a 300 W card's. Still worth the - one command, but expect it to take. +2. ✅ **RESOLVED — both card types are 300 W.** So 200 W is a cap to **67% of TGP**, the + favourable part of the concave curve, not the severe 33% cap a 600 W part would have + implied. The enforceable-floor concern largely goes away too: 200 W was borderline + against a 600 W card's minimum and is very unlikely to sit below a 300 W card's. Worth + the one command; expect it to take. - ⭐ **The protective value is real**: 4 x 300 W uncapped is ~1200 W of card, which is - roughly the neighbourhood that just overwhelmed a 1500 VA unit at FV with only TWO - Blackwell cards drawing. Capping to 800 W makes a repeat a non-event. + **ana-ml3: 2 x 200 W = 400 W of card.** Modest, and pointed — ana-ml3 lands in the + **Anaheim** rack whose breaker tripped on 2026-08-26 and 2026-09-11, one of those + caused by this very chassis before it relocated. The cap there is remediation of a + circuit with a track record, not precaution. + +### ⭐⭐ The outage arithmetic, now that the TGP is known + + 2 x Blackwell Max-Q @ 300 W ~ 600 W of card under load + host (board, 566 GB RAM, drives, + fans, PSU conversion loss) ~ 200-350 W <-- UNMEASURED, the gap + ----------- + ~ 800-950 W + + Eaton 1500 VA real watt rating ~ 900-1200 W depending on model + +**At or just over the line** — and this is what a vague "undersized" could not explain: +why it ran a full day on one card (~500-650 W, comfortably inside) and died minutes into +the second (~800-950 W, at or past the rating). The host term is the only one being +guessed at, and idle-at-the-plug with all seats down measures it directly. + +### ⚠⚠ FOUR cards is a BREAKER problem, not a UPS problem + + 4 x 300 W card + ~300 W host ~ 1500 W + 15 A circuit, 80% continuous = 1440 W + +**Four cards uncapped is marginal on a 15 A circuit with no UPS in the path at all.** So +capping belongs at **fv-ml1 too**, not only ana-ml3. If the four-card ammeter reading +confirms it, fv-ml1 needs a per-card cap or a 20 A feed before anyone loads all four +again — and note that today's incident only ever had TWO cards working. 3. ⭐ **Decode tolerates a cap far better than training does**, which is lucky given what this fleet mostly does. Decode is memory-bandwidth-bound; the perf/watt curve is diff --git a/persistent-memory.md b/persistent-memory.md index 741d979..041c934 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -207,7 +207,7 @@ last, and do NOT restart the MTP campaign. ## Recent decisions -- `[2026-09-13]` ⭐ **STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint.** Cards for the **other colo** are **RTX 6000 Ada, 300 W** (confirmed 2026-09-13; an initial ~600 W recollection was wrong), to be **power-limited to 200 W**. 4x200 W = 800 W of card, which fits a real circuit with a real UPS. This is the generalised lesson of the FV outage: decide the power envelope first and size the cards into it. At 300 W TGP, 200 W is a **67% cap — the favourable part of the concave perf/watt curve, ~10-15% throughput cost**, not the severe 33% cap a 600 W part would have meant; and 200 W is very unlikely to sit below a 300 W card's enforceable floor (still confirm with `nvidia-smi -q -d POWER | grep -iE 'power limit|default'`). ⭐ 4x300 W uncapped ≈ 1200 W of card — roughly what overwhelmed a 1500 VA unit at FV with only TWO Blackwell cards drawing; capping to 800 W makes a repeat a non-event. ⭐ **Decode tolerates caps far better than training** (memory-bandwidth-bound, concave curve). ⚠⚠ **Ada is sm_89: native FP8 but NO NVFP4** (Blackwell-only) — most of our in-house quants are NVFP4 and will NOT run accelerated there; that colo's seats want FP8 W8A8, or NVFP4 checkpoints stay on fv-ml1. ⭐ **This unparks [[parked_triton_backend_ampere_fp8]]** — a hard no on Ampere (fp8e4nv unsupported sm_86), explicitly deferred TO Ada, and sm_89 has the FP8 support it needs. **VRAM 4x48 = 192 GB** vs fv-ml1's 391 GB, so big-model placement stays at FV (Flash-Next needs 74 GiB resident on ONE card — the offload moves the table, not the experts). ⚠ **PERSIST the cap** (systemd unit + persistence mode, ordered before Docker): a hand-set limit stops holding at the next reboot, which is likely to be the very power event it existed to prevent. → `docs/runbooks/fv-site-dark-20260913.md` +- `[2026-09-13]` ⭐ **STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint.** ⭐ **Two DIFFERENT boxes, clarified by operator 2026-09-13:** **fv-ml1** = 4x **Blackwell** RTX PRO 6000 **Max-Q @ 300 W** (Max-Q is the reduced-TGP SKU; Workstation Edition is 600 W), 391 GB VRAM, deployed. **ana-ml3** = 2x **Ada Generation** RTX 6000 @ 300 W, 96 GB VRAM, **NOT YET DEPLOYED**. The 200 W cap directive is for **ana-ml3** (2x200 = 400 W) — and ⭐ ana-ml3 lands in the **Anaheim** rack whose breaker tripped 2026-08-26 and 2026-09-11, one of those caused by this very chassis before it relocated, so the cap there is remediation of a known-bad circuit, not precaution. 4x200 W = 800 W of card, which fits a real circuit with a real UPS. This is the generalised lesson of the FV outage: decide the power envelope first and size the cards into it. At 300 W TGP, 200 W is a **67% cap — the favourable part of the concave perf/watt curve, ~10-15% throughput cost**, not the severe 33% cap a 600 W part would have meant; and 200 W is very unlikely to sit below a 300 W card's enforceable floor (still confirm with `nvidia-smi -q -d POWER | grep -iE 'power limit|default'`). ⭐⭐ **The outage arithmetic now has numbers:** 2x Blackwell Max-Q @300 W ≈ 600 W of card + host (566 GB RAM, drives, fans, PSU losses) ≈ 200-350 W = **~800-950 W against a 1500 VA Eaton's real ~900-1200 W rating** — at or just over the line, which is what explains a full day on ONE card (~500-650 W, inside) and death minutes into the SECOND. The host term is the only guess; idle-at-the-plug measures it. ⚠⚠ **And FOUR cards is a BREAKER problem, not a UPS problem:** 4x300 + ~300 host ≈ **1500 W vs a 15 A circuit's 1440 W continuous (80%) derating** — so **capping belongs at fv-ml1 too**, or it needs a 20 A feed, before anyone loads all four cards again. Today's incident only ever had TWO cards working. ⭐ **Decode tolerates caps far better than training** (memory-bandwidth-bound, concave curve). ⚠⚠ **ana-ml3's Ada is sm_89: native FP8 but NO NVFP4** (Blackwell-only) — most of our in-house quants are NVFP4 and will NOT run accelerated there; ana-ml3's seats want FP8 W8A8, or NVFP4 checkpoints stay on fv-ml1. ⭐ **This unparks [[parked_triton_backend_ampere_fp8]]** — a hard no on Ampere (fp8e4nv unsupported sm_86), explicitly deferred TO Ada, and sm_89 has the FP8 support it needs. **ana-ml3 VRAM is 2x48 = 96 GB** vs fv-ml1's 391 GB, so big-model placement stays at FV (Flash-Next needs 74 GiB resident on ONE card — the offload moves the table, not the experts — so it cannot run on a 48 GB Ada card at all). ⚠ **PERSIST the cap** (systemd unit + persistence mode, ordered before Docker): a hand-set limit stops holding at the next reboot, which is likely to be the very power event it existed to prevent. → `docs/runbooks/fv-site-dark-20260913.md` - `[2026-09-13]` ⚠⚠⚠ **FV SITE DARK — every Fountain Valley address including the BMC went unreachable ~2.5 min into a two-card load test; all other sites healthy. Operator's leading hypothesis: the 1500 VA Eaton UPS overloaded and DIED.** It fits better than a breaker trip because a UPS's output rating sits far below the circuit's, making it the first protective device to give — which explains why the site let go at **two** cards loaded rather than four, and why the ~25 W OPNsense box died with it. ⚠ **Will not self-recover** (tripped needs a human, dead needs replacing) — do NOT poll FV. ⚠ **Do NOT use surge-only outlets to exceed a UPS rating**: both banks share one NEMA 5-15P inlet rated 12 A total; the surge bank bypasses the inverter, not the current limit. ⭐ **Recover `/tank/aimodels/flash-next-mtp-bench/power.log` FIRST** — all four cards every 10 s to the cut, on `/tank` not in a container, and the ONLY load measurement that exists. ⚠ **19 of 30 gateway aliases dark and NO local fallback** — every free local model was on fv-ml1, irv-ml1 runs no chat seat at all; the only non-fv chat backends are paid, and any coverage must be a NEW opt-in alias, never a silent repoint. ⭐ **OOB design gap**: OPNsense-as-subnet-router covers box-down/gateway-up and nothing for a site-wide loss, since the BMC's only route out is that gateway. ⚠ Recovery hazard: ten `restart: unless-stopped` vLLM containers will all load at once on power-up — mask Docker first, then `compose up -d` seat by seat (which also finishes the stale-homepage-label fix, since labels attach only at creation). → `docs/runbooks/fv-site-dark-20260913.md`, `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md` - `[2026-09-13]` ⭐⭐ **Qwen3.8-Flash-Next serving on ONE card with its 51B n-gram table in host RAM — the first seat whose weights do not fit its GPU.** `stacks/flash-next-seat/`, fv-ml1 GPU 2 `:8022`, plus a `gen-large` LiteLLM alias. Measured: 74.36 GiB weights resident, 14.00 GiB KV = 560,654 tokens at the full 262,144 context, 67 GiB host RSS, 75.5/212.3/387.8 tok/s at conc 1/4/8 (⚠ n=1). ⭐ The offload is vLLM **#54371 (UVA, merged 2026-09-09)** which **supersedes the paused #53899** — it has no worker process, so #53899's whole bug family (TP=1 deadlock #53960, `pidfd_getfd`/ptrace gate, stale-output-under-graphs) is designed out; in `v0.29.1rc0`, **not** `v0.29.0`. ⚠ **`text_config.ple_embedding_dtype` is the load-or-fail discriminator** for any community build. ⚠⚠ **`--kv-cache-memory` makes vLLM SKIP MEMORY PROFILING and ignore `--gpu-memory-utilization`** — 16 GiB nearly OOM'd on a 155K prefill with no visible failure; 14 GiB is the measured-safe value and vLLM's own "17.46 GiB to fully utilize" is 3.5 GiB too high. ⚠ MTP is off **pending measurement here, not written off** — the recipe's number is cross-harness and tested k=3 only, while the head is ONE layer run autoregressively, so k=1 is unpublished and may win (`services/flash-next-mtp-bench/`, one `off_A` rep banked before the outage). ⚠ A container once ran `(healthy)` with `PORTS=[]` — verify `docker port`, not the healthcheck. → `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md`