diff --git a/docs/runbooks/fv-site-dark-20260913.md b/docs/runbooks/fv-site-dark-20260913.md index ef5a72e..659c60e 100644 --- a/docs/runbooks/fv-site-dark-20260913.md +++ b/docs/runbooks/fv-site-dark-20260913.md @@ -510,13 +510,32 @@ control loop rather than a hardware feature. Its "group" notion is therefore a management-layer abstraction over per-GPU NVML calls, which is why the bank budget still resolves to N per-GPU writes underneath. -⚠⚠ **DO NOT PLAN AROUND DCGM HERE. It is datacenter-oriented and parts of it are gated -to datacenter SKUs** (Tesla / A100 / H100 class). Our cards are **professional / -workstation** parts — RTX PRO 6000 Blackwell Max-Q and RTX 6000 Ada — and several DCGM -capabilities are unsupported or degraded outside the datacenter line, plausibly including -**config enforcement, which is exactly the power path**. The -`DCGM_CONFIG_POWER_BUDGET_GROUP` experiment may simply return "unsupported on this -device": five minutes of curiosity after the real work, not a planning assumption. +✅ **VERIFIED 2026-09-13 — DCGM DOES SUPPORT OUR CARDS, and an earlier caveat in this +runbook claiming otherwise was WRONG and has been removed.** + +Supported platforms, quoted: *"All NVIDIA Maxwell™ and newer **non-datacenter** (e.g. +NVIDIA® GeForce® or NVIDIA® Quadro®) GPUs"* — plus *"Starting with v1.3, limited DCGM +functionality is available on non-datacenter GPUs."* + +And the feature-overview table settles what "limited" excludes — **not** configuration: + + Feature Group Tesla Titan Quadro GeForce + Configuration Management X X X X + +Configuration Management explicitly includes *"Power Limit: Set the maximum allowed power +consumption."* The thing actually gated on non-datacenter cards is **diagnostics**: + + GPU Diagnostics (Levels 1,2,3): All Levels [Tesla]; Level 1 [Titan/Quadro/GeForce] + +⚠ **One soft edge:** the table says "Quadro", the former name for the professional line. +RTX 6000 Ada and RTX PRO 6000 are its successors and should fall in that column, but the +table predates the rename — so that last step is inference, settled by one command on the +box. + +⭐ **So the group-budget test is worth actually running**, not a curiosity. What it does +NOT settle is *distribution*: the docs still never say how a group budget is divided, and +the NVML argument is untouched — no bank-level register means per-GPU writes either way, +so the likely finding is static even division (= 4 x 250 W). ✅ **The static cap needs none of this.** `nvidia-smi -pl 250` is plain NVML and works on these cards. DCGM would only buy the group-budget experiment and richer telemetry, and is diff --git a/persistent-memory.md b/persistent-memory.md index 1deab54..348193e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -207,7 +207,7 @@ last, and do NOT restart the MTP campaign. ## Recent decisions -- `[2026-09-13]` ⭐ **STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint.** ⭐ **Two DIFFERENT boxes, clarified by operator 2026-09-13:** **fv-ml1** = 4x **Blackwell** RTX PRO 6000 **Max-Q @ 300 W** (Max-Q is the reduced-TGP SKU; Workstation Edition is 600 W), 391 GB VRAM, deployed. **ana-ml3** = 2x **Ada Generation** RTX 6000 @ 300 W, 96 GB VRAM, **NOT YET DEPLOYED**. **CAPS: fv-ml1 → 250 W/card (83% of TGP, ~5% throughput cost); ana-ml3 → 200 W/card (67%, ~10-15%).** ⚠⚠⚠ **`nvidia-smi -pl` caps BOARD power, not WALL power** — 4x250 = 1000 W board + ~180-300 W host components = 1180-1300 W, ÷ ~0.90 PSU efficiency = **~1310-1445 W AT THE PLUG vs a 15 A circuit's 1440 W NEC continuous limit. 250 W lands ON the line, not under it** (200 W would give ~1090-1220 W, comfortable). **Procedure: set 250 W, then VERIFY at the plug under four-card load; fall back to 200 W if it reads near 1440 W.** ⚠ Caps bound SUSTAINED draw, not transients — breakers tolerate brief overload, UPS overload protection does not, so 250 W implicitly commits the fv-ml1 chassis to the PDU rather than behind the 1500 VA unit (even capped it exceeds that UPS). ⭐ **BANK-level cap asked about and answered:** `DCGM_CONFIG_POWER_BUDGET_GROUP` ("power budget for the entire group") exists alongside `DCGM_CONFIG_POWER_CAP_INDIVIDUAL`, but the docs do not say how it distributes — and the only primitive underneath is NVML's PER-GPU `nvmlDeviceSetPowerManagementLimit` (**no bank-level register**), so any group budget resolves to N per-GPU writes. "Each card free until all are loaded" is therefore a CONTROL LOOP, not a hardware feature. ⚠⚠ If one is written it must be **safe-by-default, opportunistic upward** — boot at budget/N and only RAISE after observing idle neighbours; a reactive loop overshoots during a load RAMP, which is the all-cards-at-once case it exists to prevent. **Not yet worth it: 4x250 = 1000 W IS the bank budget**, so the daemon's whole prize is the one-card-busy case (~17% more board power ≈ ~5% throughput) — rare for a SERVING fleet (one seat per card), valuable for a TRAINING window. ⚠⚠ **DO NOT plan around DCGM** (= NVIDIA's first-party **Data Center GPU Manager**, Apache-2.0, layered ABOVE NVML: nvidia-smi → NVML per-GPU primitives → DCGM daemon/dcgmi for health, diagnostics, config enforcement, groups): **it is datacenter-SKU-oriented and parts are gated to Tesla/A100/H100-class cards.** Ours are professional/workstation parts (RTX PRO 6000 Max-Q, RTX 6000 Ada), so config enforcement — the power path — may return "unsupported on this device". Test it for 5 min out of curiosity, not as a plan. ✅ **The static cap needs none of it**: `nvidia-smi -pl 250` is plain NVML and works on these cards; DCGM is probably not even installed (beszel-agent-nvidia shells out to nvidia-smi). ⚠ ana-ml3's 200 W on 2 cards (2x200 = 400 W) — and ⭐ ana-ml3 lands in the **Anaheim** rack whose breaker tripped 2026-08-26 and 2026-09-11, one of those caused by this very chassis before it relocated, so the cap there is remediation of a known-bad circuit, not precaution. 4x200 W = 800 W of card, which fits a real circuit with a real UPS. This is the generalised lesson of the FV outage: decide the power envelope first and size the cards into it. At 300 W TGP, 200 W is a **67% cap — the favourable part of the concave perf/watt curve, ~10-15% throughput cost**, not the severe 33% cap a 600 W part would have meant; and 200 W is very unlikely to sit below a 300 W card's enforceable floor (still confirm with `nvidia-smi -q -d POWER | grep -iE 'power limit|default'`). ⭐⭐ **The outage arithmetic now has numbers:** 2x Blackwell Max-Q @300 W ≈ 600 W of card + host (566 GB RAM, drives, fans, PSU losses) ≈ 200-350 W = **~800-950 W against a 1500 VA Eaton's real ~900-1200 W rating** — at or just over the line, which is what explains a full day on ONE card (~500-650 W, inside) and death minutes into the SECOND. The host term is the only guess; idle-at-the-plug measures it. ⚠⚠ **And FOUR cards is a BREAKER problem, not a UPS problem:** 4x300 + ~300 host ≈ **1500 W vs a 15 A circuit's 1440 W continuous (80%) derating** — so **capping belongs at fv-ml1 too**, or it needs a 20 A feed, before anyone loads all four cards again. Today's incident only ever had TWO cards working. ⭐ **Decode tolerates caps far better than training** (memory-bandwidth-bound, concave curve). ⚠⚠ **ana-ml3's Ada is sm_89: native FP8 but NO NVFP4** (Blackwell-only) — most of our in-house quants are NVFP4 and will NOT run accelerated there; ana-ml3's seats want FP8 W8A8, or NVFP4 checkpoints stay on fv-ml1. ⭐ **This unparks [[parked_triton_backend_ampere_fp8]]** — a hard no on Ampere (fp8e4nv unsupported sm_86), explicitly deferred TO Ada, and sm_89 has the FP8 support it needs. **ana-ml3 VRAM is 2x48 = 96 GB** vs fv-ml1's 391 GB, so big-model placement stays at FV (Flash-Next needs 74 GiB resident on ONE card — the offload moves the table, not the experts — so it cannot run on a 48 GB Ada card at all). ⚠ **PERSIST the cap** (systemd unit + persistence mode, ordered before Docker): a hand-set limit stops holding at the next reboot, which is likely to be the very power event it existed to prevent. → `docs/runbooks/fv-site-dark-20260913.md` +- `[2026-09-13]` ⭐ **STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint.** ⭐ **Two DIFFERENT boxes, clarified by operator 2026-09-13:** **fv-ml1** = 4x **Blackwell** RTX PRO 6000 **Max-Q @ 300 W** (Max-Q is the reduced-TGP SKU; Workstation Edition is 600 W), 391 GB VRAM, deployed. **ana-ml3** = 2x **Ada Generation** RTX 6000 @ 300 W, 96 GB VRAM, **NOT YET DEPLOYED**. **CAPS: fv-ml1 → 250 W/card (83% of TGP, ~5% throughput cost); ana-ml3 → 200 W/card (67%, ~10-15%).** ⚠⚠⚠ **`nvidia-smi -pl` caps BOARD power, not WALL power** — 4x250 = 1000 W board + ~180-300 W host components = 1180-1300 W, ÷ ~0.90 PSU efficiency = **~1310-1445 W AT THE PLUG vs a 15 A circuit's 1440 W NEC continuous limit. 250 W lands ON the line, not under it** (200 W would give ~1090-1220 W, comfortable). **Procedure: set 250 W, then VERIFY at the plug under four-card load; fall back to 200 W if it reads near 1440 W.** ⚠ Caps bound SUSTAINED draw, not transients — breakers tolerate brief overload, UPS overload protection does not, so 250 W implicitly commits the fv-ml1 chassis to the PDU rather than behind the 1500 VA unit (even capped it exceeds that UPS). ⭐ **BANK-level cap asked about and answered:** `DCGM_CONFIG_POWER_BUDGET_GROUP` ("power budget for the entire group") exists alongside `DCGM_CONFIG_POWER_CAP_INDIVIDUAL`, but the docs do not say how it distributes — and the only primitive underneath is NVML's PER-GPU `nvmlDeviceSetPowerManagementLimit` (**no bank-level register**), so any group budget resolves to N per-GPU writes. "Each card free until all are loaded" is therefore a CONTROL LOOP, not a hardware feature. ⚠⚠ If one is written it must be **safe-by-default, opportunistic upward** — boot at budget/N and only RAISE after observing idle neighbours; a reactive loop overshoots during a load RAMP, which is the all-cards-at-once case it exists to prevent. **Not yet worth it: 4x250 = 1000 W IS the bank budget**, so the daemon's whole prize is the one-card-busy case (~17% more board power ≈ ~5% throughput) — rare for a SERVING fleet (one seat per card), valuable for a TRAINING window. **DCGM** = NVIDIA's first-party **Data Center GPU Manager** (Apache-2.0), layered ABOVE NVML: nvidia-smi → NVML per-GPU primitives → DCGM daemon/dcgmi for health, diagnostics, config enforcement, groups. ✅ **VERIFIED 2026-09-13 (I had claimed the opposite and was WRONG): DCGM SUPPORTS our cards and power config works on them.** Supported platforms cover *"All NVIDIA Maxwell and newer NON-DATACENTER (e.g. GeForce or Quadro) GPUs"*, and the feature-overview table marks **Configuration Management ✓ for Tesla/Titan/Quadro/GeForce alike** — including *"Power Limit: set the maximum allowed power consumption"*. What IS gated on non-datacenter cards is **diagnostics** (Level 1 only vs All Levels on Tesla), not config. ⚠ Soft edge: the table says "Quadro", the former professional-line name; RTX 6000 Ada / RTX PRO 6000 are its successors and should fall in that column, but the table predates the rename — one command on the box settles it. ⭐ So the group-budget test IS worth running; what remains unsettled is *distribution*, not availability. ✅ **The static cap needs none of it**: `nvidia-smi -pl 250` is plain NVML and works on these cards; DCGM is probably not even installed (beszel-agent-nvidia shells out to nvidia-smi). ⚠ ana-ml3's 200 W on 2 cards (2x200 = 400 W) — and ⭐ ana-ml3 lands in the **Anaheim** rack whose breaker tripped 2026-08-26 and 2026-09-11, one of those caused by this very chassis before it relocated, so the cap there is remediation of a known-bad circuit, not precaution. 4x200 W = 800 W of card, which fits a real circuit with a real UPS. This is the generalised lesson of the FV outage: decide the power envelope first and size the cards into it. At 300 W TGP, 200 W is a **67% cap — the favourable part of the concave perf/watt curve, ~10-15% throughput cost**, not the severe 33% cap a 600 W part would have meant; and 200 W is very unlikely to sit below a 300 W card's enforceable floor (still confirm with `nvidia-smi -q -d POWER | grep -iE 'power limit|default'`). ⭐⭐ **The outage arithmetic now has numbers:** 2x Blackwell Max-Q @300 W ≈ 600 W of card + host (566 GB RAM, drives, fans, PSU losses) ≈ 200-350 W = **~800-950 W against a 1500 VA Eaton's real ~900-1200 W rating** — at or just over the line, which is what explains a full day on ONE card (~500-650 W, inside) and death minutes into the SECOND. The host term is the only guess; idle-at-the-plug measures it. ⚠⚠ **And FOUR cards is a BREAKER problem, not a UPS problem:** 4x300 + ~300 host ≈ **1500 W vs a 15 A circuit's 1440 W continuous (80%) derating** — so **capping belongs at fv-ml1 too**, or it needs a 20 A feed, before anyone loads all four cards again. Today's incident only ever had TWO cards working. ⭐ **Decode tolerates caps far better than training** (memory-bandwidth-bound, concave curve). ⚠⚠ **ana-ml3's Ada is sm_89: native FP8 but NO NVFP4** (Blackwell-only) — most of our in-house quants are NVFP4 and will NOT run accelerated there; ana-ml3's seats want FP8 W8A8, or NVFP4 checkpoints stay on fv-ml1. ⭐ **This unparks [[parked_triton_backend_ampere_fp8]]** — a hard no on Ampere (fp8e4nv unsupported sm_86), explicitly deferred TO Ada, and sm_89 has the FP8 support it needs. **ana-ml3 VRAM is 2x48 = 96 GB** vs fv-ml1's 391 GB, so big-model placement stays at FV (Flash-Next needs 74 GiB resident on ONE card — the offload moves the table, not the experts — so it cannot run on a 48 GB Ada card at all). ⚠ **PERSIST the cap** (systemd unit + persistence mode, ordered before Docker): a hand-set limit stops holding at the next reboot, which is likely to be the very power event it existed to prevent. → `docs/runbooks/fv-site-dark-20260913.md` - `[2026-09-13]` ⚠⚠⚠ **FV SITE DARK — every Fountain Valley address including the BMC went unreachable ~2.5 min into a two-card load test; all other sites healthy. Operator's leading hypothesis: the 1500 VA Eaton UPS overloaded and DIED.** It fits better than a breaker trip because a UPS's output rating sits far below the circuit's, making it the first protective device to give — which explains why the site let go at **two** cards loaded rather than four, and why the ~25 W OPNsense box died with it. ⚠ **Will not self-recover** (tripped needs a human, dead needs replacing) — do NOT poll FV. ⚠ **Do NOT use surge-only outlets to exceed a UPS rating**: both banks share one NEMA 5-15P inlet rated 12 A total; the surge bank bypasses the inverter, not the current limit. ⭐ **Recover `/tank/aimodels/flash-next-mtp-bench/power.log` FIRST** — all four cards every 10 s to the cut, on `/tank` not in a container, and the ONLY load measurement that exists. ⚠ **19 of 30 gateway aliases dark and NO local fallback** — every free local model was on fv-ml1, irv-ml1 runs no chat seat at all; the only non-fv chat backends are paid, and any coverage must be a NEW opt-in alias, never a silent repoint. ⭐ **OOB design gap**: OPNsense-as-subnet-router covers box-down/gateway-up and nothing for a site-wide loss, since the BMC's only route out is that gateway. ⚠ Recovery hazard: ten `restart: unless-stopped` vLLM containers will all load at once on power-up — mask Docker first, then `compose up -d` seat by seat (which also finishes the stale-homepage-label fix, since labels attach only at creation). → `docs/runbooks/fv-site-dark-20260913.md`, `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md` - `[2026-09-13]` ⭐⭐ **Qwen3.8-Flash-Next serving on ONE card with its 51B n-gram table in host RAM — the first seat whose weights do not fit its GPU.** `stacks/flash-next-seat/`, fv-ml1 GPU 2 `:8022`, plus a `gen-large` LiteLLM alias. Measured: 74.36 GiB weights resident, 14.00 GiB KV = 560,654 tokens at the full 262,144 context, 67 GiB host RSS, 75.5/212.3/387.8 tok/s at conc 1/4/8 (⚠ n=1). ⭐ The offload is vLLM **#54371 (UVA, merged 2026-09-09)** which **supersedes the paused #53899** — it has no worker process, so #53899's whole bug family (TP=1 deadlock #53960, `pidfd_getfd`/ptrace gate, stale-output-under-graphs) is designed out; in `v0.29.1rc0`, **not** `v0.29.0`. ⚠ **`text_config.ple_embedding_dtype` is the load-or-fail discriminator** for any community build. ⚠⚠ **`--kv-cache-memory` makes vLLM SKIP MEMORY PROFILING and ignore `--gpu-memory-utilization`** — 16 GiB nearly OOM'd on a 155K prefill with no visible failure; 14 GiB is the measured-safe value and vLLM's own "17.46 GiB to fully utilize" is 3.5 GiB too high. ⚠ MTP is off **pending measurement here, not written off** — the recipe's number is cross-harness and tested k=3 only, while the head is ONE layer run autoregressively, so k=1 is unpublished and may win (`services/flash-next-mtp-bench/`, one `off_A` rep banked before the outage). ⚠ A container once ran `(healthy)` with `PORTS=[]` — verify `docker port`, not the healthcheck. → `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md`