From cd941f7bf93bf0e3a85d26fadb98bf7eb8ec62b3 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 14 Sep 2026 11:00:32 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20fv-ml1=20rebal?= =?UTF-8?q?ance=20(cyberprev=E2=86=92sec,=20gen-small=20A3B,=20all=20seats?= =?UTF-8?q?=20native=20262K)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-clear snapshot. Rewrote Current state / in-flight to the post-rebalance topology (cyberprev is the sec seat, gen-small stood up, char/coder retuned, all generative seats native 262K in-band, catalog+bench shipped) and added a Recent decisions index entry + detail file 2026-09-14-fv-seat-rebalance-gen-small.md. Also commits two previously-untracked durable artifacts (fv-to-ana-nat runbook + its memory detail) so memory does not lag the work. Archival: file is over the ~300-line soft cap (377) but 0 entries archived — the 4 age-eligible (>14d) entries all carry open deferred-work pointers (guarded), and all other entries are <14 days old. Reported per the archival discipline. --- docs/runbooks/fv-to-ana-nat.md | 68 ++++++++++++ .../2026-09-13-fv-to-ana-nat.md | 15 +++ .../2026-09-14-fv-seat-rebalance-gen-small.md | 60 +++++++++++ persistent-memory.md | 101 ++++++------------ 4 files changed, 174 insertions(+), 70 deletions(-) create mode 100644 docs/runbooks/fv-to-ana-nat.md create mode 100644 persistent-memory.d/2026-09-13-fv-to-ana-nat.md create mode 100644 persistent-memory.d/2026-09-14-fv-seat-rebalance-gen-small.md diff --git a/docs/runbooks/fv-to-ana-nat.md b/docs/runbooks/fv-to-ana-nat.md new file mode 100644 index 0000000..e72621e --- /dev/null +++ b/docs/runbooks/fv-to-ana-nat.md @@ -0,0 +1,68 @@ +# FV to Anaheim: scoped Tailscale NAT repair + +Applied 2026-09-13 ~06:22Z after operator approval with explicit instruction +not to strand the subnet. Gateway: OPNsense at 10.251.50.1, current hostname +vb-gateway.internal; tailscale0 address 100.64.0.8, assigned interface opt6/MESH. + +## Persistent configuration + +Firewall / NAT / Outbound: **Hybrid** mode, retaining every automatic WAN rule. +One manual IPv4 rule: + +- Interface: MESH (opt6 / tailscale0) +- Source: 10.251.50.54/32 (fv-ml1 only) +- Destination: 10.250.0.0/16 (Anaheim only) +- Translation: interface address (100.64.0.8 currently) +- Description: `fv-ml1 to ANA via Tailscale SNAT` + +No routes, access/filter rules, WAN settings, subnet advertisements, host +Tailscale settings, or other FV source addresses were changed. Other remote +sites remain outside this fix's scope. Translation means Anaheim services may +see the gateway identity rather than fv-ml1's original LAN address. + +## Evidence and verification + +Before: requests reached ANA with source 10.251.50.54; ANA sent SYN-ACKs then +received resets attributed to that address, while FV host packet captures +showed neither incoming SYN-ACKs nor outgoing resets. Gateway itself got hub +HTTP200. Rules contained WAN NAT only, none on tailscale0. FV->ANA SSH, Beszel, +Restic and ICMP failed; ANA->FV SSH and ping worked. + +Temporary /32 host-to-hub NAT immediately fixed HTTP and restored Beszel. +Only NAT rules were loaded for this first test. Persistent /32-to-ANA-/16 rule +then saved using OPNsense write_config, followed by configctl filter reload. +Both stages had independent 240-second rollback watchdogs. Before disarming, +verified all preexisting NAT lines retained and pfctl -sr byte-identical. + +After: FV->hub HTTP200, FV->ANA PostgreSQL TCP5432 reachable, FV internet +HTTPS200, ANA->FV SSH reachable, gateway management intact. Beszel **18/18 up**. +Agent logs show WebSocket connected and intentional SSH listener shutdown; +45876 refusing after WebSocket connection is NOT a loss of host connectivity. +A single BMC ping to 10.251.50.50 did not answer; no prior BMC baseline was +taken, so BMC health is not claimed. The NAT rule excludes BMC source traffic. + +## Backup and rollback + +Root-only directory on gateway: `/root/fv-nat-repair-20260913/` contains +config.xml.before (secret-bearing, never commit), nat.before, filter.before, +nat-config.before.json, nat.after, nat-config.php and verified marker. +Automatic rollback disarmed only after verification; backup retained. + +Manual rollback, after checking for intervening NAT edits: + +```sh +php /root/fv-nat-repair-20260913/nat-config.php rollback +configctl filter reload +``` + +This restores the saved NAT config section, not unrelated configuration. +It will remove subsequent NAT edits too: inspect before using later. +Emergency runtime NAT-only restore: `pfctl -N -f /root/fv-nat-repair-20260913/nat.before`. +Never use pfctl -F all or flush all states for this repair. + +Gateway SSH key auth is unavailable; use vaulted fv-gateway/root-password or +approved infra-ops credentials, without logging secrets. This session used +/tmp/fv-elway.py (FreeBSD sh + legacy SCP + existing SSH control socket). + +References: [OPNsense hybrid NAT](https://docs.opnsense.org/manual/nat.html), +[Tailscale FreeBSD userspace routing](https://tailscale.com/docs/reference/kernel-vs-userspace-routers). diff --git a/persistent-memory.d/2026-09-13-fv-to-ana-nat.md b/persistent-memory.d/2026-09-13-fv-to-ana-nat.md new file mode 100644 index 0000000..5913f8c --- /dev/null +++ b/persistent-memory.d/2026-09-13-fv-to-ana-nat.md @@ -0,0 +1,15 @@ +# FV→ANA NAT repaired safely + +Operator approved narrow fix, caution not to strand subnet. OPNsense MESH/opt6 +(tailscale0 100.64.0.8) lacked outbound NAT for forwarded LAN traffic. Temporary +fv-ml1→hub /32 NAT proved diagnosis; persisted hybrid NAT rule source +10.251.50.54/32 destination 10.250.0.0/16 translate interface address. Existing +WAN NAT retained, filter rules byte-identical, no routes/mesh/host changes. +Both stages guarded by independent rollback timers, disarmed after verification. + +Verified hub HTTP200, ANA PostgreSQL TCP, Internet HTTPS200, reverse SSH, gateway +management; Beszel18/18 up. Agent intentionally stops SSH45876 when WebSocket +connects. BMC ping failed with no pre-change baseline; no BMC-health claim. +Other FV sources and other remote subnets not covered by this narrow fix. +Backup + rollback helper on gateway /root/fv-nat-repair-20260913. Full details: +docs/runbooks/fv-to-ana-nat.md. No commit made. diff --git a/persistent-memory.d/2026-09-14-fv-seat-rebalance-gen-small.md b/persistent-memory.d/2026-09-14-fv-seat-rebalance-gen-small.md new file mode 100644 index 0000000..6e0c306 --- /dev/null +++ b/persistent-memory.d/2026-09-14-fv-seat-rebalance-gen-small.md @@ -0,0 +1,60 @@ +# 2026-09-14 — fv-ml1 seat rebalance: cyberprev→sec, gen-small stood up, all seats to native 262K + +Second big fv-ml1 session on 2026-09-14 (after the all-night reorg + orca blocker in +[[2026-09-14-fv-seat-reorg-and-orca-blocker]]). Everything here is committed; regenerate +the live view with `scripts/seat-inventory.py` and the durable catalog is +`docs/pfi/llm-seat-catalog.md`. + +## What changed (all committed) + +1. **cyberprev quantized and promoted to the `sec` seat.** `hotdogs/Qwen3.8-27B-abliterated-cyber-preview` + (a cyber offensive-security **tool-calling SFT finetune** on an abliterated base — NOT + "an abliteration"; trained on `hotdogs/cyber-sft-agent-qwen38`, 8,400 rows / 22 tools). + ⚠ The raw hotdogs export ships **broken**: 850/1199 body tensors carry a **tripled + `language_model.` prefix** (unsloth export bug, verified in the upstream HF index) — + the naive quant passed every structural check and served pure gibberish. Fix: + name-repair the source against a known-good reference (exact 1199-tensor match), then + house mixed-NVFP4 quant → `cyberprev-nvfp4-mixed-v2`. **A coherence smoke is the gate; + structural verification cannot substitute** (see [[reference_unsloth_tripled_prefix_export_bug]]). + Commits 1418edb/b94d173/196416f. +2. **mog-sec RETIRED; cyberprev is now `sec`/`sec-reasoning`.** Gateway repointed (commit + 207798d). `sec` is a ROLE alias, so this is a promotion, not silent substitution. + ⚠ mog-sec was the fleet's ONLY offense+**defense**/blue-team seat — the sec role is now + offense-only. mog-sec stack files retained for a one-command restore. Its "persona on + stock" description was ALSO wrong — M.O.G.-SEC is itself an offense+defense SFT finetune + (commit 3906c68). All three sec candidates are Qwen3.8-27B finetunes differing by focus. +3. **gen-small stood up (NEW seat).** `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-NVFP4-Experts-Only` + — already on disk at `/tank/aimodels/qwen36-35b-a3b-heretic-nvfp4` (no download). Heretic + v1.3.0 (MPOA), 88% fewer refusals @ 0.0015 KL, 19 MTP. GPU0 :8026, MTP k=3 **measured + 69.6% accept / 3.09 len**. 3.8 was requested but no general Qwen3.8 A3B exists → 3.6 is + the fallback. Backs `summarizer` + `classifier`; new `classifier-large` → flash (accuracy + tier); `summarizer-large` stays on flash. Commit 05f3d3b. +4. **All sec/gen/char seats raised to native 262,144 with concurrency in the 1.2-2.5x band** + (operator target). coder: 8192@13.75x → 16384@4.70x (waste reclaimed; ~4.7x overshoots + 2-3x because the 1.5B weight floor sits under the util knob's resolution — a KV-byte pin + would fix it, deferred). char-rp bumped 1.02→1.22x (util 0.40→0.52); GPU1 hit its ceiling + so char-rp-fast's util cap was trimmed (its KV is pinned, so it held 2.04x) — an in-GPU + rebalance rather than a utility-seat move. +5. **Durable docs:** `docs/pfi/llm-seat-catalog.md` (curated lineage/provenance/cards/speed) + + `scripts/seat-bench.py` (reproducible warm-tok/s + depth-OOM bench). Commit dfa91a8. + +## Measured — serial depth + warm tok/s (2026-09-14, warm/conc=1/greedy/n=3) + +| seat | GPU | ctx | conc | warm tok/s | VRAM | depth clean to | OOM | +|---|---|---|---|---|---|---|---| +| cyberprev (sec) | 0 | 262144 | 1.37x | 136.6 | 47.1 GiB | 259,722 tok | none | +| gen-small | 0 | 262144 | 2.56x | 254.8 | 36.1 GiB | 254,526 tok | none | +| gen (flash-next) | 2 | 262144 | 1.31x | 170.7 | 95.3 GiB | 254,273 tok | none | +| char-rp | 1 | 262144 | 1.22x | 62.7 | 42.3 GiB | 254,858 tok | none | +| char-rp-fast | 1 | 262144 | 2.04x | 225.2 | 27.0 GiB | 254,823 tok | none | +| coder | 1 | 16384 | 4.70x | 337.3 | 5.6 GiB | 15,905 tok | none | + +Every seat prefilled near max context with **zero OOM/CUBLAS/illegal-memory** (allocator-log +verified). Numbers are uncontended single-stream ceilings — re-derive with `scripts/seat-bench.py`. + +## Open / deferred + +- **mog-sec fate** — retired; restore for blue-team coverage vs leave retired. Operator's call. +- **coder ≤3x** — needs a `--kv-cache-memory` byte pin (compose change) vs the fragile util floor. +- **gen-small 2.56x** — a hair over the 2.5x band top; trimmable, left as useful headroom for the bulk seat. +- **mog-sec / sentinel-r3 stacks** retained on disk for restore; sentinel gateway aliases deleted (they 404). diff --git a/persistent-memory.md b/persistent-memory.md index e1b6a67..2533f35 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-14 ~01:40 PT (FV RECOVERED 2026-09-13 midday; all-night fv-ml1 seat reorg — MTP on gen-large, gen consolidated, char-rp→MeroMero-v2, Sentinel-R3+dflash live. ✅ gen-large RESOLVED 2026-09-14: orca serving after a PLE bf16→FP8 convert + 2 config keys.)_ +_Last updated: 2026-09-14 ~10:30 PT (fv-ml1 FULLY REBALANCED: cyberprev quantized + promoted to the `sec` seat displacing mog-sec; NEW gen-small A3B seat (Qwen3.6-35B-A3B Heretic, MTP); all sec/gen/char seats at native 262K in the 1.2-2.5x band; coder reclaimed; curated `docs/pfi/llm-seat-catalog.md` + `scripts/seat-bench.py` shipped. All committed, serial depth-tested clean.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -115,83 +115,44 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-09-14 ~01:40 PT._ +_As of 2026-09-14 ~10:30 PT._ -**FV colo RECOVERED** 2026-09-13 midday (operator on site). Power split: fv-ml1 chassis -on **raw PDU**, OPNsense firewall on the **Eaton 5P1000 (1000 VA / 800 W** — corrected -from the "1500 VA" the outage note assumed). Root cause confirmed from -`/tank/aimodels/flash-next-mtp-bench/power.log`: a SECOND GPU ramping 12→241 W took the -box to ~800 W into the 800 W UPS. GPU caps **275 W/card = 1100 W** aggregate -(`gpu-power-cap.service`, before docker). NH3→FV routing loop fixed -(`fv-loop-guard.service` on nh3-scale — an `unreachable 10.251.0.0/16` guard, inert while -the mesh serves the route). Full recovery detail in -`docs/runbooks/fv-site-dark-20260913.md`; the site-visit measurements (breaker rating, -4-card ammeter) remain open — every power figure is arithmetic on an estimated platform draw. +**fv-ml1 seat topology is fully rebalanced and committed.** Every generative seat serves +its native context with concurrency in the operator's 1.2-2.5x band, serial depth-tested +clean (zero OOM). Live view: `scripts/seat-inventory.py` → `docs/pfi/fv-ml1-gpu-seat-inventory.md`; +durable lineage/provenance/speed: `docs/pfi/llm-seat-catalog.md`. -### ✅ RESOLVED — gen-large now serves orcarouter (was the "orca blocker") +| GPU | seat (gateway) | model | ctx | conc | tok/s | +|---|---|---|---|---|---| +| 0 | `sec`/`sec-reasoning` = **cyberprev** | Qwen3.8-27B cyber-SFT (abliterated base) | 262144 | 1.37x | 136.6 | +| 0 | `gen-small` (+`summarizer`,`classifier`) | Qwen3.6-35B-A3B Heretic (MTP 69.6%) | 262144 | 2.56x | 254.8 | +| 1 | `char-rp` | G4-MeroMero-v2-31B (Gemma4 dense, quality) | 262144 | 1.22x | 62.7 | +| 1 | `char-rp-fast` | G4-MeroMero-26B-A4B (Gemma4 MoE, speed) | 262144 | 2.04x | 225.2 | +| 1 | `coder`/`coder-fast` | Qwen2.5-Coder-1.5B | 16384 | 4.70x | 337.3 | +| 1 | reward / embed / rerank | Skywork-Reward / Qwen3-Embed / bge-rerank | — | — | — | +| 2 | `gen` (+large/reasoning/judges/summarizer-large/classifier-large) = **flash-next** | Qwen3.8-Flash-Next orcarouter (MTP) | 262144 | 1.31x | 170.7 | +| 3 | RESERVED scratch (empty) | — | — | — | — | -**gen-large (:8022) serves `orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4` as of -2026-09-14**, from `/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4-plefp8`. -Healthy, coherent, MTP k=3 accepting 60.4%. +⚠ **GPU2 (flash-next) and GPU3 (empty) are off-limits to rearrangement** (operator). GPU0/GPU1 +are the movable pair. Per-GPU: GPU0 83/98, GPU1 90/98, GPU2 95/98. -The blocker was real but **much cheaper to fix than the earlier note concluded** (it -proposed a from-source vLLM build). Two independent blockers, both config-level: - -1. **PLE loader.** `Qwen4ExpPLEEmbeddingMethod.from_quant_config` checks - `ple_embedding_dtype == "float8_e4m3fn"` as **branch 1, BEFORE any quant-config type - check**, and the `NotImplementedError` for CompressedTensorsConfig is **scoped to the - PLE path only** — experts/dense load through the ordinary compressed-tensors paths. - So declaring an FP8 PLE bypasses the blocker entirely. orca ships a **bf16** PLE, so - the fix was: convert the table bf16→FP8 (one shard file, 128 tensors, clean split) - and add the one config key. **No source build, no backport.** -2. **`ValueError: Invalid layer_type qwen_sparse_attention`.** orca labels its 12 QSA - layers `qwen_sparse_attention`; vLLM accepts only `linear_attention` / - `full_attention` and selects QSA via `indexer_n_heads`. Renamed the 12 entries. - ⚠ Verified `indexer_n_heads=4` in BOTH orca and dealignai first — without it the - rename would silently select PLAIN attention and serve a subtly wrong model. - -**Conversion, measured:** global amax 0.0894, per-shard outlier ratio only 1.66x, so one -global scale is well-conditioned. Scale chosen **exactly representable in bf16** -(2.002716e-04) so no scale-rounding error stacks on quantization; max maps to 446.17/448, -no clipping. Round-trip **2.655% RMS relative**, 0.002% underflow, 0 saturation. -MTP head (31 tensors, BF16) preserved. `weight_scale` BF16[1] matches gorbatjovy's format. - -**Measured on the live seat** (warm, conc=1, greedy 300-tok, n=5): median **167.5 tok/s** -(min 150.0 / max 170.4, spread 12.2%), MTP acceptance 60.4%, mean acceptance length 2.81, -KV 344,155 tokens @ 262,144 ctx, 1.31x concurrency. ⚠ The reorg note's dealignai figure -(~121 tok/s) came from a DIFFERENT harness — do not treat this as a measured win over -dealignai; a controlled A/B needs dealignai back on this same harness plus a noise floor. - -⚠⚠ **NO LOCAL ROLLBACK — dealignai weights DELETED 2026-09-14** on operator instruction -(125 GiB reclaimed, /tank 57% used). Reverting = re-downloading 126 GiB. The `.env` backup -names paths that no longer exist. ⚠ Not yet done, and now unbacked: quality A/B vs dealignai -(its reference arm is gone) and a 262K deep-prefill probe. The pristine 170 GiB orcarouter -download is RETAINED — it is what makes the PLE conversion redoable without re-fetching. -→ `persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md` - -### fv-ml1 seat topology now -- **GPU0:** `sec`/mog-sec (:8019, dflash k=7) · `sentinel-r3` (:8025, dflash k=7 — NEW) -- **GPU1:** `char-rp`/MeroMero-v2-31B (:8016, restored from a leftover 26B) · `char-rp-fast`/erp-seat (:8021) · reward · coder · embed · rerank -- **GPU2:** `gen-large`/flash-next (:8022) — **UP, orcarouter** (MTP k=3, 60.4% accept) -- **GPU3:** RESERVED scratch (empty; benches/quants/probes only) - -Detail + the full change list (MTP campaign, gen consolidation freeing 38 GB, char-rp -restore, Sentinel dflash cutover, the orca blocker, cyberprev quant crash, open A/B): -→ `persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md` - -### Infrastructure built this session -- **`vh/infra-reference`** (new gitea repo) — vendor API docs + hardware facts, "read the - reference BEFORE the API call" rule. Seeded: OPNsense, Proxmox, headscale, NVIDIA. -- **`scripts/seat-inventory.py`** — generates `docs/pfi/fv-ml1-gpu-seat-inventory.md` from - the LIVE box (never hand-maintained); `--check` drift mode wired to a daily user-systemd - timer on nh3-dev (`seat-inventory-drift.timer`, 09:15, althing alarm). -- **Secrets shed** — 24 OPNsense config.xml dumps + 3 plaintext passwords + API creds - from a prior session's scratchpad; post-sweep clean. Rotation of the OPNsense API creds - was BELAYED by operator (I leaked them to a /tmp file for ~1 min during the FV repair). +### Open loose ends (none blocking) +- **mog-sec fate** — retired to promote cyberprev into `sec`. It was the ONLY offense+**defense**/ + blue-team seat; the sec role is now offense-only. Stack files retained on disk; restore is a + one-command bring-up if blue-team coverage is wanted. **Operator's call.** +- **coder ~4.70x** overshoots the "2-3x" intent — the 1.5B weight+overhead floor sits under the + util knob's resolution; hitting ≤3x reliably needs a `--kv-cache-memory` byte pin (compose change). +- **gen-small 2.56x** is a hair over the 2.5x band top — trimmable, left as useful headroom for the + bulk summarizer/classifier seat. +- **sentinel-r3** stack retained on disk; its gateway aliases were deleted (they 404, no substitution). +- **Prior-session open items still open:** FV site-visit power measurements (breaker rating, 4-card + ammeter) in `docs/runbooks/fv-site-dark-20260913.md`; the deferred park/althing items in Recent decisions. ## Recent decisions +- `[2026-09-14]` **fv-ml1 rebalance: cyberprev→`sec` (mog-sec retired), NEW gen-small A3B seat, all sec/gen/char at native 262K in-band, coder reclaimed, seat catalog + bench shipped.** cyberprev = hotdogs cyber-SFT (name-repaired past a tripled-prefix unsloth export bug, house NVFP4 quant); gen-small = llmfan46 Qwen3.6-35B-A3B Heretic (already on disk), MTP 69.6%. Serial depth-tested all seats clean (0 OOM); warm tok/s 62.7-337.3. Commits 1418edb→dfa91a8. → `persistent-memory.d/2026-09-14-fv-seat-rebalance-gen-small.md` + - `[2026-09-14]` fv-ml1 all-night seat reorg — MTP k=3 on gen-large (+52%@conc1), gen consolidated onto flash-next (27B dense retired, 38 GB freed), char-rp restored to MeroMero-v2-31B, Sentinel-R3 served + dflash cutover (beat MTP 2.40 vs 2.18). ✅ gen-large RESOLVED 2026-09-14 — orcarouter serving: PLE bf16→FP8 convert + `ple_embedding_dtype` + `layer_types` rename; NO source build needed. → `persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md` - `[2026-09-13]` ⭐ **STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint.** ⭐ **Two DIFFERENT boxes, clarified by operator 2026-09-13:** **fv-ml1** = 4x **Blackwell** RTX PRO 6000 **Max-Q @ 300 W** (Max-Q is the reduced-TGP SKU; Workstation Edition is 600 W), 391 GB VRAM, deployed. **ana-ml3** = 2x **Ada Generation** RTX 6000 @ 300 W, 96 GB VRAM, **NOT YET DEPLOYED**. **CAPS: fv-ml1 → 250 W/card (83% of TGP, ~5% throughput cost); ana-ml3 → 200 W/card (67%, ~10-15%).** ⚠⚠⚠ **`nvidia-smi -pl` caps BOARD power, not WALL power** — 4x250 = 1000 W board + ~180-300 W host components = 1180-1300 W, ÷ ~0.90 PSU efficiency = **~1310-1445 W AT THE PLUG vs a 15 A circuit's 1440 W NEC continuous limit. 250 W lands ON the line, not under it** (200 W would give ~1090-1220 W, comfortable). **Procedure: set 250 W, then VERIFY at the plug under four-card load; fall back to 200 W if it reads near 1440 W.** ⚠ Caps bound SUSTAINED draw, not transients — breakers tolerate brief overload, UPS overload protection does not, so 250 W implicitly commits the fv-ml1 chassis to the PDU rather than behind the 1500 VA unit (even capped it exceeds that UPS). ⭐ **BANK-level cap asked about and answered:** `DCGM_CONFIG_POWER_BUDGET_GROUP` ("power budget for the entire group") exists alongside `DCGM_CONFIG_POWER_CAP_INDIVIDUAL`, but the docs do not say how it distributes — and the only primitive underneath is NVML's PER-GPU `nvmlDeviceSetPowerManagementLimit` (**no bank-level register**), so any group budget resolves to N per-GPU writes. "Each card free until all are loaded" is therefore a CONTROL LOOP, not a hardware feature. ⚠⚠ If one is written it must be **safe-by-default, opportunistic upward** — boot at budget/N and only RAISE after observing idle neighbours; a reactive loop overshoots during a load RAMP, which is the all-cards-at-once case it exists to prevent. **Not yet worth it: 4x250 = 1000 W IS the bank budget**, so the daemon's whole prize is the one-card-busy case (~17% more board power ≈ ~5% throughput) — rare for a SERVING fleet (one seat per card), valuable for a TRAINING window. **DCGM** = NVIDIA's first-party **Data Center GPU Manager** (Apache-2.0), layered ABOVE NVML: nvidia-smi → NVML per-GPU primitives → DCGM daemon/dcgmi for health, diagnostics, config enforcement, groups. ✅ **VERIFIED 2026-09-13 (I had claimed the opposite and was WRONG): DCGM SUPPORTS our cards and power config works on them.** Supported platforms cover *"All NVIDIA Maxwell and newer NON-DATACENTER (e.g. GeForce or Quadro) GPUs"*, and the feature-overview table marks **Configuration Management ✓ for Tesla/Titan/Quadro/GeForce alike** — including *"Power Limit: set the maximum allowed power consumption"*. What IS gated on non-datacenter cards is **diagnostics** (Level 1 only vs All Levels on Tesla), not config. ⚠ Soft edge: the table says "Quadro", the former professional-line name; RTX 6000 Ada / RTX PRO 6000 are its successors and should fall in that column, but the table predates the rename — one command on the box settles it. ⭐ So the group-budget test IS worth running; what remains unsettled is *distribution*, not availability. ✅ **The static cap needs none of it**: `nvidia-smi -pl 250` is plain NVML and works on these cards; DCGM is probably not even installed (beszel-agent-nvidia shells out to nvidia-smi). ⚠ ana-ml3's 200 W on 2 cards (2x200 = 400 W) — and ⭐ ana-ml3 lands in the **Anaheim** rack whose breaker tripped 2026-08-26 and 2026-09-11, one of those caused by this very chassis before it relocated, so the cap there is remediation of a known-bad circuit, not precaution. 4x200 W = 800 W of card, which fits a real circuit with a real UPS. This is the generalised lesson of the FV outage: decide the power envelope first and size the cards into it. At 300 W TGP, 200 W is a **67% cap — the favourable part of the concave perf/watt curve, ~10-15% throughput cost**, not the severe 33% cap a 600 W part would have meant; and 200 W is very unlikely to sit below a 300 W card's enforceable floor (still confirm with `nvidia-smi -q -d POWER | grep -iE 'power limit|default'`). ⭐⭐ **The outage arithmetic now has numbers:** 2x Blackwell Max-Q @300 W ≈ 600 W of card + host (566 GB RAM, drives, fans, PSU losses) ≈ 200-350 W = **~800-950 W against a 1500 VA Eaton's real ~900-1200 W rating** — at or just over the line, which is what explains a full day on ONE card (~500-650 W, inside) and death minutes into the SECOND. The host term is the only guess; idle-at-the-plug measures it. ⚠⚠ **And FOUR cards is a BREAKER problem, not a UPS problem:** 4x300 + ~300 host ≈ **1500 W vs a 15 A circuit's 1440 W continuous (80%) derating** — so **capping belongs at fv-ml1 too**, or it needs a 20 A feed, before anyone loads all four cards again. Today's incident only ever had TWO cards working. ⭐ **Decode tolerates caps far better than training** (memory-bandwidth-bound, concave curve). ⚠⚠ **ana-ml3's Ada is sm_89: native FP8 but NO NVFP4** (Blackwell-only) — most of our in-house quants are NVFP4 and will NOT run accelerated there; ana-ml3's seats want FP8 W8A8, or NVFP4 checkpoints stay on fv-ml1. ⭐ **This unparks [[parked_triton_backend_ampere_fp8]]** — a hard no on Ampere (fp8e4nv unsupported sm_86), explicitly deferred TO Ada, and sm_89 has the FP8 support it needs. **ana-ml3 VRAM is 2x48 = 96 GB** vs fv-ml1's 391 GB, so big-model placement stays at FV (Flash-Next needs 74 GiB resident on ONE card — the offload moves the table, not the experts — so it cannot run on a 48 GB Ada card at all). ⚠ **PERSIST the cap** (systemd unit + persistence mode, ordered before Docker): a hand-set limit stops holding at the next reboot, which is likely to be the very power event it existed to prevent. → `docs/runbooks/fv-site-dark-20260913.md`