diff --git a/docs/pfi/opnsense-api-reference.md b/docs/pfi/opnsense-api-reference.md new file mode 100644 index 0000000..c8d718a --- /dev/null +++ b/docs/pfi/opnsense-api-reference.md @@ -0,0 +1,104 @@ +# OPNsense API — vendored reference + field-shape notes + +**Source:** (fetched 2026-09-13) +**Why vendored:** the upstream page documents the REST conventions but explicitly +*not* parameter shapes — "the auto-generated API documentation captures endpoints +and HTTP methods but not all parameter details." Those shapes are where the time +goes, so the hard-won ones are recorded in § Field shapes below. + +## Authentication + +HTTP Basic auth: **key = username, secret = password**. + +```sh +curl -s -u "$KEY:$SECRET" "http:///api///" +``` + +Fleet creds are vaulted: `fv-gateway/opnsense-api-key`, `fv-gateway/opnsense-api-secret` +(see `secret get`). Authorization is per-user **Effective Privileges** — a key only +reaches endpoints its owner is authorized for. + +## Conventions + +- **Endpoint pattern:** `/api////[/...]` +- **GET** retrieves; **POST** creates, updates, or executes an action. +- Request bodies and responses are `application/json`. + +Search endpoints take a paging body and return a rows envelope: + +```json +{"current":1,"rowCount":7,"sort":{},"searchPhrase":""} +``` +```json +{"total":10,"rowCount":7,"current":1, + "rows":[{"id":"configd","locked":1,"running":1,"description":"System Configuration Daemon","name":"configd"}]} +``` + +## Service control + +`/api//service/{status,start,stop,restart,reconfigure}` — `reconfigure` +writes config **and** applies it, which is normally the one you want after a +`settings/set`. Observed status values include `disabled`, `stopped`, `running` +— note **`disabled` ≠ `stopped`**: a disabled service will not start until its +model's `enabled` field is set to `"1"`. + +## Field shapes — the part upstream does not document + +⚠ **A `settings/get` response is NOT a valid `settings/set` body.** They are +different shapes, and mixing them returns a bare +`{"errorMessage":"Unexpected error, check log for details"}` with no indication +of which field was wrong. + +**Selection / option fields.** `get` returns every option with a `selected` flag; +`set` wants only the selected key. + +```jsonc +// get +"useExitNode": {"": {"value": "None", "selected": 1}, "abc": {"value":"node-a","selected":0}} +// set +"useExitNode": "" +``` + +**Array / list fields are UUID-keyed objects, not lists.** This is the one that +costs an afternoon. `get` renders an empty array as `[]`, which misleads you into +POSTing a list. + +```jsonc +// get, when empty -> looks like a plain list +"subnets": {"subnet4": []} +// set -> object keyed by a UUID you generate, values are the item's own fields +"subnets": {"subnet4": {"": {"subnet": "10.251.0.0/16", "description": "..."}}} +``` + +**Discovering an array item's inner fields — two reliable tricks:** + +1. POST the wrong shape on purpose. Validation names the full path: + `{"result":"failed","validations":{"settings.subnets.subnet4..subnet":"A value is required."}}` +2. Many models expose a template getter: `/api//settings/get_` + returns the blank item, e.g. `get_subnet` → `{"subnet4":{"subnet":"","description":""}}` + +**Partial `set` is supported** — POST only the fields you are changing rather than +round-tripping the whole model. This sidesteps every other field's shape problem +and is the recommended approach for a surgical change. + +**Validation error format:** `{"result":"failed","validations":{"":""}}`. +A successful save is `{"result":"saved"}`. + +## Working example — enabling a subnet router (2026-09-13, FV gateway) + +```jsonc +POST /api/tailscale/settings/set +{"settings":{"enabled":"1", + "subnets":{"subnet4":{"":{"subnet":"10.251.0.0/16", + "description":"FV site subnet router"}}}}} +POST /api/tailscale/service/reconfigure {} +``` + +**Always read back.** `settings/get` after the write, and re-check anything the +change could disturb (here: `/api/firewall/source_nat/search_rule` total stayed +at 2). A write that reports success is not evidence the value landed. + +## Related + +- `docs/runbooks/fv-to-ana-nat.md` — the FV gateway's manual SNAT rule +- `docs/runbooks/fv-site-dark-20260913.md` — the outage this reference came out of diff --git a/persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md b/persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md new file mode 100644 index 0000000..643390f --- /dev/null +++ b/persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md @@ -0,0 +1,97 @@ +# 2026-09-14 — fv-ml1 seat reorganization + the gen-large "orca" blocker + +An all-night GPU-seat overhaul on fv-ml1 after the FV colo recovered. Everything +here is on the running box; regenerate the authoritative view with +`scripts/seat-inventory.py` (reads the live containers). Cross-refs: +[[2026-09-13-flash-next-seat-and-fv-outage]]. + +## Seat topology now (2026-09-14 ~01:40 PT) + +| GPU | seats | +|---|---| +| 0 | `mog-sec` (`sec` :8019, dflash k=7) · **`sentinel-r3`** (`sentinel-r3` :8025, dflash k=7 — NEW) | +| 1 | **`meromero-charrp`** (`char-rp` :8016, MeroMero-v2-31B dense — restored) · `erp-seat` (`char-rp-fast` :8021) · reward · coder · embed · rerank | +| 2 | `flash-next` (`gen-large` :8022) — **DOWN** (orca swap failed; dealignai stopped) | +| 3 | **RESERVED scratch** — empty, operator directive; benches/quants/probes only | + +## What changed tonight + +1. **flash-next gained MTP k=3.** Campaign in `services/flash-next-mtp-bench/` + measured MTP a WIN on this hardware (+29/41/27% at k1, +42/52/38% k2, +52/51/34% + k3 across conc 1/4/8), inverting vLLM's 4×H100 recipe. k=3 deployed (single-user + → conc=1 dominates). KV pinned 14→10 GiB (MTP adds 5.08 GiB draft-head weights; + 14 GiB OOMs). Warm decode ~121 tok/s. `stacks/flash-next-seat/compose.yaml`. +2. **gen consolidation.** All 8 `gen`/`gen-reasoning`/`summarizer`/`summarizer-large`/ + `classifier`/`chat-judge`/`image-judge`/`qwen-image-bench` LiteLLM entries repointed + to flash-next (:8022); the **27B dense gen seat RETIRED**, freeing 38.4 GB on GPU0. + ⚠ judge aliases now score against different weights — prior scores incomparable. +3. **char-rp restored to MeroMero-v2-31B.** Was serving a leftover-test RedHatAI 26B + MoE W4A4; operator wanted the in-house dense-31B heretic W4A16 back. `char-rp-fast` + (:8021, the 26B MoE) is the deliberate speed tier — the throughput answer, so + char-rp stays the quality seat. `stacks/meromero-charrp/`. +4. **Sentinel-R3 A/B + dflash.** `glyphsoftware/sentinel-r3` (proprietary license — + operator's call) is a REAL SFT pentest finetune vs mog-sec's persona-on-stock. + Served alongside mog-sec for A/B. Then measured dflash vs MTP on Sentinel's + finetuned body: **dflash 2.40 vs MTP 2.18 mean acceptance length (+11%)**, warm + decode ~121 tok/s (faster than sec ~102). dflash k=7 cut over. `stacks/sentinel-r3/`. + ⚠ MEASUREMENT LESSON (again): first decode bench read 39 tok/s — a COLD-boot + + concurrent-contention artifact; warm+isolated it was 121. Operator caught it by + testing the running `sec` (102 tok/s) as reference. + +## THE BLOCKER — gen-large "orca" swap (resume here) + +Operator wants gen-large on **`orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4`** +(weight-only compressed-tensors: NVFP4 experts + FP8 attn + bf16 PLE) instead of the +incumbent **dealignai** build (modelopt_fp4 **W4A4** — 4-bit activations, the +long-context degradation mode). orcarouter is the trusted author (= `gen`), better on +the activation axis. Downloaded to `/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4` +(170 GB, complete, 18 shards + MTP head, integrity-verified). + +**IT DOES NOT LOAD.** Root cause (read the loader source, definitive): +`Qwen4ExpPLEEmbeddingMethod.from_quant_config` in vLLM — even the LATEST mainline +nightly — has branches only for **ModelOpt** and **FP8** configs; for a +CompressedTensorsConfig it raises `NotImplementedError` BEFORE ever consulting the +ignore list. So: +- dealignai (modelopt_fp4) loads fine; orcarouter (compressed-tensors) cannot, on any + mainline vLLM. +- orcarouter's config also does NOT list its `ngram_embedding` in `ignore` (would need + adding) — but that's moot until the loader has a compressed-tensors branch. +- orcarouter's OWN card prescribes **TP=4 + `--enable-expert-parallel`** and + `vllm>=` — a LITERAL UNFILLED version placeholder. They + shipped a build whose runtime they couldn't pin. + +**Backport vetted → UNFIT.** `wtdcode/vllm-backport` (docker `lazymio/vllm-backport`, +212★, Apache-2.0, DCO, active) resolved this exact error (issue #72) BUT: its purpose is +OLD GPUs (images sm86/sm80/sm89 only — **no Blackwell/sm_120**), it's built around its +OWN `wtdcode/Qwen3.8-Flash-Next-AWQ-W4A16` quant not orcarouter's NVFP4, and Blackwell +has native FP4/FP8 so the whole backport premise is moot for us. Using it = source-build +for an untested arch to load a quant it never tested. + +**Options (operator decision pending):** +- (a) **Restore dealignai** — works today, identical architecture, ~90% of the quality + (loses only the weight-only-vs-W4A4 activation edge). The pragmatic interim. +- (b) Cherry-pick the ~10-line compressed-tensors PLE-loader branch onto a mainline + source build for Blackwell (mainline already has Blackwell FP4 compute). More + auditable than the backport, still a from-source build + maintenance. +- (c) Request orcarouter's own **W4A4** build (they offer it) — would load on the current + mainline image like dealignai. +- (d) Wait for mainline vLLM to merge compressed-tensors qwen4_exp. + +⚠ Operator said "don't reload dealignai unprompted, priority orca" — but orca needs +runtime work, so gen-large is intentionally DOWN pending the decision. Both configs +backed up on host: `compose.yaml.bak-20260914-orcarouter`, `.env.bak-20260914-orcarouter`. + +## Other open items + +- **cyberprev quant** (3rd sec candidate `hotdogs/Qwen3.8-27B-abliterated-cyber-preview`, + bf16 at `/tank/aimodels/cyberprev-bf16`): restart crashed on a transformers + head-count config error in `quant-work/.venv` — the SAME venv that reached 49/65 on + Sept 11, so the original run likely used the canonical vllm-image+llmcompressor path + (per `services/gen-seat-mixed-quant/README.md`), not the venv. Retry that way. +- **sec vs sentinel-r3 quality A/B** — both live and gateway-callable; operator to judge. +- ⚠ **Suspect vault entry**: `fv-gateway/infra-ops-password` is 16 chars matching a + boot-UUID prefix exactly — possibly malformed. Eyeball. +- **os-nut not installed** on the FV OPNsense — with the firewall now on the 5P1000 UPS, + a NUT transfer-to-battery event is a direct "breaker tripped" alarm nobody gets today. +- **Branch breaker rating + 4-card ammeter reading** still open — every power table is + arithmetic on an ESTIMATED ~300 W platform draw. The ammeter converts it to fact. diff --git a/persistent-memory.md b/persistent-memory.md index 52e7250..80d488f 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-13 ~00:40 PT (⚠⚠ FV COLO DARK since 06:56Z — undersized UPS under two-card load; 19 of 30 gateway aliases down, no local fallback; operator on site 2026-09-14 with PDU + ammeter. Flash-Next seat + gen-large built and verified before the outage.)_ +_Last updated: 2026-09-14 ~01:40 PT (FV RECOVERED 2026-09-13 midday; all-night fv-ml1 seat reorg — MTP on gen-large, gen consolidated, char-rp→MeroMero-v2, Sentinel-R3+dflash live. ⚠ gen-large DOWN: orca swap blocked on vLLM compressed-tensors qwen4_exp support.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -114,112 +114,57 @@ no longer deployed sidecars here. See Recent decisions.) (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`. ## Current state / in-flight -_As of 2026-09-13 ~00:40 PT._ -### ⚠⚠ FV COLO IS DARK — site visit 2026-09-14. Nothing else matters until it is back. +_As of 2026-09-14 ~01:40 PT._ -Dark since **2026-09-13 06:56Z**. Every Fountain Valley address unreachable **including -the BMC** (its only route out is the same OPNsense gateway, also dark); all three other -sites healthy. **It will not self-recover — do not poll FV addresses.** +**FV colo RECOVERED** 2026-09-13 midday (operator on site). Power split: fv-ml1 chassis +on **raw PDU**, OPNsense firewall on the **Eaton 5P1000 (1000 VA / 800 W** — corrected +from the "1500 VA" the outage note assumed). Root cause confirmed from +`/tank/aimodels/flash-next-mtp-bench/power.log`: a SECOND GPU ramping 12→241 W took the +box to ~800 W into the 800 W UPS. GPU caps **275 W/card = 1100 W** aggregate +(`gpu-power-cap.service`, before docker). NH3→FV routing loop fixed +(`fv-loop-guard.service` on nh3-scale — an `unreachable 10.251.0.0/16` guard, inert while +the mesh serves the route). Full recovery detail in +`docs/runbooks/fv-site-dark-20260913.md`; the site-visit measurements (breaker rating, +4-card ammeter) remain open — every power figure is arithmetic on an estimated platform draw. -**19 of 30 gateway aliases are down and there is NO local fallback** — every free local -model lived on fv-ml1, and irv-ml1 runs no chat seat at all. The only non-fv chat -backends are paid (z.ai, Moonshot). Any paid coverage must be a NEW opt-in alias, never a -silent repoint of `summarizer`/`gen`/`classifier`. +### ⚠ THE RESUME POINT — gen-large is DOWN -Cause per operator ruling: **the 1500 VA Eaton UPS was undersized** and gave way under -two-card GPU load. Arithmetic: 2x Blackwell Max-Q @300 W ≈ 600 W of card + ~200-350 W -host ≈ **800-950 W against the unit's real ~900-1200 W** — at or over the line, which is -what explains a full day on ONE card and failure minutes into the SECOND. +Operator wants gen-large swapped from the incumbent **dealignai** W4A4 build to +**orcarouter** weight-only NVFP4. Downloaded (170 GB, verified) but **it does not load**: +no mainline vLLM (even latest nightly) supports the compressed-tensors qwen4_exp PLE +loader — it raises `NotImplementedError` before checking the ignore list. Backport vetted +→ unfit (old-hardware fork, no Blackwell image, its own AWQ quant). Options: restore +dealignai (works today), cherry-pick the ~10-line PLE-loader branch onto a mainline +source build, request orcarouter's own W4A4, or wait for mainline. **Decision pending.** +Operator: don't reload dealignai unprompted — so gen-large stays down. Both configs +backed up on host. → `persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md` -⭐ **Everything needed for the visit — the bring-list, the measurement protocol, the -staged recovery, the three circuit questions, the split-power proposal and the open -design gaps — is in `docs/runbooks/fv-site-dark-20260913.md`. Read that, not this.** -The two highest-value items from it: recover `/tank/aimodels/flash-next-mtp-bench/power.log` -BEFORE anything else (the only load measurement that exists, GPU-only, written locally -every 10 s), and **mask Docker before the network comes up** — ten -`restart: unless-stopped` vLLM containers loading at once is the largest transient the -box can make, into whatever just failed. +### fv-ml1 seat topology now +- **GPU0:** `sec`/mog-sec (:8019, dflash k=7) · `sentinel-r3` (:8025, dflash k=7 — NEW) +- **GPU1:** `char-rp`/MeroMero-v2-31B (:8016, restored from a leftover 26B) · `char-rp-fast`/erp-seat (:8021) · reward · coder · embed · rerank +- **GPU2:** `gen-large`/flash-next (:8022) — **DOWN** +- **GPU3:** RESERVED scratch (empty; benches/quants/probes only) -### Built today, dark with the box, resumes on recovery -- **`stacks/flash-next-seat/`** — Qwen3.8-Flash-Next (abliterated) on fv-ml1 GPU 2 - `:8022`, the first seat whose weights do not fit its card: 176B total with a 51B n-gram - table pinned in host RAM and read over CUDA UVA. Measured and verified serving before - the outage — 74.36 GiB resident, 14.00 GiB KV = 560,654 tokens at the full 262,144 - context, `gen-large` answering through the gateway in 0.4 s. - → `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md` -- **`services/flash-next-mtp-bench/`** — the MTP campaign, INCOMPLETE. One clean `off_A` - rep banked (75.5 / 212.3 / 387.8 tok/s at conc 1/4/8) before the power failed. The - hypothesis is live and unpublished: the head is ONE layer run autoregressively, the - vLLM recipe only ever tested k=3, so **k=1 may win.** ⚠ Do NOT restart it during - recovery — it is the prime suspect for the outage. -- **Power caps DECIDED, unexecuted** (box dark): fv-ml1 **250 W**/card, ana-ml3 **200 W**. - ⚠ `--kv-cache-memory`-style gotcha applies: `nvidia-smi -pl` caps BOARD power, so - 4x250 + host lands ~1310-1445 W at the plug against a 15 A circuit's 1440 W continuous - limit — **verify at the plug, fall back to 200 W if it reads near the limit.** -- **Stale homepage labels on 10 containers** — the compose files are fixed, the containers - died before recreation, and a plain power-on will NOT apply labels. The staged - `compose up -d` recovery sequence fixes them as a side effect. +Detail + the full change list (MTP campaign, gen consolidation freeing 38 GB, char-rp +restore, Sentinel dflash cutover, the orca blocker, cyberprev quant crash, open A/B): +→ `persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md` -### FV pre-outage state, for reference -- fv-ml1 (ex ana-ml2) at `10.251.50.54` / mesh `100.64.0.7`; **vb-gateway** OPNsense - `10.251.50.1` / `100.64.0.8`; **BMC** `10.251.250.50`. Public `fv.phasefinal.com` → - `172.83.89.66`. ⭐ **4x RTX PRO 6000 Blackwell Max-Q @300 W = 391 GB VRAM** (every doc - said two cards until 2026-09-12). → `persistent-memory.d/2026-09-12-fv-cutover-executed.md` -- ✅ The old "fv-ml1 cannot initiate to fleet LAN IPs" gap is **CLOSED** — another session - landed a scoped OPNsense hybrid SNAT for fv-ml1→ANA at 06:22Z and verified Beszel 18/18. - → `docs/runbooks/fv-to-ana-nat.md` (that session's file, uncommitted, left alone) -- ⚠ Still open, blocked on the box: delete plaintext creds `/tmp/io.pw` + `/tmp/key.io` - (fv-ml1) and `/tmp/opn.pw` (nh3-dev) — all vaulted under `fv-gateway/`, read-back - verified. Revoke `headscale/preauth-fv-{router,client}-7d-20260912` (expire 2026-09-19). - Close the FV WAN `InfraOps` rule (scoped to alias `fleet_egress`) once done with it. -- ⚠ Eight stacks deliberately NOT pushed from canonical during the renumber because their - host copies have genuine drift; three of those (`mistral-medium-3.5`, `ms32-24b-angel`, - `qwen35-vl`) are **untracked host-only stacks** that should be brought into `stacks/`. +### Infrastructure built this session +- **`vh/infra-reference`** (new gitea repo) — vendor API docs + hardware facts, "read the + reference BEFORE the API call" rule. Seeded: OPNsense, Proxmox, headscale, NVIDIA. +- **`scripts/seat-inventory.py`** — generates `docs/pfi/fv-ml1-gpu-seat-inventory.md` from + the LIVE box (never hand-maintained); `--check` drift mode wired to a daily user-systemd + timer on nh3-dev (`seat-inventory-drift.timer`, 09:15, althing alarm). +- **Secrets shed** — 24 OPNsense config.xml dumps + 3 plaintext passwords + API creds + from a prior session's scratchpad; post-sweep clean. Rotation of the OPNsense API creds + was BELAYED by operator (I leaked them to a /tmp file for ~1 min during the FV repair). -### ana-ml3 — not yet deployed -- 2x **Ada Generation RTX 6000** @300 W, 2x48 = 96 GB. Lands in the **Anaheim** rack whose - breaker tripped 2026-08-26 and 2026-09-11, so the 200 W cap is remediation, not caution. - ⚠⚠ **sm_89 has native FP8 but NO NVFP4** — most in-house quants are NVFP4 and will not - run accelerated there; its seats want FP8 W8A8, or those checkpoints stay at FV. ⭐ It - unparks the triton-backend item (hard no on Ampere, `fp8e4nv` unsupported on sm_86, - explicitly deferred TO Ada). - -### BabyYarros — complete; pair-corpus rebuild is the next step -- Both arms trained + evaluated; Instruct renders beats 9/10 by a **lexical** metric that - overstates it (echo-contaminated). Measured on the operator's own Janis beat: the beat's - grammar controls tense (0/4 → 4/4 present), anti-echo prompting fails 11/12, and prompt - tuning alone gets ~1-2/4 usable. → `persistent-memory.d/2026-09-11-babyyarros-*.md` -- **Agreed direction:** rebuild the corpus as instruction PAIRS (option C), with only the - INSTRUCTION side LLM-generated (backtranslated beats over real Yarros paragraphs) — - generating responses would distil away the voice. Measured supply: only 318 single - paragraphs are 90-140w, so the pair unit must be a ~4-paragraph scene window - (6,445 non-overlapping, 88% corpus coverage). Critical path is response-only loss - masking in `train_voice_lora.py` (currently `labels = ids.clone()`), ~1 day. -- ⛔ Frozen adjudication still deferred — needs the gen seat, which is at FV and dark. - -### Quants — cyber-preview to re-run -- **sentinel-r3** NVFP4 complete at `/tank/aimodels/sentinel-r3-nvfp4-mixed`; acceptance - A-B still needs a serving slot. **cyber-preview** died mid-quant in the Anaheim outage — - re-runnable once FV is back. - -### Jetson AGX Orin — 64 GB, in hand, unassigned -- Operator has the 64 GB dev kit plus ~6 undeployed cameras, a depth camera and lidar for - an unbuilt robot. House runs **Home Assistant + UniFi Protect, not Frigate**; Protect - exposes RTSPS per-channel (verified, currently off). Recommendation on the table: the - Orin is the **robot brain** (Isaac ROS), not the house computer — ESH already owns four - idle hardware decoders and the house voice stack is CPU-work that belongs beside HA on - esh-docker-vm. Blocked on which depth camera and lidar. Not decided. - -### eBay listing pipeline — scoped, unstarted -- Opto-mechanical resale. Verified eBay's **Sell Inventory API** is the right surface - (bulk item/offer/publish, 25/call; requires Business Policies opt-in) and that - `getItemAspectsForCategory` turns listing generation into schema population. Plan: - EXIF-gap clustering → VLM identification → Booth ask for approval → bulk publish. - Nothing built. ## Recent decisions +- `[2026-09-14]` fv-ml1 all-night seat reorg — MTP k=3 on gen-large (+52%@conc1), gen consolidated onto flash-next (27B dense retired, 38 GB freed), char-rp restored to MeroMero-v2-31B, Sentinel-R3 served + dflash cutover (beat MTP 2.40 vs 2.18). ⚠ gen-large DOWN: orcarouter swap blocked — no mainline vLLM loads compressed-tensors qwen4_exp; backport vetted unfit. → `persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md` + - `[2026-09-13]` ⭐ **STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint.** ⭐ **Two DIFFERENT boxes, clarified by operator 2026-09-13:** **fv-ml1** = 4x **Blackwell** RTX PRO 6000 **Max-Q @ 300 W** (Max-Q is the reduced-TGP SKU; Workstation Edition is 600 W), 391 GB VRAM, deployed. **ana-ml3** = 2x **Ada Generation** RTX 6000 @ 300 W, 96 GB VRAM, **NOT YET DEPLOYED**. **CAPS: fv-ml1 → 250 W/card (83% of TGP, ~5% throughput cost); ana-ml3 → 200 W/card (67%, ~10-15%).** ⚠⚠⚠ **`nvidia-smi -pl` caps BOARD power, not WALL power** — 4x250 = 1000 W board + ~180-300 W host components = 1180-1300 W, ÷ ~0.90 PSU efficiency = **~1310-1445 W AT THE PLUG vs a 15 A circuit's 1440 W NEC continuous limit. 250 W lands ON the line, not under it** (200 W would give ~1090-1220 W, comfortable). **Procedure: set 250 W, then VERIFY at the plug under four-card load; fall back to 200 W if it reads near 1440 W.** ⚠ Caps bound SUSTAINED draw, not transients — breakers tolerate brief overload, UPS overload protection does not, so 250 W implicitly commits the fv-ml1 chassis to the PDU rather than behind the 1500 VA unit (even capped it exceeds that UPS). ⭐ **BANK-level cap asked about and answered:** `DCGM_CONFIG_POWER_BUDGET_GROUP` ("power budget for the entire group") exists alongside `DCGM_CONFIG_POWER_CAP_INDIVIDUAL`, but the docs do not say how it distributes — and the only primitive underneath is NVML's PER-GPU `nvmlDeviceSetPowerManagementLimit` (**no bank-level register**), so any group budget resolves to N per-GPU writes. "Each card free until all are loaded" is therefore a CONTROL LOOP, not a hardware feature. ⚠⚠ If one is written it must be **safe-by-default, opportunistic upward** — boot at budget/N and only RAISE after observing idle neighbours; a reactive loop overshoots during a load RAMP, which is the all-cards-at-once case it exists to prevent. **Not yet worth it: 4x250 = 1000 W IS the bank budget**, so the daemon's whole prize is the one-card-busy case (~17% more board power ≈ ~5% throughput) — rare for a SERVING fleet (one seat per card), valuable for a TRAINING window. **DCGM** = NVIDIA's first-party **Data Center GPU Manager** (Apache-2.0), layered ABOVE NVML: nvidia-smi → NVML per-GPU primitives → DCGM daemon/dcgmi for health, diagnostics, config enforcement, groups. ✅ **VERIFIED 2026-09-13 (I had claimed the opposite and was WRONG): DCGM SUPPORTS our cards and power config works on them.** Supported platforms cover *"All NVIDIA Maxwell and newer NON-DATACENTER (e.g. GeForce or Quadro) GPUs"*, and the feature-overview table marks **Configuration Management ✓ for Tesla/Titan/Quadro/GeForce alike** — including *"Power Limit: set the maximum allowed power consumption"*. What IS gated on non-datacenter cards is **diagnostics** (Level 1 only vs All Levels on Tesla), not config. ⚠ Soft edge: the table says "Quadro", the former professional-line name; RTX 6000 Ada / RTX PRO 6000 are its successors and should fall in that column, but the table predates the rename — one command on the box settles it. ⭐ So the group-budget test IS worth running; what remains unsettled is *distribution*, not availability. ✅ **The static cap needs none of it**: `nvidia-smi -pl 250` is plain NVML and works on these cards; DCGM is probably not even installed (beszel-agent-nvidia shells out to nvidia-smi). ⚠ ana-ml3's 200 W on 2 cards (2x200 = 400 W) — and ⭐ ana-ml3 lands in the **Anaheim** rack whose breaker tripped 2026-08-26 and 2026-09-11, one of those caused by this very chassis before it relocated, so the cap there is remediation of a known-bad circuit, not precaution. 4x200 W = 800 W of card, which fits a real circuit with a real UPS. This is the generalised lesson of the FV outage: decide the power envelope first and size the cards into it. At 300 W TGP, 200 W is a **67% cap — the favourable part of the concave perf/watt curve, ~10-15% throughput cost**, not the severe 33% cap a 600 W part would have meant; and 200 W is very unlikely to sit below a 300 W card's enforceable floor (still confirm with `nvidia-smi -q -d POWER | grep -iE 'power limit|default'`). ⭐⭐ **The outage arithmetic now has numbers:** 2x Blackwell Max-Q @300 W ≈ 600 W of card + host (566 GB RAM, drives, fans, PSU losses) ≈ 200-350 W = **~800-950 W against a 1500 VA Eaton's real ~900-1200 W rating** — at or just over the line, which is what explains a full day on ONE card (~500-650 W, inside) and death minutes into the SECOND. The host term is the only guess; idle-at-the-plug measures it. ⚠⚠ **And FOUR cards is a BREAKER problem, not a UPS problem:** 4x300 + ~300 host ≈ **1500 W vs a 15 A circuit's 1440 W continuous (80%) derating** — so **capping belongs at fv-ml1 too**, or it needs a 20 A feed, before anyone loads all four cards again. Today's incident only ever had TWO cards working. ⭐ **Decode tolerates caps far better than training** (memory-bandwidth-bound, concave curve). ⚠⚠ **ana-ml3's Ada is sm_89: native FP8 but NO NVFP4** (Blackwell-only) — most of our in-house quants are NVFP4 and will NOT run accelerated there; ana-ml3's seats want FP8 W8A8, or NVFP4 checkpoints stay on fv-ml1. ⭐ **This unparks [[parked_triton_backend_ampere_fp8]]** — a hard no on Ampere (fp8e4nv unsupported sm_86), explicitly deferred TO Ada, and sm_89 has the FP8 support it needs. **ana-ml3 VRAM is 2x48 = 96 GB** vs fv-ml1's 391 GB, so big-model placement stays at FV (Flash-Next needs 74 GiB resident on ONE card — the offload moves the table, not the experts — so it cannot run on a 48 GB Ada card at all). ⚠ **PERSIST the cap** (systemd unit + persistence mode, ordered before Docker): a hand-set limit stops holding at the next reboot, which is likely to be the very power event it existed to prevent. → `docs/runbooks/fv-site-dark-20260913.md` - `[2026-09-13]` ⚠⚠⚠ **FV SITE DARK — every Fountain Valley address including the BMC went unreachable ~2.5 min into a two-card load test; all other sites healthy. Operator's leading hypothesis: the 1500 VA Eaton UPS overloaded and DIED.** It fits better than a breaker trip because a UPS's output rating sits far below the circuit's, making it the first protective device to give — which explains why the site let go at **two** cards loaded rather than four, and why the ~25 W OPNsense box died with it. ⚠ **Will not self-recover** (tripped needs a human, dead needs replacing) — do NOT poll FV. ⚠ **Do NOT use surge-only outlets to exceed a UPS rating**: both banks share one NEMA 5-15P inlet rated 12 A total; the surge bank bypasses the inverter, not the current limit. ⭐ **Recover `/tank/aimodels/flash-next-mtp-bench/power.log` FIRST** — all four cards every 10 s to the cut, on `/tank` not in a container, and the ONLY load measurement that exists. ⚠ **19 of 30 gateway aliases dark and NO local fallback** — every free local model was on fv-ml1, irv-ml1 runs no chat seat at all; the only non-fv chat backends are paid, and any coverage must be a NEW opt-in alias, never a silent repoint. ⭐ **OOB design gap**: OPNsense-as-subnet-router covers box-down/gateway-up and nothing for a site-wide loss, since the BMC's only route out is that gateway. ⚠ Recovery hazard: ten `restart: unless-stopped` vLLM containers will all load at once on power-up — mask Docker first, then `compose up -d` seat by seat (which also finishes the stale-homepage-label fix, since labels attach only at creation). → `docs/runbooks/fv-site-dark-20260913.md`, `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md` diff --git a/stacks/sentinel-r3/.env.example b/stacks/sentinel-r3/.env.example new file mode 100644 index 0000000..c4d9a85 --- /dev/null +++ b/stacks/sentinel-r3/.env.example @@ -0,0 +1,10 @@ +SENTINEL_IMAGE=vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013 +API_KEY= +SENTINEL_GPU_ID=0 +SENTINEL_PORT=8025 +SENTINEL_MODEL=/tank/aimodels/sentinel-r3-nvfp4-mixed +SENTINEL_GPU_MEM_UTIL=0.40 +SENTINEL_MAX_MODEL_LEN=163840 +SENTINEL_KV_CACHE_MEMORY=8589934592 +SENTINEL_MAX_NUM_BATCHED_TOKENS=4096 +SENTINEL_DRAFT_MODEL=/tank/aimodels/qwen38-27b-dflash2-drafter diff --git a/stacks/sentinel-r3/compose.yaml b/stacks/sentinel-r3/compose.yaml new file mode 100644 index 0000000..5c2c8d0 --- /dev/null +++ b/stacks/sentinel-r3/compose.yaml @@ -0,0 +1,124 @@ +# sentinel-r3 — Sentinel-R3 pen-test seat, A/B candidate ALONGSIDE mog-sec on fv-ml1 GPU 0. +# +# glyphsoftware/sentinel-r3: a REAL SFT finetune of stock Qwen/Qwen3.8-27B on 1,230 +# authorized-pentest agent trajectories (recon -> foothold -> privesc -> writeup) over a +# 19-tool surface that matches our own harness. Contrast mog-sec, which is a persona +# system prompt on stock weights. Quantized in-house to the same mixed NVFP4 W4A4(MLP +# 0-55) + FP8 W8A8(attn/lm_head/MLP 56-63) recipe as mog-sec/gen. → sentinel-r3-nvfp4-mixed.PROVENANCE.txt +# +# ⚠ PROPRIETARY LICENSE (Glyph Proprietary v1.0) — operator's fair-use/licensee call, unlike +# mog-sec's Apache. Served here on operator instruction 2026-09-14. +# +# ⚠ dflash speculative decoding ENABLED 2026-09-14 after measurement (see the spec-config note +# below). Originally served without spec to establish a clean baseline; the probe measured +# dflash k=7 at 2.40 acceptance length vs MTP k=3 at 2.18 on this finetuned body. +# (historical) SERVED WITHOUT --speculative-config ON PURPOSE. The MTP head is a VERBATIM base graft +# (source shipped zero mtp.*), and its acceptance on this SFT-finetuned body is UNVERIFIED — +# the gate is a measured >=~40% on a probe serve, not an assumption. A clean no-spec boot +# establishes the quality baseline for the mog-sec A/B first; MTP acceptance is a separate +# measurement (add the qwen3_5_mtp spec-config and read the acceptance metric). +# +# ⚠ max-model-len 163840, NOT native 262K. Sentinel is the SAME base + hybrid Qwen3_5 arch as +# mog-sec, so the identical deep-context lesson applies: what the KV pool HOLDS and what the +# card PROCESSES at depth are different numbers, and mog-sec crashed five times before 163840 +# bought a clean 400-refusal above the measured ceiling instead of an engine death. Do not +# raise without re-running the deep-ctx probe on THIS checkpoint. +# +# ⚠ Serve with a PROSE system prompt — Sentinel was trained on prose tool descriptions, not +# structured `tools=`. + +name: sentinel-r3 + +services: + vllm-sentinel-r3: + image: ${SENTINEL_IMAGE:-vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013} + container_name: ${SENTINEL_CONTAINER_NAME:-vllm-sentinel-r3} + restart: unless-stopped + ipc: host + ports: + - "${SENTINEL_PORT:-8025}:8000" + volumes: + - /tank/aimodels/huggingface:/hfcache + - ${SENTINEL_MODEL:-/tank/aimodels/sentinel-r3-nvfp4-mixed}:/model:ro + - ${SENTINEL_DRAFT_MODEL:-/tank/aimodels/qwen38-27b-dflash2-drafter}:/drafter:ro + environment: + - VLLM_API_KEY=${API_KEY:-} + command: + - /model + - --served-model-name + - ${SENTINEL_SERVED_NAME:-sentinel-r3} + - ${SENTINEL_SERVED_NAME_THINK:-sentinel-r3-thinking} + - --host + - 0.0.0.0 + - --port + - "8000" + - --quantization + - ${SENTINEL_QUANT:-compressed-tensors} + - --gpu-memory-utilization + - "${SENTINEL_GPU_MEM_UTIL:-0.40}" + # KV pinned in bytes — same discipline as mog-sec/erp-seat: a ratio yields a different + # cache depending on what else is resident at start, an explicit figure is reproducible. + # 8 GiB is generous for an A/B probe (conc 1/4/8 short prompts never approach it). + - --kv-cache-memory + - "${SENTINEL_KV_CACHE_MEMORY:-8589934592}" + - --max-model-len + - "${SENTINEL_MAX_MODEL_LEN:-163840}" + - --max-num-seqs + - "${SENTINEL_MAX_NUM_SEQS:-16}" + - --max-num-batched-tokens + - "${SENTINEL_MAX_NUM_BATCHED_TOKENS:-4096}" + - --trust-remote-code + - --dtype + - auto + - --mamba-cache-dtype + - float32 + - --kv-cache-dtype + - ${SENTINEL_KV_CACHE_DTYPE:-fp8} + - --enable-prefix-caching + - --enable-chunked-prefill + - --limit-mm-per-prompt + - '${SENTINEL_LIMIT_MM:-{"image": 4}}' + - --mm-processor-kwargs + - '${SENTINEL_MM_PROCESSOR_KWARGS:-{"size": {"longest_edge": 4194304, "shortest_edge": 65536}}}' + - --reasoning-parser + - ${SENTINEL_REASONING_PARSER:-qwen3} + - --default-chat-template-kwargs + - '{"reasoning_effort": "${SENTINEL_REASONING_EFFORT:-medium}"}' + - --enable-auto-tool-choice + - --tool-call-parser + - qwen3_coder + # dflash speculative decoding — MEASURED 2026-09-14 on THIS finetuned body: dflash k=7 + # gives mean acceptance length 2.40 vs MTP k=3 at 2.18 (+11%, clean n=2 separation). + # dflash wins on reach (accepts a longer tail to k=7) despite equal ~60% position-1 + # acceptance. The drafter is the same qwen38-27b-dflash2-drafter mog-sec uses (Sentinel + # shares its 64-layer Qwen3.8-27B base). Costs +3.6 GB for the drafter. + - --speculative-config + - '${SENTINEL_SPEC_CONFIG:-{"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7}}' + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: + - "${SENTINEL_GPU_ID:-0}" + capabilities: + - gpu + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 900s + networks: + - tnet + labels: + - homepage.group=AI - Inference + - homepage.name=Sentinel-R3 (pen-test A/B) + - homepage.icon=mdi-shield-search + - homepage.description=SFT pen-test finetune of Qwen3.8-27B, A/B candidate vs mog-sec (fv-ml1 GPU 0) + - homepage.href=http://10.251.50.54:${SENTINEL_PORT:-8025}/docs + +networks: + tnet: + name: traefik-net + external: true