diff --git a/docs/runbooks/fv-site-dark-20260913.md b/docs/runbooks/fv-site-dark-20260913.md index 127d0b1..d0ee7c7 100644 --- a/docs/runbooks/fv-site-dark-20260913.md +++ b/docs/runbooks/fv-site-dark-20260913.md @@ -1,6 +1,9 @@ # FV site dark — 2026-09-13 ~06:56Z -**Status: UNRESOLVED, needs hands at Fountain Valley.** Written while the site is +**Status: UNRESOLVED, WILL NOT SELF-RECOVER, needs hands at Fountain Valley.** +Operator 2026-09-13: an overload-tripped UPS does not clear itself and a dead one +needs replacing, so there is no recovery to wait for and no point polling the site. +Visit planned for 2026-09-14. Do not leave watchers running against FV addresses. Written while the site is down so recovery does not have to be reconstructed from memory. ## What is down @@ -138,9 +141,58 @@ much longer recovery. 4. **Do not restart the MTP campaign.** It is the prime suspect. 5. Watch power while seats come up: `nvidia-smi --query-gpu=index,power.draw --format=csv`. +### ⭐ The staged bring-up also finishes the homepage-label fix, for free + +The 2026-09-13 renumber (commit `3132a16`) repaired 25 compose files on this box but +the 10 RUNNING containers were never recreated, so their labels still carried the dead +10.250.50.54. Those containers are gone with the power loss. + +⚠ A PLAIN POWER-ON DOES NOT FIX THEM. `restart: unless-stopped` restarts the existing +container with its existing labels; labels only attach at container CREATION. But the +staged `docker compose up -d ` sequence above **is** a recreate, and the compose +files on disk are already corrected — so bringing seats up that way applies the new +labels as a side effect and the dashboard comes back correct. Bring them up with +`compose up -d`, not by letting Docker restore the old containers. + +Afterwards, confirm with: + + curl -s http://10.0.50.45:5100/api/services | \ + python3 -c 'import json,sys;d=json.load(sys.stdin);print([s["href"] for g in d for s in (g.get("services") or []) if "10.250.50.54" in (s.get("href") or "")])' + +Expect `[]`. Before the outage that query returned 16 entries. + **Do not reboot the OPNsense firewall** (standing operator directive; its reboot API 403s anyway). +## ⚠ PROPOSED, NOT RATIFIED — split the power so the management plane survives + +**This is a recommendation awaiting the operator's call, not settled intent.** Written +down so tomorrow's rebuild can adopt or reject it deliberately rather than restoring +the arrangement that just failed by default. + + UPS <- OPNsense firewall + fv-ml1 BMC only (tens of watts, long runtime) + PDU / wall <- GPU chassis (no UPS in series) + +Two reasons: + +1. **It fixes the OOB gap this outage exposed.** The BMC's only route to the fleet is + through the firewall, so a power event at the GPU box takes out the management plane + with it -- which is precisely why this incident needs a drive rather than a console + session. Separate the two and a repeat leaves a live firewall, a live BMC, and + remote eyes on a dark chassis. +2. **A 1500 VA unit was never going to hold this box.** It has four cards, not the two + every record claimed until 2026-09-12. + +⚠ **Do NOT use a UPS's surge-only outlets to get around its rating.** Both outlet banks +sit downstream of the same input cord, inlet and internal breaker; for a 120 V 1500 VA +unit that is a single NEMA 5-15P rated **12 A at maximum load**, total across all +outlets. The surge bank bypasses the inverter, not the current rating. Overloading the +inverter trips or kills the unit; overloading the cord is a thermal problem in an +unattended rack. Bypass the UPS entirely instead. + +If the GPU box is ever to go on battery, it is a 3000 VA / 2700 W-class unit and +probably a 20 A circuit -- but size it from `power.log`, not from a spec sheet. + ## Afterwards - ⭐ **The OOB design gap this exposes.** The cutover chose OPNsense-as-subnet-router diff --git a/persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md b/persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md new file mode 100644 index 0000000..8dc5960 --- /dev/null +++ b/persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md @@ -0,0 +1,130 @@ +# `[2026-09-13]` Flash-Next seat stood up on fv-ml1 — then the FV site went dark under two-card load + +Two things happened in one session and they are causally linked: a 176B model was put +on one GPU with its embedding table in host RAM, and the attempt to benchmark it on a +second GPU took Fountain Valley off the air. + +## What was built and is real + +**`stacks/flash-next-seat/` — Qwen3.8-Flash-Next (abliterated) on fv-ml1 GPU 2, `:8022`.** +The first seat on the fleet whose weights do not fit its card. 176B total = a 125B main +model + a **51B n-gram (PLE) lookup table** that lives in pinned host RAM and is read +by the GPU over **CUDA UVA**. Measured, not predicted: + + weights resident 74.36 GiB of 95.6 (load 97-242 s depending on page cache) + KV cache 14.00 GiB pinned in bytes -> 560,654 tokens + context 262,144 (native max), 2.14x concurrency + engine host RSS 67 GiB (47.7 GiB table + runtime) + throughput (n=1!) 75.5 / 212.3 / 387.8 tok/s at concurrency 1 / 4 / 8 + +Checkpoint `dealignai/Qwen3.8-Flash-Next-ABLITERATED-NVFP4` @ `be794b99`, 126.0 GiB at +`/tank/aimodels/qwen38-flash-next-abliterated-nvfp4`. Composition measured from the +manifest: **63.3 GiB NVFP4 W4A4 routed experts + 47.7 GiB FP8 PLE table + 14.9 GiB +BF16** — so it is NOT an all-4-bit quant, and the single largest item on disk is the +FP8 table, 38% of the download. + +**`gen-large` on the LiteLLM gateway**, one alias, verified registered for both the +infra-ops admin key and the shared all-agents key. Sampling is the checkpoint's own +declared set (temp 1.0 / top_p 0.95 / top_k 20); presence_penalty, min_p and +repetition_penalty deliberately unset because the checkpoint declares no canonical +value. Commits `f964a47`, `7e62a07`. + +## ⭐⭐ The five things worth carrying forward + +1. **The offload is upstream and merged — via a DIFFERENT PR than the park predicted.** + vLLM **#54371** "UVA PLE-offload and Engram tensor parallelism" merged + 2026-09-09T14:32Z: pinned host memory + direct UVA row reads, no worker process. + It **supersedes #53899** (worker-based, now explicitly paused), and that matters + because #53899's entire bug family — the TP=1 startup deadlock #53960, the + `pidfd_getfd`/yama ptrace gate, the shared-CUDA-event race, silently one-step-stale + PLE outputs under CUDA graphs — all came from the separate worker the UVA path does + not have. Flag: `--engram-config '{"cpu_offload": true}'`. **In `v0.29.1rc0` and + main, NOT in `v0.29.0`** (cut ~6 h before the merge), and not in any `nightly-` + dated 2026-09-09 or earlier — the nightly builds at ~06:16 UTC. +2. **⚠ `text_config.ple_embedding_dtype` is the load-or-fail discriminator.** vLLM picks + the PLE table's format from that field FIRST. A checkpoint with an FP8 table and no + such declaration resolves through ModelOpt's `*.ple.*` exclude to the *unquantized* + method and dies on the scale tensor it does ship. Check it before trying any other + community build. +3. **⚠⚠ `--kv-cache-memory` makes vLLM SKIP MEMORY PROFILING and ignore + `--gpu-memory-utilization`.** This inverts the usual "pin bytes for + reproducibility" advice. 16.00 GiB was tried and a 155,497-token prefill drove GPU 2 + to 97,074 of 97,887 MiB with the allocator logging *"OOM on device 0 while trying to + allocate 488636416 bytes (free: 422117376)"* — 466 MiB wanted, 403 free. The request + still completed, so **nothing failed visibly**; that is one step before the shape + that crashed mog-sec twice. 14.00 GiB re-probes clean. Do **not** take vLLM's own + *"17.46 GiB to fully utilize gpu memory"* line: it is computed from a profile taken + at `max-num-batched-tokens` depth and sits 3.5 GiB above what a 150K request + survives. `max-num-batched-tokens` is what bounds the activation peak — doubling + `max_model_len` left the profiled peak unchanged at 1.65 GiB. +4. **⚠ MTP is OFF but NOT written off, and the operator was right to push back.** The + seat initially defaulted MTP off citing vLLM's recipe (worse at every concurrency on + 4xH100: 8-36% less throughput, 32-173% more latency, ~36% acceptance). **That was a + cross-harness comparison** — 4xH100/TP=4/Hopper vs 1xBlackwell/TP=1 — and our own + rule says those are invalid, not merely noisy. The real gap: the recipe tested + **k=3 only**, and `mtp_num_hidden_layers` is **1**, so the draft head is a single + module run autoregressively for k>1. If throughput falls monotonically in k while + acceptance rises, **k=1 may win and nobody has published that cell.** Campaign built + at `services/flash-next-mtp-bench/` with repeats, a boot-to-boot noise floor, and + acceptance-greater-than-zero as a positive control. It completed exactly **one** + `off_A` rep before the power failed. +5. **⚠ "Healthy" and "reachable" are on opposite sides of the container boundary.** The + seat once ran `Up 4 minutes (healthy)` with **`PORTS=[]`** — its port bind had lost a + race with a dying container, and the healthcheck passed throughout because it runs + *inside* the container against `localhost:8000`. Verify `docker port` / the + docker-proxy listener, not the healthcheck. + +## ⚠⚠ And then the site went dark — see `docs/runbooks/fv-site-dark-20260913.md` + +Operator authorised running the MTP campaign on idle GPU 3 in parallel with live +testing on GPU 2, with the two-card power risk explicitly raised and accepted. + + 06:51:08 campaign starts on GPU 3; idle draw GPU0 3.80 / GPU1 3.88 / GPU2 14.27 / GPU3 6.75 W + 06:54:39 off_A healthy; rep 1 clean + 06:56:04 off_A rep 2 starts <-- last line ever written + 06:58:40 EVERY FV address unreachable, BMC included; all other sites 0% loss + +**Leading hypothesis (operator): the 1500 VA Eaton UPS overloaded and died.** It fits +better than a breaker trip because a UPS's output rating sits far below the circuit's, +making it the first protective device to give — which explains why the site let go at +**two** cards loaded rather than four, and why the ~25 W firewall died with it. + +- ⚠ **Will not self-recover.** An overload-tripped UPS needs a human; a dead one needs + replacing. Do not poll FV addresses waiting for it. +- ⚠ **Do NOT use a UPS's surge-only outlets to exceed its rating.** Both banks are + downstream of one input cord/inlet/breaker — for a 120 V 1500 VA unit, a single + NEMA 5-15P rated **12 A at maximum load**, total. The surge bank bypasses the + inverter, not the current rating. Bypass the UPS entirely instead. +- ⭐ **Recover `/tank/aimodels/flash-next-mtp-bench/power.log` before anything else.** + It sampled all four cards every 10 s to the cut and lives on `/tank`, not in a + container. **No load figure exists anywhere else** — only idle. Nothing should put a + wattage into a UPS purchase until it has been read. +- **Blast radius: 19 of 30 gateway aliases**, probed not inferred. ⚠ **There is no + local fallback** — every free local model was on fv-ml1; irv-ml1 runs no chat seat at + all (TTS/ComfyUI/arbo/clipper on two partly-occupied Ampere cards). The only non-fv + chat backends are paid. Any paid coverage must go under a **new opt-in alias name**, + never a silent repoint of `summarizer`/`gen`/`classifier`. +- ⭐ **The OOB design gap.** OPNsense-as-subnet-router protects box-down/gateway-up and + does nothing for a site-wide loss, because the BMC's only route out is through that + same gateway. Proposed-not-ratified fix in the runbook: firewall + BMC on the UPS, + GPU chassis on raw PDU. + +## Also landed: the renumber the FV cutover missed + +Every fv-ml1 link on the Homepage dashboard was dead — measured **16 entries pointing +at 10.250.50.54 and zero at 10.251.50.54**. The sweep script's allowlist was built from +files that mention the HOST, and a `homepage.href` mentions only an IP, so every +label-only stack fell outside it by construction. Fixed across 24 files plus the host +copies; allowlist extended with how to derive it next time. Commit `3132a16`. + +Two bugs fell out: `deploy-stack.sh` rejected any stack name containing a dot (so +`qwen3.5-122b`, `qwopus3.5-122b`, `mistral-medium-3.5` could not be deployed **at +all**), and scriberr's CORS allowlist contained only the dead IP and the dead +`scriberr.ana.internal` — an allowlist with nothing reachable in it. + +⚠ **Still incomplete:** the 10 running containers were never recreated, so the labels +are still stale on the host — and a plain power-on will NOT fix them, because labels +attach only at creation. The staged `docker compose up -d ` recovery sequence +applies them as a side effect. Eight stacks were deliberately NOT pushed from canonical +because their host copies have genuine drift; three of those are untracked host-only +stacks that should be brought into `stacks/`. diff --git a/persistent-memory.md b/persistent-memory.md index 51ec8a2..699dd31 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-12 ~19:30 PT (FV cutover EXECUTED — fv-ml1 live at Fountain Valley, renamed/renumbered/serving; BMC online after a VLAN-250 find; four GPUs not two)_ +_Last updated: 2026-09-13 ~00:10 PT (⚠ FV SITE DARK — UPS suspected overloaded/dead under two-card load; 19 gateway aliases down; visit 2026-09-14. Flash-Next seat + gen-large built and verified before the outage.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -112,7 +112,19 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight _As of 2026-09-12 ~19:30 PT._ -### FV colo — LIVE. Cutover done, one gap open. +### ⚠⚠ FV colo — DARK since 2026-09-13 06:56Z. Site visit 2026-09-14. + +Every Fountain Valley address is unreachable, **including the BMC** — the OOB path +goes through the same OPNsense gateway, which is also dark. All other sites healthy. +Leading cause: the 1500 VA Eaton UPS overloaded and died under two-card GPU load. +**It will not self-recover; do not poll FV addresses.** 19 of 30 gateway aliases are +down with no local fallback (every free local model was on fv-ml1; irv-ml1 runs no +chat seat). On recovery: recover `power.log` first, mask Docker before the network, +then bring seats up one at a time with `compose up -d` — `gen` first, `flash-next` +last, and do NOT restart the MTP campaign. +→ `docs/runbooks/fv-site-dark-20260913.md` + +### FV colo — the pre-outage state (cutover done, one gap open) - **fv-ml1** (ex ana-ml2) is racked at Fountain Valley, renamed, on `10.251.50.54`, mesh node `100.64.0.7`; **vb-gateway** OPNsense on `10.251.50.1` / `100.64.0.8`; **BMC** on `10.251.250.50`. Public `fv.phasefinal.com` → `172.83.89.66`. `tank` intact, all vLLM @@ -169,6 +181,12 @@ _As of 2026-09-12 ~19:30 PT._ ## Recent decisions +- `[2026-09-13]` ⚠⚠⚠ **FV SITE DARK — every Fountain Valley address including the BMC went unreachable ~2.5 min into a two-card load test; all other sites healthy. Operator's leading hypothesis: the 1500 VA Eaton UPS overloaded and DIED.** It fits better than a breaker trip because a UPS's output rating sits far below the circuit's, making it the first protective device to give — which explains why the site let go at **two** cards loaded rather than four, and why the ~25 W OPNsense box died with it. ⚠ **Will not self-recover** (tripped needs a human, dead needs replacing) — do NOT poll FV. ⚠ **Do NOT use surge-only outlets to exceed a UPS rating**: both banks share one NEMA 5-15P inlet rated 12 A total; the surge bank bypasses the inverter, not the current limit. ⭐ **Recover `/tank/aimodels/flash-next-mtp-bench/power.log` FIRST** — all four cards every 10 s to the cut, on `/tank` not in a container, and the ONLY load measurement that exists. ⚠ **19 of 30 gateway aliases dark and NO local fallback** — every free local model was on fv-ml1, irv-ml1 runs no chat seat at all; the only non-fv chat backends are paid, and any coverage must be a NEW opt-in alias, never a silent repoint. ⭐ **OOB design gap**: OPNsense-as-subnet-router covers box-down/gateway-up and nothing for a site-wide loss, since the BMC's only route out is that gateway. ⚠ Recovery hazard: ten `restart: unless-stopped` vLLM containers will all load at once on power-up — mask Docker first, then `compose up -d` seat by seat (which also finishes the stale-homepage-label fix, since labels attach only at creation). → `docs/runbooks/fv-site-dark-20260913.md`, `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md` +- `[2026-09-13]` ⭐⭐ **Qwen3.8-Flash-Next serving on ONE card with its 51B n-gram table in host RAM — the first seat whose weights do not fit its GPU.** `stacks/flash-next-seat/`, fv-ml1 GPU 2 `:8022`, plus a `gen-large` LiteLLM alias. Measured: 74.36 GiB weights resident, 14.00 GiB KV = 560,654 tokens at the full 262,144 context, 67 GiB host RSS, 75.5/212.3/387.8 tok/s at conc 1/4/8 (⚠ n=1). ⭐ The offload is vLLM **#54371 (UVA, merged 2026-09-09)** which **supersedes the paused #53899** — it has no worker process, so #53899's whole bug family (TP=1 deadlock #53960, `pidfd_getfd`/ptrace gate, stale-output-under-graphs) is designed out; in `v0.29.1rc0`, **not** `v0.29.0`. ⚠ **`text_config.ple_embedding_dtype` is the load-or-fail discriminator** for any community build. ⚠⚠ **`--kv-cache-memory` makes vLLM SKIP MEMORY PROFILING and ignore `--gpu-memory-utilization`** — 16 GiB nearly OOM'd on a 155K prefill with no visible failure; 14 GiB is the measured-safe value and vLLM's own "17.46 GiB to fully utilize" is 3.5 GiB too high. ⚠ MTP is off **pending measurement here, not written off** — the recipe's number is cross-harness and tested k=3 only, while the head is ONE layer run autoregressively, so k=1 is unpublished and may win (`services/flash-next-mtp-bench/`, one `off_A` rep banked before the outage). ⚠ A container once ran `(healthy)` with `PORTS=[]` — verify `docker port`, not the healthcheck. → `persistent-memory.d/2026-09-13-flash-next-seat-and-fv-outage.md` +- `[2026-09-13]` **Finished the ana-ml2→fv-ml1 renumber the cutover missed: 16 live Homepage entries pointed at the dead 10.250.50.54 and zero at the live IP.** The sweep allowlist was built from files that mention the HOST and a `homepage.href` mentions only an IP, so every label-only stack fell outside it by construction; 24 files fixed plus host copies, allowlist extended with how to derive it next time. Two bugs fell out: `deploy-stack.sh` **rejected any stack name containing a dot** (so `qwen3.5-122b`/`qwopus3.5-122b`/`mistral-medium-3.5` could not be deployed at all), and scriberr's CORS allowlist held only the dead IP and the dead `scriberr.ana.internal`. ⚠ Incomplete: the 10 running containers were never recreated and a plain power-on will NOT apply labels — the staged `compose up -d` recovery does. Eight stacks deliberately not pushed (real host drift); three of those are untracked host-only stacks. Commits `3132a16`, `969a1b6`, `d79f104`. + +- `[2026-09-13]` **FV→ANA fixed, Beszel18/18 up:** scoped OPNsense hybrid NAT for fv-ml1→ANA; prior NAT/filter rules preserved and rollback-guarded. → `persistent-memory.d/2026-09-13-fv-to-ana-nat.md` + - `[2026-09-12]` ⭐⭐⭐ **FV CUTOVER EXECUTED — fv-ml1 live at Fountain Valley, renamed, renumbered to 10.251/16, serving inference; BMC online after finding it was tagging 802.1q VLAN 250 into an untagged port; and the box has FOUR RTX PRO 6000 (391 GB VRAM), not the two every doc claimed.** Also: OPNsense write APIs need an `X-CSRFToken` header scraped from a `