memory: snapshot — Flash-Next seat on one card, and the FV outage that followed

Durable capture so tomorrow's session does not have to reconstruct either half.

Built and verified before the power failed: Qwen3.8-Flash-Next serving on a single
RTX PRO 6000 with its 51B n-gram table pinned in host RAM and read over CUDA UVA --
74.36 GiB weights resident, 14.00 GiB KV for 560,654 tokens at the full 262,144
context, 67 GiB host RSS -- plus a gen-large gateway alias verified end to end.

The five findings worth carrying: the offload is #54371 (UVA, merged) which supersedes
the paused worker-based #53899 and designs out its entire bug family;
text_config.ple_embedding_dtype is the load-or-fail discriminator for any community
build; --kv-cache-memory makes vLLM SKIP memory profiling and ignore
gpu-memory-utilization, which inverts the usual pin-bytes advice and let a 16 GiB pin
nearly OOM with no visible failure; MTP is off pending measurement here rather than
written off, because the recipe's number is cross-harness and tested k=3 only while the
head is one layer run autoregressively; and a container once reported (healthy) with no
published port at all, because the healthcheck runs inside the boundary it was trusted
to validate.

Then the outage. Records it as will-not-self-recover, so no session wastes effort
polling a dead site, and carries the three things that change the visit: bypass the UPS
rather than using its surge-only bank (both banks share one 12 A inlet -- the surge
bank bypasses the inverter, not the current rating), recover power.log before anything
else because it is the only load measurement that exists anywhere, and bring seats up
one at a time because ten restart:unless-stopped containers loading at once is the
largest transient the box can make into whatever just failed.

Also records what is still half-done: the stale homepage labels on the 10 containers
that died before they could be recreated, which the staged bring-up fixes as a side
effect, and the eight drifted stacks plus three untracked host-only stacks that were
deliberately left for a deliberate reconciliation.

docs/runbooks/fv-site-dark-20260913.md

@@ -1,6 +1,9 @@
 # FV site dark — 2026-09-13 ~06:56Z
 
-**Status: UNRESOLVED, needs hands at Fountain Valley.** Written while the site is
+**Status: UNRESOLVED, WILL NOT SELF-RECOVER, needs hands at Fountain Valley.**
+Operator 2026-09-13: an overload-tripped UPS does not clear itself and a dead one
+needs replacing, so there is no recovery to wait for and no point polling the site.
+Visit planned for 2026-09-14. Do not leave watchers running against FV addresses. Written while the site is
 down so recovery does not have to be reconstructed from memory.
 
 ## What is down
@@ -138,9 +141,58 @@ much longer recovery.
 4. **Do not restart the MTP campaign.** It is the prime suspect.
 5. Watch power while seats come up: `nvidia-smi --query-gpu=index,power.draw --format=csv`.
 
+### ⭐ The staged bring-up also finishes the homepage-label fix, for free
+
+The 2026-09-13 renumber (commit `3132a16`) repaired 25 compose files on this box but
+the 10 RUNNING containers were never recreated, so their labels still carried the dead
+10.250.50.54. Those containers are gone with the power loss.
+
+⚠ A PLAIN POWER-ON DOES NOT FIX THEM. `restart: unless-stopped` restarts the existing
+container with its existing labels; labels only attach at container CREATION. But the
+staged `docker compose up -d <svc>` sequence above **is** a recreate, and the compose
+files on disk are already corrected — so bringing seats up that way applies the new
+labels as a side effect and the dashboard comes back correct. Bring them up with
+`compose up -d`, not by letting Docker restore the old containers.
+
+Afterwards, confirm with:
+
+    curl -s http://10.0.50.45:5100/api/services | \
+      python3 -c 'import json,sys;d=json.load(sys.stdin);print([s["href"] for g in d for s in (g.get("services") or []) if "10.250.50.54" in (s.get("href") or "")])'
+
+Expect `[]`. Before the outage that query returned 16 entries.
+
 **Do not reboot the OPNsense firewall** (standing operator directive; its reboot API
 403s anyway).
 
+## ⚠ PROPOSED, NOT RATIFIED — split the power so the management plane survives
+
+**This is a recommendation awaiting the operator's call, not settled intent.** Written
+down so tomorrow's rebuild can adopt or reject it deliberately rather than restoring
+the arrangement that just failed by default.
+
+    UPS         <- OPNsense firewall + fv-ml1 BMC only   (tens of watts, long runtime)
+    PDU / wall  <- GPU chassis                            (no UPS in series)
+
+Two reasons:
+
+1. **It fixes the OOB gap this outage exposed.** The BMC's only route to the fleet is
+   through the firewall, so a power event at the GPU box takes out the management plane
+   with it -- which is precisely why this incident needs a drive rather than a console
+   session. Separate the two and a repeat leaves a live firewall, a live BMC, and
+   remote eyes on a dark chassis.
+2. **A 1500 VA unit was never going to hold this box.** It has four cards, not the two
+   every record claimed until 2026-09-12.
+
+⚠ **Do NOT use a UPS's surge-only outlets to get around its rating.** Both outlet banks
+sit downstream of the same input cord, inlet and internal breaker; for a 120 V 1500 VA
+unit that is a single NEMA 5-15P rated **12 A at maximum load**, total across all
+outlets. The surge bank bypasses the inverter, not the current rating. Overloading the
+inverter trips or kills the unit; overloading the cord is a thermal problem in an
+unattended rack. Bypass the UPS entirely instead.
+
+If the GPU box is ever to go on battery, it is a 3000 VA / 2700 W-class unit and
+probably a 20 A circuit -- but size it from `power.log`, not from a spec sheet.
+
 ## Afterwards
 
 - ⭐ **The OOB design gap this exposes.** The cutover chose OPNsense-as-subnet-router