memory: ana-ml2 -> fv-ml1 Fountain Valley relocation + leave-rack-dark decisions

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-09-11 16:00 PT (BabyYarros COMPLETE — both arms trained + evaluated: voice moved toward Yarros above the noise floor, instruct renders beats 9/10; Anaheim colo DOWN at facility level ~15:0x PT, agent dispatched, ~90min ETA; sentinel-r3 + cyber-preview NVFP4 quants done/interrupted by the outage)_
+_Last updated: 2026-09-11 ~17:40 PT (ana-ml2 relocating to a NEW Fountain Valley colo tomorrow as fv-ml1 — power draw is the root cause of the repeated Anaheim breaker trips; cutover fully prepped + rename sweep scripted; Anaheim otherwise recovered, rack left DARK until the move; BabyYarros complete + evaluated earlier today)_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -167,10 +167,14 @@ _As of 2026-09-11 09:00 PT._
 
 ## Recent decisions
 
-- `[2026-09-11]` **Beszel priority 1: all six installed; Synology hub verification pending.** Vaulted password resolved access; nh3-nas registered with four alerts, but cross-site hub connectivity then timed out. Earlier five verified, 13/13 at that point. → `persistent-memory.d/2026-09-11-beszel-priority1.md`
+- `[2026-09-11]` **Beszel priority 2 complete:** both DB hosts and both PBS hosts verified, 16 new alerts; fleet 17/18 up (ana-ml2 down). → `persistent-memory.d/2026-09-11-beszel-priority2.md`
+
+- `[2026-09-11]` **Beszel priority 1 complete: all six installed and verified.** After Anaheim recovery, live Synology samples and alerts verified; fleet 13/14 up, only known ana-ml2 outage remains. Configs not committed. → `persistent-memory.d/2026-09-11-beszel-priority1.md`
 
 - `[2026-09-11]` **Sentinel-R3 pulled, MTP-grafted, and quantized as a M.O.G.-SEC seat candidate — quant DONE, acceptance UNVERIFIED (blocked on GPU space).** Operator got access to `glyphsoftware/sentinel-r3` and asked to compare vs the running M.O.G.-SEC seat + pull if promising, then "quant it with a grafted mtp head". **It is promising and a better FIT**: same base (stock Qwen3.8-27B), same `qwen3_5` hybrid arch, same 262K, vision-intact — but M.O.G.-SEC is a persona on stock weights while Sentinel-R3 is a REAL SFT finetune on 1,230 authorized-pentest agent trajectories over a 19-tool surface that **matches our own harness** (Bash/Read/Write/Edit/Grep/Glob/Agent/Task*/Monitor/…). Card is unusually honest (flags its own mmlu-cybersec 0.88 as within-noise of base). **HF check: M.O.G.-SEC repo unchanged** (sha still our pinned `deede6779…`). **MTP: Sentinel ships ZERO mtp tensors**; grafted the verbatim base head from `qwen38-27b-uncensored-bf16` (compare_mtp_head → IDENTICAL) — lineage correct since Sentinel's base is stock Qwen3.8-27B and that head is a verbatim base graft. ⚠ **Acceptance is UNVERIFIED and may differ from the 47.7% the head hits on STOCK weights** — it now reads hidden states from an SFT-finetuned body (the exact Stage-1b residual risk). Quant = the standard mixed NVFP4-W4A4(MLP 0-55) + FP8-W8A8(attn/linear_attn/lm_head/MLP 56-63) recipe, ran CUDA_VISIBLE_DEVICES=1 on GPU1 free space, no seat downtime, 51→22 GB. post_quant carried the head forward + re-injected `re:^mtp.*` (llm-compressor prunes it → the 0%-accept bug). Structural verify clean: 1968 tensors, 0 unresolved, 15 mtp, 333 visual, ignore has mtp+visual. **Artifact `/tank/aimodels/sentinel-r3-nvfp4-mixed` (+ `.PROVENANCE.txt`).** ⚠ **License is PROPRIETARY** (Glyph Proprietary v1.0, all-rights-reserved) — operator's fair-use/licensee call, not apache like M.O.G.-SEC. ⚠ **Serving/A-B is BLOCKED on GPU space**: weights are 22 GB, GPU0 has 7.6 free / GPU1 19.9 — a probe serve needs a freed co-tenant slot (~25 GB), which is a material-consequence call. Serve with the PROSE system prompt (trained on prose tools, not structured `tools=`). → `/tank/aimodels/sentinel-r3-nvfp4-mixed.PROVENANCE.txt`
 - `[2026-09-11]` ⭐ **MEASURED: two concurrent training jobs on pfi-gx10 are 13% NET SLOWER than running them back to back — VRAM is not the constraint and never was.** Operator asked to run the two BabyYarros arms in parallel if VRAM allowed. It does, comfortably: **18.4 GiB per 4B LoRA job, 36 of 121 GiB with both up, 98 GiB free.** But the GB10 is a *capacity* box, not a throughput box, and the binding constraint is memory bandwidth. Solo baseline **37.10 s/it (n=6, 0.05% spread)**; with a second job both arms settled at **~85 s/it — 2.29x each**, so combined throughput 0.0235 vs 0.0270 steps/s solo. Not a clean 2x split: the box is *past* its roofline and pays a contention penalty on top. ⭐ **Control: killing the second job returned the first to 37 s/it on the very next step**, so the slowdown tracked contention and reversed with it. Chaining finished both arms ~43 min earlier than concurrency would have. **General form: on this box, `nvidia-smi` free memory tells you nothing about whether a second job is affordable.** Decision rule was pre-registered before the numbers were read (<55 s/it keep both, ≥2x chain). `scripts/yarros-corpus/{launch-yarros-4b-base,chain-yarros-4b-base}.sh`; the shared-GPU bypass is an explicit argument, never a default.
+- `[2026-09-11]` ⭐ **ana-ml2 → fv-ml1: relocating to a NEW Fountain Valley colo TOMORROW (operator decision). Its power draw (dual Blackwell PRO 6000, ~1.5 kW peak) is the ROOT CAUSE of the repeated Anaheim rack-breaker trips (2026-08-26, 2026-09-11) — moving it to its own circuit fixes the recurring whole-site outage.** New site `fv`, same shape as Anaheim: server subnet **10.251.50.0/24** (fv-ml1 = **10.251.50.54**, mirroring the old host octet), mgmt/BMC **10.251.250.0/24** (fv-ml1-bmc = **10.251.250.50**). **OPNsense firewall is the multi-homed gateway** (`.1` in every FV VLAN) **AND the tailscale/headscale subnet-router advertising 10.251.0.0/16** — chosen over ana-ml2-as-endpoint specifically because the firewall stays up when the GPU box is down, giving out-of-band BMC access over the mesh — the exact thing the fleet LACKED during today's outage (no OOB path, BMC islanded). **Rename to `fv-ml1`, full `fv.internal` DNS name.** DNS approach: **PIGGYBACK** — `dns-sync` builds `name.site.zone` with no check that the site is in the `sites:` block, so `fv-ml1`/`fv-ml1-bmc` records with `site: fv` resolve fleet-wide from the existing ana/esh/nh3 resolvers immediately; add a real `fv` resolver only when FV needs LOCAL resolution (OPNsense can't host the AdGuard the sync targets — it's FreeBSD/Unbound). **Clean cutover: the box is already down (BMC dark, no power since the outage), and `/tank` is LOCAL ZFS with NO NFS from ana-nas, so data travels with the chassis.** ⚠ Load-bearing repoint = `stacks/litellm/conf/config.yaml` (~10 `api_base: 10.250.50.54:{8015,8016,8018,8019}` → `10.251.50.54`; darkens every inference alias if missed) — gateway STAYS on ana-docker so fv-ml1 serves cross-site (FV↔Anaheim metro, fine). Everything staged, nothing deployed: **runbook `docs/runbooks/fv-ml1-cutover.md`** (commit `ce04f9d`; exact DNS + LiteLLM commands) + **`scripts/fv-ml1-rename-sweep.sh`** (`8400f3a`; scoped, dry-run default, history/provenance-safe, manual-review list for judgement calls).
+- `[2026-09-11]` **Anaheim rack LEFT DARK until the move (operator decision).** ana-ml2 is the ONLY host still down post-recovery (BMC dark = no power); rather than power it on tonight just to shut it down for the truck tomorrow, it stays off. I hold vaulted IPMI creds (`ana-ml2/bmc-{infra-ops,password}`) but there is nothing to bring up — the box relocates as fv-ml1.
 - `[2026-09-11]` ⚠ **RECOVERY FOOT-GUN, will recur every colo power event: crowdsec crashes on the hard power-off and traefik's bouncer fail-CLOSES — empty-body 403 on EVERY HTTP service behind traefik (gitea, homepage, …) while the apps themselves are fine.** Signature (bifrost-dev reported it, gitea-shaped): HTTPS returns `403 content-length 0, no app body` on all routes, but git-over-SSH works (SSH bypasses traefik). Diagnosis: `gitea` direct on `localhost:3000` = 200 (app healthy), through traefik = 403; `crowdsec` container `Exited (255)`; `cscli decisions list` EMPTY (not an IP-ban). **The bouncer plugin does NOT self-recover from a startup-time LAPI-unreachable race** — even after crowdsec is healthy again, traefik keeps 403ing until traefik itself is restarted. **FIX: `docker start crowdsec` (its data/config are LOCAL volumes, comes up clean), wait for `cscli lapi status` = OK, THEN `docker restart traefik`** so the plugin re-inits against the live LAPI. Verified 403→200 on gitea API/web/PyPI-index from an off-box vantage. This unblocked bifrost-dev's 1.2.0 PyPI publish (+ worldtree/wyrd/ratatoskr) and any HTTP gitea access; heid's SSH pushes were never affected. → add to the recovery runbook: **crowdsec+traefik restart is a standard post-power-loss step.**
 - `[2026-09-11]` **Anaheim colo recovered ~16:39 PT EXCEPT ana-ml2 (bare metal, NO power — its BMC 10.250.250.50 is dark on standby, unlike same-subnet pfi-pve which is up → needs a physical PDU/PSU/breaker fix, not a boot).** pfi-pve + all its VMs (ana-docker/ana-nas/ana-wg/corviduo-dev/pbs-ana) auto-started clean (on-boot gap held this time). LiteLLM came back up on its own (transient `unhealthy` during startup → serving). ⚠ **Public WAN (38.120.12.44) ICMP still blocked from outside but HTTPS works fleet-internally** (mesh-routed). ana-ml2 down blocks the gen/summarizer/mog-sec seats AND the cyber-preview quant re-run. **I hold vaulted IPMI creds (`ana-ml2/bmc-{infra-ops,password}`) to power-on + boot-watch the instant its BMC returns.**
 - `[2026-09-11]` ⭐⭐ **BabyYarros COMPLETE — both arms trained AND evaluated; the voice moved toward Yarros above the measured noise floor, and the instruct arm renders beats 9/10.** Training: **Base best held-out 2.5263 @ ckpt-125** (overfits within the epoch — best is the checkpoint, not the shipped step-178 adapter), **Instruct 2.6114 @ 178** (still descending, undertrained if anything). Base-wins-held-out / Instruct-holds-instruction replicates Brontë at a near-identical **0.085-nat** gap. **Eval (gx10, seat-free, done during the Anaheim outage):** three voice arms + instruct beat→paragraph. `delta_cb` (Burrows over char-bigrams vs held-out Yarros) ordering **base-125 0.549 < instruct 0.631 < base-unadapted 0.706**, same-author target 0.463; both adapters clear the **0.046 measured noise floor** (within-arm seed spread, not the same-author distance — first cut mis-framed that) — base **+0.157**, instruct **+0.076** vs control. ⚠ One seed-pair per arm, so the ordering CORROBORATES the independent held-out-loss ordering rather than settling it. **Beats (instruct, chat template, Yarros SYS): on-beat 9/10** (it takes direction after raw-text training — the Skaldsong question, answered yes), in-band 5/10, ran-on 7/10 (length + clean-close discipline is the weak axis, same trade as Brontë). Booth: **http://10.100.10.50:8090/b/babyyarros-voice/**. Tooling `scripts/r49-corpus/{voice_prompts_yarros.json,gen_beats_chat_yarros.py,voice_distance.py,build_booth_yarros.py}`, commit `5558d9c`. ⛔ **DEFERRED to power-return** (needs the ana-ml2 gen seat): the frozen adjudication's romantasy control panel, a 2nd seed, and the beat-incumbent leg.