memory: snapshot — run-5 authorized-but-stalled (brokkr), fleet-TLS pattern, /mnt/smithy remount, backup router-exclusions, irv DNS; session open follow-ups

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-09-06 (infra: NASPool rebuild + headscale cutover incl. irv-ml1)_
+_Last updated: 2026-09-07 (fleet-ops: irv DNS, ESH outage recovery, backup router-exclusions, booth feature, run-4 gate coord, cc-channel, /mnt/smithy remount, fleet-TLS pattern)_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -108,6 +108,30 @@ no longer deployed sidecars here. See Recent decisions.)
   (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
 ## Current state / in-flight
 
+_As of 2026-09-07 (fleet-ops session). SUPERSEDES the "run 4 is TRAINING" framing lower down:
+run 4 is DONE (gated STILL-COUPLED); run 5 is AUTHORIZED but stalled. Live open items:_
+
+- **⚠ ERP run 5 authorized but NOT running — brokkr sat on it; a full GX10 window wasted
+  (operator, 2026-09-07).** Operator already gave the third eligibility grant and thought it was
+  training; brokkr-smithy-dev never built the clean roots / `recipe-r5.json` / launched. GX10 idle
+  (0% GPU) since the run-4 gate. Raw sources ARE staged (nh3-nas `/volume1/smithy/datasets/raw/{ccdv__govreport-summarization,pszemraj__qmsum-cleaned}`, ~09-06 23:06); clean-v1 derived roots +
+  `recipe-r5.json` + launch still owed by brokkr. Run 5 = dependency-forcing slot (GovReport+QMSum
+  replace airoboros; single variable; tests whether long-context-dense data breaks the diversity↔coherence
+  coupling run 4 left STILL-COUPLED). **FOLLOW-UP: nudge brokkr to build+launch — don't waste the next window.**
+- **run-4 gate = STILL-COUPLED** (RESULT-run04-gate.md, brokkr-smithy). Corpus dilution kept the
+  diversity gain, did NOT remove the safety/coherence regression. **Base-abliteration LABEL unresolved —
+  operator's ruling owed** (recipe says `-heretic`, provenance says stock; telemetry leans STOCK: base
+  refused 77.7% hard, exact-repro of 3c). brokkr flags his lean as inference until the operator rules.
+- **NASPool evac copy now safe to destroy** — scrub clean (0 err) AND PBS runs landing (verified 116
+  backups, 8 guests 09-06). `ospool/naspool-evac` (1.65T) can go once ONE Backrest run is confirmed:
+  `zfs destroy -r ospool/naspool-evac` + drop `@evac`. pfi-pve PSU1 dead + backplane bays 9/10 dead
+  (cold spares → next colo visit).
+- **MEMORY.md (auto-memory index) ~23.3KB, near the 24.4KB read cap** — needs a compaction pass soon
+  or fresh sessions may fail to load it. Operator offered; not yet done.
+- **irv-ml1 on-site window / YTVC still open from 09-06:** FortiGate WAN SSH still temporarily open
+  (close when the edge is retired); YTVC still DOWN (dante retired, scoped exit-node egress unwired);
+  reverse-tunnel / wg0-delete decisions pending the operator's Irvine access.
+
 _Infra session 2026-09-06 (NASPool rebuild + full headscale cutover incl. irv-ml1) — open
 follow-ups; the ERP / althing / fiber items further down belong to other streams, untouched:_
 
@@ -502,6 +526,12 @@ below is a live commitment or a known-open risk._
 
 ## Recent decisions
 
+- `[2026-09-07]` **Fleet internal TLS pattern shipped** — caddy (cloudflare-plugin build, `~/.local/bin/caddy-cf`, `fleet-tls-caddy.service`) on nh3-dev is the wildcard cert authority: publicly-trusted LE `*.nh3.phasefinal.com` via Cloudflare DNS-01, resolved internally by an AdGuard split-horizon rewrite (all 3 resolvers). `talk` self-terminates on :8092 with the trusted cert (operator's in-container-TLS ruling), renewal auto-synced by `fleet-tls-cert-check.timer`. Interstitial gone; secure-context+AudioWorklet verified via headless Chromium. Pattern + foot-guns (restart-disrupts-inflight → clients need retry; wildcard = name-only, never IP) → auto-memory `reference_fleet_internal_tls_pattern`.
+- `[2026-09-07]` **cc-channel registered for this infra-ops session's wake** — `althing-route` cc route → the CC session's `$XDG_RUNTIME_DIR/cc-socks/<pid>.sock`; herald pokes the socket directly at a turn boundary. Replaces the FIFO/poll waiter that Claude Code 2.1.257 kept killing while the seat sat in `shell`. Session-local — re-declare per session.
+- `[2026-09-07]` **irv-ml1 /mnt/smithy remount fixed post-cutover** — export allowed `10.0.0.0/8` (old wg0) but not the mesh `100.64.0.0/10` irv-ml1 now sources from → all-uid "permission denied"; added the mesh range to the nh3-nas smithy export + remounted (clientaddr now 100.64.0.6). nh3-nas admin = `infra-ops` PASSWORD auth (vault `nh3-nas/infra-ops-password`), sudo ALL, SFTP subsystem OFF. → auto-memory `reference_irv_ml1_gpu_r14` (corrected).
+- `[2026-09-07]` **irv-ml1.nh3.internal DNS repointed** to the live Irvine LAN IP `10.6.110.50` (was the dead wg0 `10.100.79.3`); CLAUDE.md fleet-row + placement-rule updated to mesh reality. commit `0336e03`.
+- `[2026-09-07]` **Subnet routers excluded from vzdump fleet-wide** (ana-scale 114/pfi-pve, nh3-scale 107/nh3-pve, esh-scale 108/esh-pve) so a hung backup can't blackhole a site; nh3-headscale (106, control plane) KEPT; ESH backup moved 02:15→03:30. Root cause of this morning's ESH outage: an overnight vzdump left CT108 (esh-scale) locked → whole site dark. → auto-memory `feedback_esh_backup_window_0330`.
+- `[2026-09-07]` **Booth link board: pin/favorite + multi-select delete + newest-first** (booth-v0.1.8, commit `76fdf45`, tag `booth-v0.1.8`) — pins in a `.pins` sidecar (content-ids), one `<form>` + `formaction` buttons so ×/★/bulk-delete all degrade with JS off.
 - `[2026-09-06]` **Headscale cutover COMPLETE — all three site-pairs on the mesh; Site Magic + both IPsec tunnels DORMANT.** Operator disabled Site Magic (UI); NH3↔ESH re-homed to a direct 8ms path. Exit nodes advertised at all three sites (multi-location egress proxy) with source preservation kept via a selective-masquerade rule (NoSNAT + `mesh-exit-masq.service` per router). Throughput 761/464 Mb/s vs old 250 IPsec. ⚠ FortiGate WAN-SSH left open (temp, scoped NH3+ESH). Method: disable tunnel FIRST then add mesh route. → `persistent-memory.d/2026-09-06-headscale-cutover.md`
 - `[2026-09-06]` **Headscale overlay mesh: control plane live at `headscale.phasefinal.com` (CT 106 nh3-pve) + subnet routers nh3-scale/esh-scale/ana-scale serving their /16s; nh3-dev enrolled. NOT cut over — Site Magic + IPsec still carry site-to-site.** ⚠ accept-routes-before-return-path black-holed nh3-dev's LAN for a minute. infra-ops user added on all four PVE hosts. → `persistent-memory.d/2026-09-06-headscale-mesh-phase1.md`
 - `[2026-09-06]` **pfi-pve NASPool REBUILT as six-wide raidz2 after a backplane fault killed bays 9/10** (Route C hybrid, operator-directed): parked 1.65T on ospool, destroyed, recreated, restored, backup tier back 04:03Z; guests never stopped (ALL boot disks are on ospool — the prior brief had this wrong). Legacy vzdump pruned to newest-per-guest by omission. OPEN: destroy `ospool/naspool-evac` after scrub + one backup cycle; backplane swap next visit; PSU1 still dead. → `persistent-memory.d/2026-09-06-pfi-pve-naspool-raidz2-rebuild.md`