diff --git a/persistent-memory.md b/persistent-memory.md index b519f39..56e9674 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-06 (infra: NASPool rebuild + headscale cutover incl. irv-ml1)_ +_Last updated: 2026-09-07 (fleet-ops: irv DNS, ESH outage recovery, backup router-exclusions, booth feature, run-4 gate coord, cc-channel, /mnt/smithy remount, fleet-TLS pattern)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -108,6 +108,30 @@ no longer deployed sidecars here. See Recent decisions.) (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`. ## Current state / in-flight +_As of 2026-09-07 (fleet-ops session). SUPERSEDES the "run 4 is TRAINING" framing lower down: +run 4 is DONE (gated STILL-COUPLED); run 5 is AUTHORIZED but stalled. Live open items:_ + +- **⚠ ERP run 5 authorized but NOT running — brokkr sat on it; a full GX10 window wasted + (operator, 2026-09-07).** Operator already gave the third eligibility grant and thought it was + training; brokkr-smithy-dev never built the clean roots / `recipe-r5.json` / launched. GX10 idle + (0% GPU) since the run-4 gate. Raw sources ARE staged (nh3-nas `/volume1/smithy/datasets/raw/{ccdv__govreport-summarization,pszemraj__qmsum-cleaned}`, ~09-06 23:06); clean-v1 derived roots + + `recipe-r5.json` + launch still owed by brokkr. Run 5 = dependency-forcing slot (GovReport+QMSum + replace airoboros; single variable; tests whether long-context-dense data breaks the diversity↔coherence + coupling run 4 left STILL-COUPLED). **FOLLOW-UP: nudge brokkr to build+launch — don't waste the next window.** +- **run-4 gate = STILL-COUPLED** (RESULT-run04-gate.md, brokkr-smithy). Corpus dilution kept the + diversity gain, did NOT remove the safety/coherence regression. **Base-abliteration LABEL unresolved — + operator's ruling owed** (recipe says `-heretic`, provenance says stock; telemetry leans STOCK: base + refused 77.7% hard, exact-repro of 3c). brokkr flags his lean as inference until the operator rules. +- **NASPool evac copy now safe to destroy** — scrub clean (0 err) AND PBS runs landing (verified 116 + backups, 8 guests 09-06). `ospool/naspool-evac` (1.65T) can go once ONE Backrest run is confirmed: + `zfs destroy -r ospool/naspool-evac` + drop `@evac`. pfi-pve PSU1 dead + backplane bays 9/10 dead + (cold spares → next colo visit). +- **MEMORY.md (auto-memory index) ~23.3KB, near the 24.4KB read cap** — needs a compaction pass soon + or fresh sessions may fail to load it. Operator offered; not yet done. +- **irv-ml1 on-site window / YTVC still open from 09-06:** FortiGate WAN SSH still temporarily open + (close when the edge is retired); YTVC still DOWN (dante retired, scoped exit-node egress unwired); + reverse-tunnel / wg0-delete decisions pending the operator's Irvine access. + _Infra session 2026-09-06 (NASPool rebuild + full headscale cutover incl. irv-ml1) — open follow-ups; the ERP / althing / fiber items further down belong to other streams, untouched:_ @@ -502,6 +526,12 @@ below is a live commitment or a known-open risk._ ## Recent decisions +- `[2026-09-07]` **Fleet internal TLS pattern shipped** — caddy (cloudflare-plugin build, `~/.local/bin/caddy-cf`, `fleet-tls-caddy.service`) on nh3-dev is the wildcard cert authority: publicly-trusted LE `*.nh3.phasefinal.com` via Cloudflare DNS-01, resolved internally by an AdGuard split-horizon rewrite (all 3 resolvers). `talk` self-terminates on :8092 with the trusted cert (operator's in-container-TLS ruling), renewal auto-synced by `fleet-tls-cert-check.timer`. Interstitial gone; secure-context+AudioWorklet verified via headless Chromium. Pattern + foot-guns (restart-disrupts-inflight → clients need retry; wildcard = name-only, never IP) → auto-memory `reference_fleet_internal_tls_pattern`. +- `[2026-09-07]` **cc-channel registered for this infra-ops session's wake** — `althing-route` cc route → the CC session's `$XDG_RUNTIME_DIR/cc-socks/.sock`; herald pokes the socket directly at a turn boundary. Replaces the FIFO/poll waiter that Claude Code 2.1.257 kept killing while the seat sat in `shell`. Session-local — re-declare per session. +- `[2026-09-07]` **irv-ml1 /mnt/smithy remount fixed post-cutover** — export allowed `10.0.0.0/8` (old wg0) but not the mesh `100.64.0.0/10` irv-ml1 now sources from → all-uid "permission denied"; added the mesh range to the nh3-nas smithy export + remounted (clientaddr now 100.64.0.6). nh3-nas admin = `infra-ops` PASSWORD auth (vault `nh3-nas/infra-ops-password`), sudo ALL, SFTP subsystem OFF. → auto-memory `reference_irv_ml1_gpu_r14` (corrected). +- `[2026-09-07]` **irv-ml1.nh3.internal DNS repointed** to the live Irvine LAN IP `10.6.110.50` (was the dead wg0 `10.100.79.3`); CLAUDE.md fleet-row + placement-rule updated to mesh reality. commit `0336e03`. +- `[2026-09-07]` **Subnet routers excluded from vzdump fleet-wide** (ana-scale 114/pfi-pve, nh3-scale 107/nh3-pve, esh-scale 108/esh-pve) so a hung backup can't blackhole a site; nh3-headscale (106, control plane) KEPT; ESH backup moved 02:15→03:30. Root cause of this morning's ESH outage: an overnight vzdump left CT108 (esh-scale) locked → whole site dark. → auto-memory `feedback_esh_backup_window_0330`. +- `[2026-09-07]` **Booth link board: pin/favorite + multi-select delete + newest-first** (booth-v0.1.8, commit `76fdf45`, tag `booth-v0.1.8`) — pins in a `.pins` sidecar (content-ids), one `
` + `formaction` buttons so ×/★/bulk-delete all degrade with JS off. - `[2026-09-06]` **Headscale cutover COMPLETE — all three site-pairs on the mesh; Site Magic + both IPsec tunnels DORMANT.** Operator disabled Site Magic (UI); NH3↔ESH re-homed to a direct 8ms path. Exit nodes advertised at all three sites (multi-location egress proxy) with source preservation kept via a selective-masquerade rule (NoSNAT + `mesh-exit-masq.service` per router). Throughput 761/464 Mb/s vs old 250 IPsec. ⚠ FortiGate WAN-SSH left open (temp, scoped NH3+ESH). Method: disable tunnel FIRST then add mesh route. → `persistent-memory.d/2026-09-06-headscale-cutover.md` - `[2026-09-06]` **Headscale overlay mesh: control plane live at `headscale.phasefinal.com` (CT 106 nh3-pve) + subnet routers nh3-scale/esh-scale/ana-scale serving their /16s; nh3-dev enrolled. NOT cut over — Site Magic + IPsec still carry site-to-site.** ⚠ accept-routes-before-return-path black-holed nh3-dev's LAN for a minute. infra-ops user added on all four PVE hosts. → `persistent-memory.d/2026-09-06-headscale-mesh-phase1.md` - `[2026-09-06]` **pfi-pve NASPool REBUILT as six-wide raidz2 after a backplane fault killed bays 9/10** (Route C hybrid, operator-directed): parked 1.65T on ospool, destroyed, recreated, restored, backup tier back 04:03Z; guests never stopped (ALL boot disks are on ospool — the prior brief had this wrong). Legacy vzdump pruned to newest-per-guest by omission. OPEN: destroy `ospool/naspool-evac` after scrub + one backup cycle; backplane swap next visit; PSU1 still dead. → `persistent-memory.d/2026-09-06-pfi-pve-naspool-raidz2-rebuild.md`