memory: snapshot — headscale cutover complete + NASPool rebuild; session open follow-ups

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-09-06_
+_Last updated: 2026-09-06 (infra: NASPool rebuild + headscale cutover)_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -108,6 +108,26 @@ no longer deployed sidecars here. See Recent decisions.)
   (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
 ## Current state / in-flight
 
+_Infra session 2026-09-06 (NASPool + headscale) — open follow-ups from this session,
+none blocking; the ERP / althing / fiber items further down belong to other streams and
+were not touched:_
+
+- **NASPool parked copy still on ospool** — `ospool/naspool-evac` (1.65T) + `NASPool/*@evac`
+  snapshots. Destroy ONLY after the new raidz2 scrub is clean (it is, 0 errors 04:43Z) AND
+  one Backrest (01:00 PDT) + one PBS run succeed. Then `zfs destroy -r ospool/naspool-evac`
+  and drop the `@evac` snaps. ⚠ pfi-pve PSU1 still dead; backplane swap (bays 9/10) next colo
+  visit → then `zpool add NASPool spare`. Runbook `docs/runbooks/pfi-pve-naspool-rebuild.md`.
+- **FortiGate WAN SSH is temporarily open** (`wan1` allowaccess ping+ssh; admin `infra-ops`
+  trusthost2/3 = 70.230.226.88 NH3 + 23.164.40.160 ESH). Safety net for the cutover — CLOSE it
+  when the edge is retired (OPNsense/R420). `ssh infra-ops@38.120.12.42`.
+- **SOCKS proxy NOT retired** — operator asked, but yt-voice-clipper on irv-ml1 still uses
+  `YTVC_PROXY=socks5h://10.100.10.50:1080` (live, up 12d) and irv-ml1 isn't a mesh node. The
+  exit node is whole-host; the dante proxy does scoped per-yt-dlp egress → the two aren't
+  interchangeable. Keep the proxy, or first migrate irv-ml1 onto the mesh (tailscale's own
+  `--socks5-server` could replace it). Surfaced; operator to decide.
+- **infra-ops now on all four PVE hypervisors** (pfi-pve/nh3-pve/esh-pve/esh-pve-nas) — PVE
+  ships without sudo, `apt install sudo` first or elway hangs on a password prompt.
+
 _As of 2026-09-05 06:35 PDT — **ERP run 4 is TRAINING on pfi-gx10.** Everything else
 below is a live commitment or a known-open risk._
 
@@ -473,7 +493,7 @@ below is a live commitment or a known-open risk._
 
 ## Recent decisions
 
-- `[2026-09-06]` **Headscale cutover: colo↔NH3 + colo↔ESH IPsec now DORMANT, the mesh carries both (verified bidirectional); NH3↔ESH Site Magic pre-staged but needs the operator's UniFi-UI disable (`api.err.NoEdit`).** FortiGate WAN-SSH safety net added (temp, scoped to NH3+ESH egress). Method: disable tunnel FIRST then add mesh route, or the two fight (asymmetric drop). → `persistent-memory.d/2026-09-06-headscale-cutover.md`
+- `[2026-09-06]` **Headscale cutover COMPLETE — all three site-pairs on the mesh; Site Magic + both IPsec tunnels DORMANT.** Operator disabled Site Magic (UI); NH3↔ESH re-homed to a direct 8ms path. Exit nodes advertised at all three sites (multi-location egress proxy) with source preservation kept via a selective-masquerade rule (NoSNAT + `mesh-exit-masq.service` per router). Throughput 761/464 Mb/s vs old 250 IPsec. ⚠ FortiGate WAN-SSH left open (temp, scoped NH3+ESH). Method: disable tunnel FIRST then add mesh route. → `persistent-memory.d/2026-09-06-headscale-cutover.md`
 - `[2026-09-06]` **Headscale overlay mesh: control plane live at `headscale.phasefinal.com` (CT 106 nh3-pve) + subnet routers nh3-scale/esh-scale/ana-scale serving their /16s; nh3-dev enrolled. NOT cut over — Site Magic + IPsec still carry site-to-site.** ⚠ accept-routes-before-return-path black-holed nh3-dev's LAN for a minute. infra-ops user added on all four PVE hosts. → `persistent-memory.d/2026-09-06-headscale-mesh-phase1.md`
 - `[2026-09-06]` **pfi-pve NASPool REBUILT as six-wide raidz2 after a backplane fault killed bays 9/10** (Route C hybrid, operator-directed): parked 1.65T on ospool, destroyed, recreated, restored, backup tier back 04:03Z; guests never stopped (ALL boot disks are on ospool — the prior brief had this wrong). Legacy vzdump pruned to newest-per-guest by omission. OPEN: destroy `ospool/naspool-evac` after scrub + one backup cycle; backplane swap next visit; PSU1 still dead. → `persistent-memory.d/2026-09-06-pfi-pve-naspool-raidz2-rebuild.md`
 - `[2026-09-05]` **A peer's "2.7x serving-stack effect" was a coin flip — the operator rejected it on instinct and the arithmetic backed him.** Each floor was `|b0-b1|` from n=2; the ratio is half-Cauchy, P=0.452. ⚠ The disconfirming evidence sat in brokkr's own sentence, and he named *why*: the claim was his and flattering. → `persistent-memory.d/2026-09-05-floor-claim-n2-retraction.md`