memory: headscale cutover — IPsec dormant, mesh primary, Site Magic pending operator

2026-09-06 00:02:15 -07:00
parent d25c096b6a
commit df8440e4b8
2 changed files with 23 additions and 0 deletions
@@ -0,0 +1,22 @@
+# 2026-09-06 — Headscale cutover: IPsec dormant, mesh primary; Site Magic pending operator
+
+Operator goal (/goal): replace Site Magic + IPsec with headscale, tunnels dormant as backup;
+"if paranoid, enable world-accessible SSH on the FortiGate first." Full detail + method +
+follow-ups in `docs/pfi/headscale-mesh-plan.md` § CUTOVER EXECUTED. Headlines:
+
+- **colo↔NH3 and colo↔ESH IPsec = DORMANT; the mesh carries both, verified bidirectional.**
+  NH3 UDM `pfi-nh3-ana` + ESH UDM `esh-ana` set enabled=false (API). Mesh /16 routes added on
+  both UDMs and the FortiGate (→ ana-scale 10.250.50.45 / nh3-scale 10.100.50.46 / esh-scale
+  10.0.50.65). Dependent flows OK over mesh: restic ESH→rest-server-ana, FortiGate mgmt.
+- **NH3↔ESH Site Magic NOT cut by API** — `sdwan-mesh-tunnel` = `api.err.NoEdit` (cloud
+  orchestrated). Routes PRE-STAGED + shadowed; DERP path 9ms ready. **Operator disables it in
+  the UniFi UI**, then the mesh takes over. Told the operator "mesh is online" → he does it.
+- **FortiGate WAN SSH safety net (TEMPORARY):** wan1 allowaccess ping+ssh; admin infra-ops
+  trusthost2/3 = NH3 70.230.226.88 + ESH 23.164.40.160 (not 0.0.0.0). Reach it at
+  `ssh infra-ops@38.120.12.42`. Config backed up flash `pre-wan-ssh-cutover-20260906`. Remove
+  when the edge (being replaced by OPNsense/R420) is retired.
+- ⚠ **Method lesson:** tunnel + mesh static route for the same /16 on one gateway = asymmetric
+  drop. Disable the tunnel FIRST, then add the route. Broke colo once doing it tunnel-up; rolled
+  back. See [[incident_crowdsec_cgnat_false_ban]] (same day) and the plan doc.
+- Dormancy = disabled+retained (flip UDM object back to enabled=true to restore); NO auto
+  failover wired. Bonus: exit nodes → free multi-location egress proxy (parked).
@@ -473,6 +473,7 @@ below is a live commitment or a known-open risk._

 ## Recent decisions

+- `[2026-09-06]` **Headscale cutover: colo↔NH3 + colo↔ESH IPsec now DORMANT, the mesh carries both (verified bidirectional); NH3↔ESH Site Magic pre-staged but needs the operator's UniFi-UI disable (`api.err.NoEdit`).** FortiGate WAN-SSH safety net added (temp, scoped to NH3+ESH egress). Method: disable tunnel FIRST then add mesh route, or the two fight (asymmetric drop). → `persistent-memory.d/2026-09-06-headscale-cutover.md`
 - `[2026-09-06]` **Headscale overlay mesh: control plane live at `headscale.phasefinal.com` (CT 106 nh3-pve) + subnet routers nh3-scale/esh-scale/ana-scale serving their /16s; nh3-dev enrolled. NOT cut over — Site Magic + IPsec still carry site-to-site.** ⚠ accept-routes-before-return-path black-holed nh3-dev's LAN for a minute. infra-ops user added on all four PVE hosts. → `persistent-memory.d/2026-09-06-headscale-mesh-phase1.md`
 - `[2026-09-06]` **pfi-pve NASPool REBUILT as six-wide raidz2 after a backplane fault killed bays 9/10** (Route C hybrid, operator-directed): parked 1.65T on ospool, destroyed, recreated, restored, backup tier back 04:03Z; guests never stopped (ALL boot disks are on ospool — the prior brief had this wrong). Legacy vzdump pruned to newest-per-guest by omission. OPEN: destroy `ospool/naspool-evac` after scrub + one backup cycle; backplane swap next visit; PSU1 still dead. → `persistent-memory.d/2026-09-06-pfi-pve-naspool-raidz2-rebuild.md`
 - `[2026-09-05]` **A peer's "2.7x serving-stack effect" was a coin flip — the operator rejected it on instinct and the arithmetic backed him.** Each floor was `|b0-b1|` from n=2; the ratio is half-Cauchy, P=0.452. ⚠ The disconfirming evidence sat in brokkr's own sentence, and he named *why*: the claim was his and flattering. → `persistent-memory.d/2026-09-05-floor-claim-n2-retraction.md`