feat(mesh): cut colo<->NH3 and colo<->ESH onto headscale (IPsec dormant); Site Magic pre-staged for operator UI disable

2026-09-06 00:01:43 -07:00
parent e7f8d1f227
commit d25c096b6a
1 changed files with 48 additions and 0 deletions
@@ -215,3 +215,51 @@ weighing when prioritising the cut-over.
 persists) whitelists 23.164.40.160; SIGHUP reload, `crowdsec -t` clean, parser loaded.
 TEMPORARY — remove when ESH gets its static IP. crowdsec stack is NOT in stacks/ canonical
 (only stacks-mirror), so this lives on the host; re-mirror with sync-stacks.sh.
+
+## CUTOVER EXECUTED 2026-09-06 — IPsec dormant, mesh primary; Site Magic pending operator
+
+Operator goal: replace Site Magic + IPsec with headscale, tunnels dormant as backup.
+
+**Safety net first:** FortiGate WAN SSH enabled — `wan1 set allowaccess ping ssh`, admin
+`infra-ops` trusthost2/3 = 70.230.226.88/32 (NH3 egress) + 23.164.40.160/32 (ESH egress);
+trusthost1 10.0.0.0/8 kept. Config backed up to flash `pre-wan-ssh-cutover-20260906`.
+Verified: `ssh infra-ops@38.120.12.42` works from nh3-dev, tunnel-independent. ⚠ TEMPORARY —
+scoped to two source IPs, not 0.0.0.0; remove after the edge is retired. Drive the FortiGate
+via WAN SSH (`scratchpad/fgw.py`) during/after cutover, not over the tunnel.
+
+**Method (learned the hard way):** the tunnel and a mesh static route for the same /16 on the
+same gateway FIGHT — forward/return split across transports, asymmetric traffic dropped
+(broke colo from NH3 at 06:xx while both were up; rolled back). Correct order per pair:
+disable the tunnel end(s) FIRST (route withdraws), THEN add the mesh /16 route (now
+unambiguous). Never do it with the tunnel up.
+
+**colo↔NH3 (IPsec):** NH3 UDM `pfi-nh3-ana` disabled (enabled=false, API); mesh routes
+NH3 `10.250.0.0/16→10.100.50.46 (nh3-scale)`, FortiGate `10.100.0.0/16→10.250.50.45
+(ana-scale)`. Verified: nh3-dev→ana-docker + pfi-pve OPEN; FortiGate reachable via mesh AND
+WAN SSH; ESH→rest-server-ana (restic) OPEN. FortiGate phase1 left enabled (no peer → idle);
+re-enable NH3 side to restore.
+
+**colo↔ESH (IPsec):** ESH UDM `esh-ana` disabled; mesh routes ESH
+`10.250.0.0/16→10.0.50.65 (esh-scale)`, FortiGate `10.0.0.0/16→10.250.50.45`. Verified
+bidirectional: esh-docker-vm↔ana-docker/pfi-pve OPEN both ways. Proves ESH (CGNAT) mesh over
+public/DERP.
+
+**NH3↔ESH (Site Magic) — NOT done by API.** `sdwan-mesh-tunnel` objects (`ESH-UDMPM` on NH3,
+`PFI-UDMSE` on ESH) return `api.err.NoEdit` — Site Magic is cloud-orchestrated, not togglable
+via the classic REST API. Mesh routes PRE-STAGED (NH3 `10.0.0.0/16→nh3-scale`, ESH
+`10.100.0.0/16→esh-scale`), currently SHADOWED by Site Magic (traffic still on the SD-WAN
+tunnel, traceroute via 192.168.1.x transit). esh-scale↔nh3-scale hold a DERP(lax) 9ms path
+independent of Site Magic → when the operator disables Site Magic in the UniFi UI / Site
+Manager, the staged routes take over immediately. **Operator to toggle; mesh verified ready.**
+
+**Dormancy = disabled-but-retained**, not auto-failover (overlapping /16s across two
+transports reintroduce the asymmetry). Re-enable a pair = flip the UDM object back to
+enabled=true (+ re-enable Site Magic in UI). No automatic mesh→tunnel failover is wired.
+
+**Bonus (operator noted):** exit nodes at each site give a free multi-location egress proxy
+(`tailscale up --advertise-exit-node` + approve); replaces the nh3-dev SOCKS5 residential
+egress with a native feature. Not yet configured — parked follow-up.
+
+**Follow-ups:** operator disables Site Magic; (optional) disable FortiGate phase1 ends for
+tidiness; advertise exit nodes; self-hosted DERP at ESH on the 2G circuit; remove the
+FortiGate WAN-SSH trusthosts when the edge is retired; PVE 8.4 upgrade on pfi-pve/nh3-pve.