From d25c096b6a26d94ac52f7ffaf1d245b332e68984 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 6 Sep 2026 00:01:43 -0700 Subject: [PATCH] feat(mesh): cut colo<->NH3 and colo<->ESH onto headscale (IPsec dormant); Site Magic pre-staged for operator UI disable --- docs/pfi/headscale-mesh-plan.md | 48 +++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/docs/pfi/headscale-mesh-plan.md b/docs/pfi/headscale-mesh-plan.md index a8e2a71..1b55647 100644 --- a/docs/pfi/headscale-mesh-plan.md +++ b/docs/pfi/headscale-mesh-plan.md @@ -215,3 +215,51 @@ weighing when prioritising the cut-over. persists) whitelists 23.164.40.160; SIGHUP reload, `crowdsec -t` clean, parser loaded. TEMPORARY — remove when ESH gets its static IP. crowdsec stack is NOT in stacks/ canonical (only stacks-mirror), so this lives on the host; re-mirror with sync-stacks.sh. + +## CUTOVER EXECUTED 2026-09-06 — IPsec dormant, mesh primary; Site Magic pending operator + +Operator goal: replace Site Magic + IPsec with headscale, tunnels dormant as backup. + +**Safety net first:** FortiGate WAN SSH enabled — `wan1 set allowaccess ping ssh`, admin +`infra-ops` trusthost2/3 = 70.230.226.88/32 (NH3 egress) + 23.164.40.160/32 (ESH egress); +trusthost1 10.0.0.0/8 kept. Config backed up to flash `pre-wan-ssh-cutover-20260906`. +Verified: `ssh infra-ops@38.120.12.42` works from nh3-dev, tunnel-independent. ⚠ TEMPORARY — +scoped to two source IPs, not 0.0.0.0; remove after the edge is retired. Drive the FortiGate +via WAN SSH (`scratchpad/fgw.py`) during/after cutover, not over the tunnel. + +**Method (learned the hard way):** the tunnel and a mesh static route for the same /16 on the +same gateway FIGHT — forward/return split across transports, asymmetric traffic dropped +(broke colo from NH3 at 06:xx while both were up; rolled back). Correct order per pair: +disable the tunnel end(s) FIRST (route withdraws), THEN add the mesh /16 route (now +unambiguous). Never do it with the tunnel up. + +**colo↔NH3 (IPsec):** NH3 UDM `pfi-nh3-ana` disabled (enabled=false, API); mesh routes +NH3 `10.250.0.0/16→10.100.50.46 (nh3-scale)`, FortiGate `10.100.0.0/16→10.250.50.45 +(ana-scale)`. Verified: nh3-dev→ana-docker + pfi-pve OPEN; FortiGate reachable via mesh AND +WAN SSH; ESH→rest-server-ana (restic) OPEN. FortiGate phase1 left enabled (no peer → idle); +re-enable NH3 side to restore. + +**colo↔ESH (IPsec):** ESH UDM `esh-ana` disabled; mesh routes ESH +`10.250.0.0/16→10.0.50.65 (esh-scale)`, FortiGate `10.0.0.0/16→10.250.50.45`. Verified +bidirectional: esh-docker-vm↔ana-docker/pfi-pve OPEN both ways. Proves ESH (CGNAT) mesh over +public/DERP. + +**NH3↔ESH (Site Magic) — NOT done by API.** `sdwan-mesh-tunnel` objects (`ESH-UDMPM` on NH3, +`PFI-UDMSE` on ESH) return `api.err.NoEdit` — Site Magic is cloud-orchestrated, not togglable +via the classic REST API. Mesh routes PRE-STAGED (NH3 `10.0.0.0/16→nh3-scale`, ESH +`10.100.0.0/16→esh-scale`), currently SHADOWED by Site Magic (traffic still on the SD-WAN +tunnel, traceroute via 192.168.1.x transit). esh-scale↔nh3-scale hold a DERP(lax) 9ms path +independent of Site Magic → when the operator disables Site Magic in the UniFi UI / Site +Manager, the staged routes take over immediately. **Operator to toggle; mesh verified ready.** + +**Dormancy = disabled-but-retained**, not auto-failover (overlapping /16s across two +transports reintroduce the asymmetry). Re-enable a pair = flip the UDM object back to +enabled=true (+ re-enable Site Magic in UI). No automatic mesh→tunnel failover is wired. + +**Bonus (operator noted):** exit nodes at each site give a free multi-location egress proxy +(`tailscale up --advertise-exit-node` + approve); replaces the nh3-dev SOCKS5 residential +egress with a native feature. Not yet configured — parked follow-up. + +**Follow-ups:** operator disables Site Magic; (optional) disable FortiGate phase1 ends for +tidiness; advertise exit nodes; self-hosted DERP at ESH on the 2G circuit; remove the +FortiGate WAN-SSH trusthosts when the edge is retired; PVE 8.4 upgrade on pfi-pve/nh3-pve.