esh-pfi-infrastructure/persistent-memory.d/2026-09-06-headscale-cutover.md

2026-09-06 — Headscale cutover: IPsec dormant, mesh primary; Site Magic pending operator

Operator goal (/goal): replace Site Magic + IPsec with headscale, tunnels dormant as backup; "if paranoid, enable world-accessible SSH on the FortiGate first." Full detail + method + follow-ups in docs/pfi/headscale-mesh-plan.md § CUTOVER EXECUTED. Headlines:

• colo↔NH3 and colo↔ESH IPsec = DORMANT; the mesh carries both, verified bidirectional. NH3 UDM pfi-nh3-ana + ESH UDM esh-ana set enabled=false (API). Mesh /16 routes added on both UDMs and the FortiGate (→ ana-scale 10.250.50.45 / nh3-scale 10.100.50.46 / esh-scale 10.0.50.65). Dependent flows OK over mesh: restic ESH→rest-server-ana, FortiGate mgmt.
• NH3↔ESH Site Magic NOT cut by API — sdwan-mesh-tunnel = api.err.NoEdit (cloud orchestrated). Routes PRE-STAGED + shadowed; DERP path 9ms ready. Operator disables it in the UniFi UI, then the mesh takes over. Told the operator "mesh is online" → he does it.
• FortiGate WAN SSH safety net (TEMPORARY): wan1 allowaccess ping+ssh; admin infra-ops trusthost2/3 = NH3 70.230.226.88 + ESH 23.164.40.160 (not 0.0.0.0). Reach it at ssh infra-ops@38.120.12.42. Config backed up flash pre-wan-ssh-cutover-20260906. Remove when the edge (being replaced by OPNsense/R420) is retired.
• ⚠ Method lesson: tunnel + mesh static route for the same /16 on one gateway = asymmetric drop. Disable the tunnel FIRST, then add the route. Broke colo once doing it tunnel-up; rolled back. See incident_crowdsec_cgnat_false_ban (same day) and the plan doc.
• Dormancy = disabled+retained (flip UDM object back to enabled=true to restore); NO auto failover wired. Bonus: exit nodes → free multi-location egress proxy (parked).