esh-pfi-infrastructure/persistent-memory.d/2026-09-06-headscale-cutover.md

2026-09-06 — Headscale cutover COMPLETE: all three site-pairs on the mesh

DONE. Operator disabled Site Magic in the UI; NH3↔ESH re-homed to a DIRECT mesh path (8ms, no DERP). Full 6-direction matrix OPEN. Site Magic disabled, both IPsec tunnels dormant, headscale is the sole active site-to-site transport. ana-wg WG fallback untouched. Tunnels re-enablable for backup.

Operator goal (/goal): replace Site Magic + IPsec with headscale, tunnels dormant as backup; "if paranoid, enable world-accessible SSH on the FortiGate first." Full detail + method + follow-ups in docs/pfi/headscale-mesh-plan.md § CUTOVER EXECUTED. Headlines:

• colo↔NH3 and colo↔ESH IPsec = DORMANT; the mesh carries both, verified bidirectional. NH3 UDM pfi-nh3-ana + ESH UDM esh-ana set enabled=false (API). Mesh /16 routes added on both UDMs and the FortiGate (→ ana-scale 10.250.50.45 / nh3-scale 10.100.50.46 / esh-scale 10.0.50.65). Dependent flows OK over mesh: restic ESH→rest-server-ana, FortiGate mgmt.
• NH3↔ESH Site Magic NOT cut by API — sdwan-mesh-tunnel = api.err.NoEdit (cloud orchestrated). Routes PRE-STAGED + shadowed; DERP path 9ms ready. Operator disables it in the UniFi UI, then the mesh takes over. Told the operator "mesh is online" → he does it.
• FortiGate WAN SSH safety net (TEMPORARY): wan1 allowaccess ping+ssh; admin infra-ops trusthost2/3 = NH3 70.230.226.88 + ESH 128.177.138.182 (static since 09-08; was CGNAT 23.164.40.160) (not 0.0.0.0). Reach it at ssh infra-ops@38.120.12.42. Config backed up flash pre-wan-ssh-cutover-20260906. Remove when the edge (being replaced by OPNsense/R420) is retired.
• ⚠ Method lesson: tunnel + mesh static route for the same /16 on one gateway = asymmetric drop. Disable the tunnel FIRST, then add the route. Broke colo once doing it tunnel-up; rolled back. See incident_crowdsec_cgnat_false_ban (same day) and the plan doc.
• Dormancy = disabled+retained (flip UDM object back to enabled=true to restore); NO auto failover wired. Bonus: exit nodes → free multi-location egress proxy (parked).

Exit nodes (2026-09-06, operator-requested)

All three routers advertise+serve exit nodes (approved). Clients pick location: tailscale set --exit-node=nh3-scale|esh-scale|ana-scale. NH3 = residential egress (70.230.226.88) → replaces the nh3-dev SOCKS5 proxy. Exit nodes + source preservation BOTH work via a selective-masquerade rule (NoSNAT kept true; mesh-exit-masq.service per router masquerades only internet-bound exit traffic, RETURNs fleet dests). Verified: colo sees real NH3 host; nh3-dev via colo exit → egress 38.120.12.42. A node advertising an exit node can't consume one — test from the laptop/iPad, not the routers.