esh-pfi-infrastructure/docs/pfi/headscale-mesh-plan.md

Headscale overlay mesh — plan (2026-09-06)

Status: planning. Operator direction 2026-09-06: replace Site Magic (NH3↔ESH) and the FortiGate IPsec tunnels (colo↔NH3, colo↔ESH) with a self-hosted Tailscale-protocol overlay (Headscale), one ultra-light LXC per PVE host, keeping the old tunnels as an emergency backup. This document is the plan that a .contract.md will be cut from; nothing here is provisioned yet.

Why Headscale (decided 2026-09-06)

Tailscale clients are the best in class on the devices actually used for remote access (iPad, Mac, Linux); the future OPNsense colo edge can be a Tailscale node natively (os-tailscale); Headscale is one binary + one SQLite file to self-host, versus NetBird's six-service stack with a mandatory IdP. NetBird's built-in UI and HA routing groups are real but not decisive for three sites. UDP-blocked networks (in-flight Wi-Fi) work because the client falls back to a relay over TCP 443; plain WireGuard (ana-wg) cannot.

Site facts that drive placement

site	edge	v4	v6	change risk
ANA colo	FortiGate 80F → OPNsense on R420 within the month	public static	pending	HIGH — edge in flux; incident history 2026 (breaker, PSU1, WAN admin closed)
NH3	UDM SE, stable	dynamic-but-stable (DDNS)	single /64, reserved for meshing	low
ESH	UDM Pro Max	CGNAT now, static soon; 2G symmetric soon	/56	low, improving

Control plane at NH3 now (only stable edge; static v6 + DDNS v4 → one DNS name). Reassess ESH once its static v4 lands — it becomes the relay site regardless (2G symmetric). Never the colo: the mesh's brain must not live in the building it exists to reach in an emergency. Migration later = copy one SQLite file + move the DNS name; clients follow the URL and never re-enrol.

Topology

                       headscale.phasefinal.com  (A via DDNS, AAAA static)
                                 │ HTTPS 443 (+ DERP later)
   ┌─────────────────────────────┼───────────────────────────────┐
   │ NH3  nh3-pve                │                                │
   │   CT nh3-headscale  ──control plane (1 core / 512M / 8G)     │
   │   CT nh3-mesh-rtr   ──subnet router, advertises 10.100.0.0/16│
   ├──────────────────────────────────────────────────────────────┤
   │ ESH  esh-pve                                                 │
   │   CT esh-mesh-rtr   ──subnet router, advertises 10.0.0.0/16  │  (relay/DERP here later)
   ├──────────────────────────────────────────────────────────────┤
   │ ANA  pfi-pve                                                 │
   │   CT ana-mesh-rtr   ──subnet router, advertises 10.250.0.0/16│  (OPNsense takes this over later)
   └──────────────────────────────────────────────────────────────┘
   Per-device clients: nh3-dev, laptops/iPad, corviduo-dev, ana-ml2 … (MagicDNS names)

Four unprivileged Debian 13 LXCs, each 1 vCPU / 512 MB / 8 GB, onboot=1, backed up by the existing pbs-ana job. Routers need /dev/net/tun passed in and ip_forward (both routine for unprivileged CTs; net sysctls are namespaced).

Proposed ids/names (DHCP with reservation like every existing CT; recorded in dns/internal.yaml as <name>.<site>.internal):

PVE	next id	CT	note
nh3-pve (root@10.100.250.60, PVE 8.4.1)	106	nh3-headscale	control plane
nh3-pve	107	nh3-mesh-rtr	subnet router
esh-pve (root@10.0.250.35, PVE 8.4.20)	108	esh-mesh-rtr	subnet router
pfi-pve (root@10.250.250.31, PVE 8.3.5)	114	ana-mesh-rtr	subnet router; ana-wg (113) stays as independent WG fallback

debian-13-standard_13.6-1_amd64.tar.zst is in pveam available on all three (not yet downloaded on any).

Access & credentials — all in hand (verified 2026-09-06)

need	have
provision LXCs on all three PVEs	infra-ops@ with NOPASSWD sudo on all four PVE hosts since 2026-09-06 (elway bootstrap-infra-ops-user.yaml; PVE needed sudo installed first); root@ also works
public DNS name + DDNS	Cloudflare all-zones DNS-edit token, vault nh3-dev/.config/cloudflare/infra-ops-dns-token
NH3 UDM port-forward 443 → nh3-headscale; static routes on both UDMs	UDM API keys, vault unifi/pfi-udmse-api-key, unifi/esh-udmpm-api-key (classic /rest/* read+write)
colo static route toward ana-mesh-rtr	FortiGate infra-ops SSH pw vaulted; reachable at 10.250.0.1 via the tunnel (execute backup config first). Moot once OPNsense lands
TLS for headscale	Let's Encrypt via Cloudflare DNS-01 (same token) — no inbound 80 needed
secrets (pre-auth keys, API key, DB)	vault under nh3-headscale/…

Nothing outstanding on credentials. Inputs still needed from the operator: confirm names/ids above; 443 direct on the UDM vs behind the existing Caddy on nh3-dev; ACL posture (flat "everything can reach everything" first, tighten later, is the recommendation).

Phases

0. Pre-flight (no changes): confirm NH3 v6 /64 address for the AAAA; confirm DHCP reservation ranges on the three sites; pveam download Debian 13 on all three.
1. Control plane: CT nh3-headscale; headscale in a container or the .deb (prefer .deb — fewer layers in a 512M CT); Caddy/own TLS via DNS-01; UDM forward 443; DDNS updater (UDM → Cloudflare, or a ddclient/cron in the CT); DERP = Tailscale public map initially; headplane UI optional. Backup: /var/lib/headscale/db.sqlite is the only state — PBS covers the CT; add a nightly sqlite3 .backup to backupStore too.
2. First nodes: nh3-dev + the operator's laptop/iPad enrol → prove MagicDNS, prove TCP-443 relay path from a UDP-blocked network.
3. Subnet routers: three router CTs, --advertise-routes per site, approve routes in headscale, SNAT off (--snat-subnet-routes=false) so source IPs survive, static routes on each site gateway pointing the other two /16s at the local router. Test site-to-site from clientless hosts.
4. Cut over: move day-to-day traffic onto the mesh; disable (not delete) Site Magic and the two IPsec tunnels. Running all three at once makes route precedence murky on the UDM and produces asymmetric paths. ana-wg stays as the out-of-band WG fallback.
5. Later: relay (DERP) at ESH when the 2G circuit is in; OPNsense as the colo router node; HA router pairs (Tailscale HA subnet routers) per site; tighten ACLs.

Risks / open questions

• Control-plane outage does not drop existing tunnels but stalls new logins and key rotation — hence the SQLite backup and the "not at the colo" rule.
• Tailscale public DERPs carry relayed (encrypted) traffic through third-party infra until a self-hosted DERP exists. Acceptable at first; fix in phase 5.
• CGNAT at ESH: outbound-only is fine; nothing at ESH needs to be dialled inbound.
• Headscale lags Tailscale feature-wise (tailnet lock, some ACL syntax). Not relevant to phases 0–4.
• VM 106 on pfi-pve was once named "PFI-Tailscale" (pre-2026 inventory). No config survives; nothing to reuse.

Status — 2026-09-06 05:35Z: phases 0–1 done, phase 3 half-done, NOT cut over

Operator inputs (05:00Z): 443 direct on the UDM; names nh3-headscale, nh3-scale, esh-scale, ana-scale; ids my call → sequential. ACLs: flat (database policy mode, empty policy) until told otherwise.

Live:

what	where	detail
control plane	CT 106 nh3-headscale 10.100.50.45 (nh3-pve, Debian 12, 1c/512M/8G)	headscale v0.29.3 (.deb), https://headscale.phasefinal.com on 443 direct, Let's Encrypt via TLS-ALPN-01 (no Caddy, no DNS-01), SQLite at /var/lib/headscale/db.sqlite, DERP = Tailscale public map, MagicDNS base mesh.phasefinal.com, split DNS internal → the three AdGuards, override_local_dns: false. Public health check 200 with a valid cert.
public name	Cloudflare A headscale.phasefinal.com → NH3 WAN (70.230.226.88 at creation), unproxied, TTL 120	DDNS: ~/.local/bin/headscale-ddns.sh on nh3-dev, user timer headscale-ddns.timer every 10 min, token pulled from the vault at run time (nothing stored in the CT). No AAAA yet — NH3 LAN v6 is deliberately off.
UDM forward	NH3 UDM SE headscale-443: WAN tcp/443 → 10.100.50.45:443	added via the classic REST API. Hairpin from inside NH3 works.
routers	CT 107 nh3-scale 10.100.50.46 (100.64.0.1) · CT 108 esh-scale 10.0.50.65 (100.64.0.2, Debian 13) · CT 114 ana-scale 10.250.50.45 (100.64.0.3)	tailscale 1.102.3, /dev/net/tun passed, forwarding on, each advertising its site /16, routes approved and serving, --snat-subnet-routes=false, --accept-routes=false (see lesson 1).
first client	nh3-dev (100.64.0.4)	--accept-routes=false, --accept-dns=false. Direct paths to all three routers (7–10 ms).
DNS	*.internal names for all four CTs synced to the three AdGuards
backups	pfi-pve + nh3-pve jobs are all; esh-pve job vmid list extended with 108
vault	headscale/preauth-router-48h-20260906, headscale/preauth-client-48h-20260906 (reusable, expire 2026-09-08)	mint fresh ones on the CT: headscale preauthkeys create --user 1 --reusable --expiration 48h

Lessons (do not repeat):

1. ⚠ --accept-routes on a client BEFORE the return path exists breaks that client's LAN. nh3-dev accepted all three /16s including its OWN site's; Linux table-52 routes beat the main table, so every 10.100.x packet went into the tunnel to nh3-scale and nothing came back (no SNAT → the far host has no route to 100.64/10). Took ssh to nh3-pve, esh-pve and pfi-pve down for ~1 min until tailscale set --accept-routes=false. Phase order is therefore: return path first (either SNAT on the routers, or static routes for 100.64.0.0/10 → local router on every site gateway), then accept-routes on clients, and exclude a node's own site route.
2. The routers currently find each other "direct" via LAN addresses (10.0.50.65:41641 etc.) — i.e. the mesh is riding inside the old Site Magic / IPsec tunnels. Public- endpoint / relay path discovery is untested until those tunnels are disabled (phase 4). Test that with one tunnel down before declaring cut-over ready.
3. pfi-pve (PVE 8.3.5) and nh3-pve (8.4.1) refuse the Debian 13 template (unsupported debian version '13.6'); esh-pve (8.4.20) accepts it. Three CTs are Debian 12, esh-scale is Debian 13. A failed pct create followed by an >> into the conf leaves a stub conf that makes pct say "missing arch" — delete the stub. Follow-up: bring pfi-pve/nh3-pve up to current 8.4.
4. headscale 0.29 removed randomize_client_port and renamed ephemeral_node_inactivity_timeout → node.ephemeral.inactivity_timeout; with a bad key headscale serve silently falls back to defaults (127.0.0.1:8080). Always headscale configtest first.

Next (in order): (a) decide the return path — recommendation: static routes for 100.64.0.0/10 on each site gateway → local router (keeps source IPs), SNAT as the fallback; (b) enrol the operator's devices with the client key; (c) test a UDP-blocked path (relay over 443); (d) disable ONE old tunnel and re-test router paths; (e) cut over: static routes for the other two /16s on each gateway, disable Site Magic + IPsec; (f) v6 AAAA once NH3 LAN v6 is on; (g) self-hosted DERP at ESH; (h) PVE upgrades on pfi-pve/nh3-pve.

06:05Z — return path built and verified (operator approved 05:50Z)

Static route 100.64.0.0/10 → local router on all three gateways: NH3 UDM (mesh-100.64-via-nh3-scale → 10.100.50.46, classic REST), ESH UDM (mesh-100.64-via-esh-scale → 10.0.50.65; ⚠ that vault item is a 19-line note — extract with secret get unifi/esh-udmpm-api-key | grep '^key:' | awk '{print $2}', a raw pipe gives nginx 400 and the NH3 key gives 401), FortiGate (config router static, dst 100.64.0.0/255.192.0.0, gw 10.250.50.45, device servers; config backed up to flash as pre-mesh-route-20260906 first). Routers re-set to --accept-routes=true — a router MUST accept the other sites' routes or its replies to remote-LAN sources leave via its default gateway with a 100.64 source and die; a node never accepts its own advertised route, so the nh3-dev failure mode does not apply to routers.

Verified (LAN host → gateway → local router → mesh → far router, and back): nh3-docker→ana-scale 9 ms, esh-docker-vm→ana-scale and →nh3-scale 0% loss, ana-docker→nh3-scale and →esh-scale 0% loss. All three sites can reach mesh addresses from plain LAN hosts with no client installed.

LAN-resident clients (nh3-dev) keep --accept-routes=false: they reach other sites via their gateway like any LAN host. --accept-routes is for roaming devices (laptop/iPad off-site) only. Consequence: a LAN-resident client is reachable by its mesh IP only from the mesh, not from other sites' LANs (ana-docker→100.64.0.4 fails by design; →100.64.0.1 works).

Enrolling GUI devices (Mac/iPad) without OIDC: the apps do not take a pre-auth key. Point the app at https://headscale.phasefinal.com, it opens a registration page showing a headscale nodes register --user pfi --key mkey:… command; run that on nh3-headscale (pct exec 106 -- headscale nodes register --user 1 --key <mkey>). CLI Linux/macOS builds can use the vaulted client key instead.

06:20Z — first roaming client verified

Operator's MacBook enrolled via the GUI (Option-click → Debug → Custom Login Server; headscale 0.29 registration command is headscale auth register --auth-id hskey-authreq-… --user pfi), node 5 vhlk-mba26 (100.64.0.5), "Use Tailscale subnets" on. From ESH: ping 100.64.0.3 and ssh infra-ops@ana-docker.ana.internal both work → colo subnet route + split DNS for *.internal proven from a client. (Path was still via the ESH LAN / old tunnels.)

Note — the mesh is also the durable fix for CGNAT CrowdSec false-bans

2026-09-06: CrowdSec on ana-docker banned ESH's shared CGNAT egress (23.164.40.160, custom/gitea-aggressive-crawl) and the fortigate-mirror bouncer pushed it to the colo edge, blackholing Matrix/gitea/chat for the whole ESH site (see ~/.claude/.../memory/incident_crowdsec_cgnat_false_ban.md). Once ESH consumes colo services over the mesh (100.64/10 via ana-scale) rather than the public FortiGate VIP, that traffic never reaches CrowdSec — a concrete win beyond replacing the tunnels, worth weighing when prioritising the cut-over.

06:35Z — ESH egress whitelisted in CrowdSec (operator-directed, temporary)

/opt/docker/conf/crowdsec/postoverflows/s01-whitelist/pfi-esh-egress.yaml (bind mount, persists) whitelists 23.164.40.160; SIGHUP reload, crowdsec -t clean, parser loaded. TEMPORARY — remove when ESH gets its static IP. crowdsec stack is NOT in stacks/ canonical (only stacks-mirror), so this lives on the host; re-mirror with sync-stacks.sh.

CUTOVER EXECUTED 2026-09-06 — IPsec dormant, mesh primary; Site Magic pending operator

Operator goal: replace Site Magic + IPsec with headscale, tunnels dormant as backup.

Safety net first: FortiGate WAN SSH enabled — wan1 set allowaccess ping ssh, admin infra-ops trusthost2/3 = 70.230.226.88/32 (NH3 egress) + 23.164.40.160/32 (ESH egress); trusthost1 10.0.0.0/8 kept. Config backed up to flash pre-wan-ssh-cutover-20260906. Verified: ssh infra-ops@38.120.12.42 works from nh3-dev, tunnel-independent. ⚠ TEMPORARY — scoped to two source IPs, not 0.0.0.0; remove after the edge is retired. Drive the FortiGate via WAN SSH (scratchpad/fgw.py) during/after cutover, not over the tunnel.

Method (learned the hard way): the tunnel and a mesh static route for the same /16 on the same gateway FIGHT — forward/return split across transports, asymmetric traffic dropped (broke colo from NH3 at 06:xx while both were up; rolled back). Correct order per pair: disable the tunnel end(s) FIRST (route withdraws), THEN add the mesh /16 route (now unambiguous). Never do it with the tunnel up.

colo↔NH3 (IPsec): NH3 UDM pfi-nh3-ana disabled (enabled=false, API); mesh routes NH3 10.250.0.0/16→10.100.50.46 (nh3-scale), FortiGate 10.100.0.0/16→10.250.50.45 (ana-scale). Verified: nh3-dev→ana-docker + pfi-pve OPEN; FortiGate reachable via mesh AND WAN SSH; ESH→rest-server-ana (restic) OPEN. FortiGate phase1 left enabled (no peer → idle); re-enable NH3 side to restore.

colo↔ESH (IPsec): ESH UDM esh-ana disabled; mesh routes ESH 10.250.0.0/16→10.0.50.65 (esh-scale), FortiGate 10.0.0.0/16→10.250.50.45. Verified bidirectional: esh-docker-vm↔ana-docker/pfi-pve OPEN both ways. Proves ESH (CGNAT) mesh over public/DERP.

NH3↔ESH (Site Magic) — NOT done by API. sdwan-mesh-tunnel objects (ESH-UDMPM on NH3, PFI-UDMSE on ESH) return api.err.NoEdit — Site Magic is cloud-orchestrated, not togglable via the classic REST API. Mesh routes PRE-STAGED (NH3 10.0.0.0/16→nh3-scale, ESH 10.100.0.0/16→esh-scale), currently SHADOWED by Site Magic (traffic still on the SD-WAN tunnel, traceroute via 192.168.1.x transit). esh-scale↔nh3-scale hold a DERP(lax) 9ms path independent of Site Magic → when the operator disables Site Magic in the UniFi UI / Site Manager, the staged routes take over immediately. Operator to toggle; mesh verified ready.

Dormancy = disabled-but-retained, not auto-failover (overlapping /16s across two transports reintroduce the asymmetry). Re-enable a pair = flip the UDM object back to enabled=true (+ re-enable Site Magic in UI). No automatic mesh→tunnel failover is wired.

Bonus (operator noted): exit nodes at each site give a free multi-location egress proxy (tailscale up --advertise-exit-node + approve); replaces the nh3-dev SOCKS5 residential egress with a native feature. Not yet configured — parked follow-up.

Follow-ups: operator disables Site Magic; (optional) disable FortiGate phase1 ends for tidiness; advertise exit nodes; self-hosted DERP at ESH on the 2G circuit; remove the FortiGate WAN-SSH trusthosts when the edge is retired; PVE 8.4 upgrade on pfi-pve/nh3-pve.

✅ CUTOVER COMPLETE 2026-09-06 — all three site-pairs on the mesh

Operator disabled Site Magic in the UniFi UI. NH3↔ESH re-homed to the mesh immediately — traceroute now esh-scale(10.0.50.65)→nh3-scale(100.64.0.1)→dest, no 192.168.1.x SD-WAN transit; esh-scale↔nh3-scale hole-punched a DIRECT public path (70.230.226.88:41641, 8ms), not even DERP. Full matrix verified, all six directions OPEN: NH3↔colo, NH3↔ESH, colo↔ESH. FortiGate mgmt via WAN SSH OPEN; ana-wg WG fallback untouched.

State: Site Magic disabled (UI); both IPsec tunnels dormant (UDM ends disabled, config retained); headscale is the sole active site-to-site transport. Tunnels re-enablable for backup (Site Magic in UI; IPsec by flipping the UDM objects enabled=true). Goal met.

2026-09-06 — exit nodes advertised (multi-location egress proxy)

All three routers advertise + serve exit nodes; approved on headscale (each shows 0.0.0.0/0 + ::/0 Serving). Clients see three: nh3-scale (100.64.0.1, NH3 residential egress 70.230.226.88), esh-scale (100.64.0.2, ESH 23.164.40.160), ana-scale (100.64.0.3, colo 38.120.12.42).

⚠ Design reversal: exit nodes REQUIRE SNAT, so --snat-subnet-routes was flipped back to true (NoSNAT=false) on all three routers — this undoes the earlier source-preserving choice. Consequence: cross-site subnet-route traffic now appears to come from the local router's IP, not the original host. Site-to-site re-verified OK after the flip. If per-host source visibility is later needed, the alternative is NoSNAT + a manual selective masquerade rule (-s 100.64.0.0/10 ! -d 10.0.0.0/8 -o eth0 -j MASQUERADE) on each router, made persistent — not done.

Use one from a client: tailscale set --exit-node=nh3-scale (or esh-scale / ana-scale), --exit-node= to clear. A node that itself advertises an exit node cannot consume one (so the routers can't be used to test egress; use the laptop/iPad). Replaces the nh3-dev SOCKS5 residential-egress proxy — retire that once confirmed on real clients.