Headscale overlay mesh — plan (2026-09-06)

Status: planning. Operator direction 2026-09-06: replace Site Magic (NH3↔ESH) and the FortiGate IPsec tunnels (colo↔NH3, colo↔ESH) with a self-hosted Tailscale-protocol overlay (Headscale), one ultra-light LXC per PVE host, keeping the old tunnels as an emergency backup. This document is the plan that a .contract.md will be cut from; nothing here is provisioned yet.

Why Headscale (decided 2026-09-06)

Tailscale clients are the best in class on the devices actually used for remote access (iPad, Mac, Linux); the future OPNsense colo edge can be a Tailscale node natively (os-tailscale); Headscale is one binary + one SQLite file to self-host, versus NetBird's six-service stack with a mandatory IdP. NetBird's built-in UI and HA routing groups are real but not decisive for three sites. UDP-blocked networks (in-flight Wi-Fi) work because the client falls back to a relay over TCP 443; plain WireGuard (ana-wg) cannot.

Site facts that drive placement

site	edge	v4	v6	change risk
ANA colo	FortiGate 80F → OPNsense on R420 within the month	public static	pending	HIGH — edge in flux; incident history 2026 (breaker, PSU1, WAN admin closed)
NH3	UDM SE, stable	dynamic-but-stable (DDNS)	single /64, reserved for meshing	low
ESH	UDM Pro Max	CGNAT now, static soon; 2G symmetric soon	/56	low, improving

Control plane at NH3 now (only stable edge; static v6 + DDNS v4 → one DNS name). Reassess ESH once its static v4 lands — it becomes the relay site regardless (2G symmetric). Never the colo: the mesh's brain must not live in the building it exists to reach in an emergency. Migration later = copy one SQLite file + move the DNS name; clients follow the URL and never re-enrol.

Topology

                       headscale.phasefinal.com  (A via DDNS, AAAA static)
                                 │ HTTPS 443 (+ DERP later)
   ┌─────────────────────────────┼───────────────────────────────┐
   │ NH3  nh3-pve                │                                │
   │   CT nh3-headscale  ──control plane (1 core / 512M / 8G)     │
   │   CT nh3-mesh-rtr   ──subnet router, advertises 10.100.0.0/16│
   ├──────────────────────────────────────────────────────────────┤
   │ ESH  esh-pve                                                 │
   │   CT esh-mesh-rtr   ──subnet router, advertises 10.0.0.0/16  │  (relay/DERP here later)
   ├──────────────────────────────────────────────────────────────┤
   │ ANA  pfi-pve                                                 │
   │   CT ana-mesh-rtr   ──subnet router, advertises 10.250.0.0/16│  (OPNsense takes this over later)
   └──────────────────────────────────────────────────────────────┘
   Per-device clients: nh3-dev, laptops/iPad, corviduo-dev, ana-ml2 … (MagicDNS names)

Four unprivileged Debian 13 LXCs, each 1 vCPU / 512 MB / 8 GB, onboot=1, backed up by the existing pbs-ana job. Routers need /dev/net/tun passed in and ip_forward (both routine for unprivileged CTs; net sysctls are namespaced).

Proposed ids/names (DHCP with reservation like every existing CT; recorded in dns/internal.yaml as <name>.<site>.internal):

PVE	next id	CT	note
nh3-pve (`root@10.100.250.60`, PVE 8.4.1)	106	`nh3-headscale`	control plane
nh3-pve	107	`nh3-mesh-rtr`	subnet router
esh-pve (`root@10.0.250.35`, PVE 8.4.20)	108	`esh-mesh-rtr`	subnet router
pfi-pve (`root@10.250.250.31`, PVE 8.3.5)	114	`ana-mesh-rtr`	subnet router; ana-wg (113) stays as independent WG fallback

debian-13-standard_13.6-1_amd64.tar.zst is in pveam available on all three (not yet downloaded on any).

Access & credentials — all in hand (verified 2026-09-06)

need	have
provision LXCs on all three PVEs	`infra-ops@` with NOPASSWD sudo on all four PVE hosts since 2026-09-06 (elway `bootstrap-infra-ops-user.yaml`; PVE needed `sudo` installed first); `root@` also works
public DNS name + DDNS	Cloudflare all-zones DNS-edit token, vault `nh3-dev/.config/cloudflare/infra-ops-dns-token`
NH3 UDM port-forward 443 → nh3-headscale; static routes on both UDMs	UDM API keys, vault `unifi/pfi-udmse-api-key`, `unifi/esh-udmpm-api-key` (classic `/rest/*` read+write)
colo static route toward ana-mesh-rtr	FortiGate infra-ops SSH pw vaulted; reachable at 10.250.0.1 via the tunnel (`execute backup config` first). Moot once OPNsense lands
TLS for headscale	Let's Encrypt via Cloudflare DNS-01 (same token) — no inbound 80 needed
secrets (pre-auth keys, API key, DB)	vault under `nh3-headscale/…`

Nothing outstanding on credentials. Inputs still needed from the operator: confirm names/ids above; 443 direct on the UDM vs behind the existing Caddy on nh3-dev; ACL posture (flat "everything can reach everything" first, tighten later, is the recommendation).

Phases

Pre-flight (no changes): confirm NH3 v6 /64 address for the AAAA; confirm DHCP reservation ranges on the three sites; pveam download Debian 13 on all three.
Control plane: CT nh3-headscale; headscale in a container or the .deb (prefer .deb — fewer layers in a 512M CT); Caddy/own TLS via DNS-01; UDM forward 443; DDNS updater (UDM → Cloudflare, or a ddclient/cron in the CT); DERP = Tailscale public map initially; headplane UI optional. Backup: /var/lib/headscale/db.sqlite is the only state — PBS covers the CT; add a nightly sqlite3 .backup to backupStore too.
First nodes: nh3-dev + the operator's laptop/iPad enrol → prove MagicDNS, prove TCP-443 relay path from a UDP-blocked network.
Subnet routers: three router CTs, --advertise-routes per site, approve routes in headscale, SNAT off (--snat-subnet-routes=false) so source IPs survive, static routes on each site gateway pointing the other two /16s at the local router. Test site-to-site from clientless hosts.
Cut over: move day-to-day traffic onto the mesh; disable (not delete) Site Magic and the two IPsec tunnels. Running all three at once makes route precedence murky on the UDM and produces asymmetric paths. ana-wg stays as the out-of-band WG fallback.
Later: relay (DERP) at ESH when the 2G circuit is in; OPNsense as the colo router node; HA router pairs (Tailscale HA subnet routers) per site; tighten ACLs.

Risks / open questions

Control-plane outage does not drop existing tunnels but stalls new logins and key rotation — hence the SQLite backup and the "not at the colo" rule.
Tailscale public DERPs carry relayed (encrypted) traffic through third-party infra until a self-hosted DERP exists. Acceptable at first; fix in phase 5.
CGNAT at ESH: outbound-only is fine; nothing at ESH needs to be dialled inbound.
Headscale lags Tailscale feature-wise (tailnet lock, some ACL syntax). Not relevant to phases 0–4.
VM 106 on pfi-pve was once named "PFI-Tailscale" (pre-2026 inventory). No config survives; nothing to reuse.

Status — 2026-09-06 05:35Z: phases 0–1 done, phase 3 half-done, NOT cut over

Operator inputs (05:00Z): 443 direct on the UDM; names nh3-headscale, nh3-scale, esh-scale, ana-scale; ids my call → sequential. ACLs: flat (database policy mode, empty policy) until told otherwise.

Live:

what	where	detail
control plane	CT 106 `nh3-headscale` 10.100.50.45 (nh3-pve, Debian 12, 1c/512M/8G)	headscale v0.29.3 (.deb), `https://headscale.phasefinal.com` on 443 direct, Let's Encrypt via TLS-ALPN-01 (no Caddy, no DNS-01), SQLite at `/var/lib/headscale/db.sqlite`, DERP = Tailscale public map, MagicDNS base `mesh.phasefinal.com`, split DNS `internal` → the three AdGuards, `override_local_dns: false`. Public health check 200 with a valid cert.
public name	Cloudflare A `headscale.phasefinal.com` → NH3 WAN (70.230.226.88 at creation), unproxied, TTL 120	DDNS: `~/.local/bin/headscale-ddns.sh` on nh3-dev, user timer `headscale-ddns.timer` every 10 min, token pulled from the vault at run time (nothing stored in the CT). No AAAA yet — NH3 LAN v6 is deliberately off.
UDM forward	NH3 UDM SE `headscale-443`: WAN tcp/443 → 10.100.50.45:443	added via the classic REST API. Hairpin from inside NH3 works.
routers	CT 107 `nh3-scale` 10.100.50.46 (100.64.0.1) · CT 108 `esh-scale` 10.0.50.65 (100.64.0.2, Debian 13) · CT 114 `ana-scale` 10.250.50.45 (100.64.0.3)	tailscale 1.102.3, `/dev/net/tun` passed, forwarding on, each advertising its site /16, routes approved and serving, `--snat-subnet-routes=false`, `--accept-routes=false` (see lesson 1).
first client	nh3-dev (100.64.0.4)	`--accept-routes=false`, `--accept-dns=false`. Direct paths to all three routers (7–10 ms).
DNS	`*.internal` names for all four CTs synced to the three AdGuards
backups	pfi-pve + nh3-pve jobs are `all`; esh-pve job vmid list extended with 108
vault	`headscale/preauth-router-48h-20260906`, `headscale/preauth-client-48h-20260906` (reusable, expire 2026-09-08)	mint fresh ones on the CT: `headscale preauthkeys create --user 1 --reusable --expiration 48h`

Lessons (do not repeat):

⚠ --accept-routes on a client BEFORE the return path exists breaks that client's LAN. nh3-dev accepted all three /16s including its OWN site's; Linux table-52 routes beat the main table, so every 10.100.x packet went into the tunnel to nh3-scale and nothing came back (no SNAT → the far host has no route to 100.64/10). Took ssh to nh3-pve, esh-pve and pfi-pve down for ~1 min until tailscale set --accept-routes=false. Phase order is therefore: return path first (either SNAT on the routers, or static routes for 100.64.0.0/10 → local router on every site gateway), then accept-routes on clients, and exclude a node's own site route.
The routers currently find each other "direct" via LAN addresses (10.0.50.65:41641 etc.) — i.e. the mesh is riding inside the old Site Magic / IPsec tunnels. Public- endpoint / relay path discovery is untested until those tunnels are disabled (phase 4). Test that with one tunnel down before declaring cut-over ready.
pfi-pve (PVE 8.3.5) and nh3-pve (8.4.1) refuse the Debian 13 template (unsupported debian version '13.6'); esh-pve (8.4.20) accepts it. Three CTs are Debian 12, esh-scale is Debian 13. A failed pct create followed by an >> into the conf leaves a stub conf that makes pct say "missing arch" — delete the stub. Follow-up: bring pfi-pve/nh3-pve up to current 8.4.
headscale 0.29 removed randomize_client_port and renamed ephemeral_node_inactivity_timeout → node.ephemeral.inactivity_timeout; with a bad key headscale serve silently falls back to defaults (127.0.0.1:8080). Always headscale configtest first.

Next (in order): (a) decide the return path — recommendation: static routes for 100.64.0.0/10 on each site gateway → local router (keeps source IPs), SNAT as the fallback; (b) enrol the operator's devices with the client key; (c) test a UDP-blocked path (relay over 443); (d) disable ONE old tunnel and re-test router paths; (e) cut over: static routes for the other two /16s on each gateway, disable Site Magic + IPsec; (f) v6 AAAA once NH3 LAN v6 is on; (g) self-hosted DERP at ESH; (h) PVE upgrades on pfi-pve/nh3-pve.

12 KiB Raw Blame History Unescape Escape