Headscale overlay mesh — plan (2026-09-06)

Status: planning. Operator direction 2026-09-06: replace Site Magic (NH3↔ESH) and the FortiGate IPsec tunnels (colo↔NH3, colo↔ESH) with a self-hosted Tailscale-protocol overlay (Headscale), one ultra-light LXC per PVE host, keeping the old tunnels as an emergency backup. This document is the plan that a .contract.md will be cut from; nothing here is provisioned yet.

Why Headscale (decided 2026-09-06)

Tailscale clients are the best in class on the devices actually used for remote access (iPad, Mac, Linux); the future OPNsense colo edge can be a Tailscale node natively (os-tailscale); Headscale is one binary + one SQLite file to self-host, versus NetBird's six-service stack with a mandatory IdP. NetBird's built-in UI and HA routing groups are real but not decisive for three sites. UDP-blocked networks (in-flight Wi-Fi) work because the client falls back to a relay over TCP 443; plain WireGuard (ana-wg) cannot.

Site facts that drive placement

site	edge	v4	v6	change risk
ANA colo	FortiGate 80F → OPNsense on R420 within the month	public static	pending	HIGH — edge in flux; incident history 2026 (breaker, PSU1, WAN admin closed)
NH3	UDM SE, stable	dynamic-but-stable (DDNS)	single /64, reserved for meshing	low
ESH	UDM Pro Max	CGNAT now, static soon; 2G symmetric soon	/56	low, improving

Control plane at NH3 now (only stable edge; static v6 + DDNS v4 → one DNS name). Reassess ESH once its static v4 lands — it becomes the relay site regardless (2G symmetric). Never the colo: the mesh's brain must not live in the building it exists to reach in an emergency. Migration later = copy one SQLite file + move the DNS name; clients follow the URL and never re-enrol.

Topology

                       headscale.phasefinal.com  (A via DDNS, AAAA static)
                                 │ HTTPS 443 (+ DERP later)
   ┌─────────────────────────────┼───────────────────────────────┐
   │ NH3  nh3-pve                │                                │
   │   CT nh3-headscale  ──control plane (1 core / 512M / 8G)     │
   │   CT nh3-mesh-rtr   ──subnet router, advertises 10.100.0.0/16│
   ├──────────────────────────────────────────────────────────────┤
   │ ESH  esh-pve                                                 │
   │   CT esh-mesh-rtr   ──subnet router, advertises 10.0.0.0/16  │  (relay/DERP here later)
   ├──────────────────────────────────────────────────────────────┤
   │ ANA  pfi-pve                                                 │
   │   CT ana-mesh-rtr   ──subnet router, advertises 10.250.0.0/16│  (OPNsense takes this over later)
   └──────────────────────────────────────────────────────────────┘
   Per-device clients: nh3-dev, laptops/iPad, corviduo-dev, ana-ml2 … (MagicDNS names)

Four unprivileged Debian 13 LXCs, each 1 vCPU / 512 MB / 8 GB, onboot=1, backed up by the existing pbs-ana job. Routers need /dev/net/tun passed in and ip_forward (both routine for unprivileged CTs; net sysctls are namespaced).

Proposed ids/names (DHCP with reservation like every existing CT; recorded in dns/internal.yaml as <name>.<site>.internal):

PVE	next id	CT	note
nh3-pve (`root@10.100.250.60`, PVE 8.4.1)	106	`nh3-headscale`	control plane
nh3-pve	107	`nh3-mesh-rtr`	subnet router
esh-pve (`root@10.0.250.35`, PVE 8.4.20)	108	`esh-mesh-rtr`	subnet router
pfi-pve (`root@10.250.250.31`, PVE 8.3.5)	114	`ana-mesh-rtr`	subnet router; ana-wg (113) stays as independent WG fallback

debian-13-standard_13.6-1_amd64.tar.zst is in pveam available on all three (not yet downloaded on any).

Access & credentials — all in hand (verified 2026-09-06)

need	have
provision LXCs on all three PVEs	`ssh root@` works on pfi-pve, nh3-pve, esh-pve (`infra-ops@` is refused on all three PVE hosts)
public DNS name + DDNS	Cloudflare all-zones DNS-edit token, vault `nh3-dev/.config/cloudflare/infra-ops-dns-token`
NH3 UDM port-forward 443 → nh3-headscale; static routes on both UDMs	UDM API keys, vault `unifi/pfi-udmse-api-key`, `unifi/esh-udmpm-api-key` (classic `/rest/*` read+write)
colo static route toward ana-mesh-rtr	FortiGate infra-ops SSH pw vaulted; reachable at 10.250.0.1 via the tunnel (`execute backup config` first). Moot once OPNsense lands
TLS for headscale	Let's Encrypt via Cloudflare DNS-01 (same token) — no inbound 80 needed
secrets (pre-auth keys, API key, DB)	vault under `nh3-headscale/…`

Nothing outstanding on credentials. Inputs still needed from the operator: confirm names/ids above; 443 direct on the UDM vs behind the existing Caddy on nh3-dev; ACL posture (flat "everything can reach everything" first, tighten later, is the recommendation).

Phases

Pre-flight (no changes): confirm NH3 v6 /64 address for the AAAA; confirm DHCP reservation ranges on the three sites; pveam download Debian 13 on all three.
Control plane: CT nh3-headscale; headscale in a container or the .deb (prefer .deb — fewer layers in a 512M CT); Caddy/own TLS via DNS-01; UDM forward 443; DDNS updater (UDM → Cloudflare, or a ddclient/cron in the CT); DERP = Tailscale public map initially; headplane UI optional. Backup: /var/lib/headscale/db.sqlite is the only state — PBS covers the CT; add a nightly sqlite3 .backup to backupStore too.
First nodes: nh3-dev + the operator's laptop/iPad enrol → prove MagicDNS, prove TCP-443 relay path from a UDP-blocked network.
Subnet routers: three router CTs, --advertise-routes per site, approve routes in headscale, SNAT off (--snat-subnet-routes=false) so source IPs survive, static routes on each site gateway pointing the other two /16s at the local router. Test site-to-site from clientless hosts.
Cut over: move day-to-day traffic onto the mesh; disable (not delete) Site Magic and the two IPsec tunnels. Running all three at once makes route precedence murky on the UDM and produces asymmetric paths. ana-wg stays as the out-of-band WG fallback.
Later: relay (DERP) at ESH when the 2G circuit is in; OPNsense as the colo router node; HA router pairs (Tailscale HA subnet routers) per site; tighten ACLs.

Risks / open questions

Control-plane outage does not drop existing tunnels but stalls new logins and key rotation — hence the SQLite backup and the "not at the colo" rule.
Tailscale public DERPs carry relayed (encrypted) traffic through third-party infra until a self-hosted DERP exists. Acceptable at first; fix in phase 5.
CGNAT at ESH: outbound-only is fine; nothing at ESH needs to be dialled inbound.
Headscale lags Tailscale feature-wise (tailnet lock, some ACL syntax). Not relevant to phases 0–4.
VM 106 on pfi-pve was once named "PFI-Tailscale" (pre-2026 inventory). No config survives; nothing to reuse.

7.4 KiB Raw Blame History Unescape Escape