165 lines
12 KiB
Markdown
165 lines
12 KiB
Markdown
# Headscale overlay mesh — plan (2026-09-06)
|
||
|
||
**Status:** planning. Operator direction 2026-09-06: replace Site Magic (NH3↔ESH) and the
|
||
FortiGate IPsec tunnels (colo↔NH3, colo↔ESH) with a self-hosted Tailscale-protocol overlay
|
||
(Headscale), one ultra-light LXC per PVE host, keeping the old tunnels as an emergency
|
||
backup. This document is the plan that a `.contract.md` will be cut from; nothing here is
|
||
provisioned yet.
|
||
|
||
## Why Headscale (decided 2026-09-06)
|
||
|
||
Tailscale clients are the best in class on the devices actually used for remote access
|
||
(iPad, Mac, Linux); the future OPNsense colo edge can be a Tailscale node natively
|
||
(`os-tailscale`); Headscale is one binary + one SQLite file to self-host, versus NetBird's
|
||
six-service stack with a mandatory IdP. NetBird's built-in UI and HA routing groups are
|
||
real but not decisive for three sites. UDP-blocked networks (in-flight Wi-Fi) work because
|
||
the client falls back to a relay over TCP 443; plain WireGuard (ana-wg) cannot.
|
||
|
||
## Site facts that drive placement
|
||
|
||
| site | edge | v4 | v6 | change risk |
|
||
|---|---|---|---|---|
|
||
| ANA colo | FortiGate 80F → **OPNsense on R420 within the month** | public static | pending | HIGH — edge in flux; incident history 2026 (breaker, PSU1, WAN admin closed) |
|
||
| NH3 | UDM SE, stable | dynamic-but-stable (DDNS) | single /64, **reserved for meshing** | low |
|
||
| ESH | UDM Pro Max | CGNAT now, **static soon**; 2G symmetric soon | /56 | low, improving |
|
||
|
||
**Control plane at NH3 now** (only stable edge; static v6 + DDNS v4 → one DNS name).
|
||
**Reassess ESH** once its static v4 lands — it becomes the relay site regardless (2G
|
||
symmetric). **Never the colo:** the mesh's brain must not live in the building it exists
|
||
to reach in an emergency. Migration later = copy one SQLite file + move the DNS name;
|
||
clients follow the URL and never re-enrol.
|
||
|
||
## Topology
|
||
|
||
```
|
||
headscale.phasefinal.com (A via DDNS, AAAA static)
|
||
│ HTTPS 443 (+ DERP later)
|
||
┌─────────────────────────────┼───────────────────────────────┐
|
||
│ NH3 nh3-pve │ │
|
||
│ CT nh3-headscale ──control plane (1 core / 512M / 8G) │
|
||
│ CT nh3-mesh-rtr ──subnet router, advertises 10.100.0.0/16│
|
||
├──────────────────────────────────────────────────────────────┤
|
||
│ ESH esh-pve │
|
||
│ CT esh-mesh-rtr ──subnet router, advertises 10.0.0.0/16 │ (relay/DERP here later)
|
||
├──────────────────────────────────────────────────────────────┤
|
||
│ ANA pfi-pve │
|
||
│ CT ana-mesh-rtr ──subnet router, advertises 10.250.0.0/16│ (OPNsense takes this over later)
|
||
└──────────────────────────────────────────────────────────────┘
|
||
Per-device clients: nh3-dev, laptops/iPad, corviduo-dev, ana-ml2 … (MagicDNS names)
|
||
```
|
||
|
||
Four unprivileged Debian 13 LXCs, each 1 vCPU / 512 MB / 8 GB, `onboot=1`, backed up by
|
||
the existing `pbs-ana` job. Routers need `/dev/net/tun` passed in and `ip_forward` (both
|
||
routine for unprivileged CTs; net sysctls are namespaced).
|
||
|
||
Proposed ids/names (DHCP with reservation like every existing CT; recorded in
|
||
`dns/internal.yaml` as `<name>.<site>.internal`):
|
||
|
||
| PVE | next id | CT | note |
|
||
|---|---|---|---|
|
||
| nh3-pve (`root@10.100.250.60`, PVE 8.4.1) | 106 | `nh3-headscale` | control plane |
|
||
| nh3-pve | 107 | `nh3-mesh-rtr` | subnet router |
|
||
| esh-pve (`root@10.0.250.35`, PVE 8.4.20) | 108 | `esh-mesh-rtr` | subnet router |
|
||
| pfi-pve (`root@10.250.250.31`, PVE 8.3.5) | 114 | `ana-mesh-rtr` | subnet router; ana-wg (113) stays as independent WG fallback |
|
||
|
||
`debian-13-standard_13.6-1_amd64.tar.zst` is in `pveam available` on all three (not yet
|
||
downloaded on any).
|
||
|
||
## Access & credentials — all in hand (verified 2026-09-06)
|
||
|
||
| need | have |
|
||
|---|---|
|
||
| provision LXCs on all three PVEs | `infra-ops@` with NOPASSWD sudo on all four PVE hosts since 2026-09-06 (elway `bootstrap-infra-ops-user.yaml`; PVE needed `sudo` installed first); `root@` also works |
|
||
| public DNS name + DDNS | Cloudflare all-zones DNS-edit token, vault `nh3-dev/.config/cloudflare/infra-ops-dns-token` |
|
||
| NH3 UDM port-forward 443 → nh3-headscale; static routes on both UDMs | UDM API keys, vault `unifi/pfi-udmse-api-key`, `unifi/esh-udmpm-api-key` (classic `/rest/*` read+write) |
|
||
| colo static route toward ana-mesh-rtr | FortiGate infra-ops SSH pw vaulted; reachable at 10.250.0.1 via the tunnel (`execute backup config` first). Moot once OPNsense lands |
|
||
| TLS for headscale | Let's Encrypt via Cloudflare DNS-01 (same token) — no inbound 80 needed |
|
||
| secrets (pre-auth keys, API key, DB) | vault under `nh3-headscale/…` |
|
||
|
||
Nothing outstanding on credentials. Inputs still needed from the operator: confirm names/ids
|
||
above; 443 direct on the UDM vs behind the existing Caddy on nh3-dev; ACL posture (flat
|
||
"everything can reach everything" first, tighten later, is the recommendation).
|
||
|
||
## Phases
|
||
|
||
0. **Pre-flight (no changes):** confirm NH3 v6 /64 address for the AAAA; confirm DHCP
|
||
reservation ranges on the three sites; `pveam download` Debian 13 on all three.
|
||
1. **Control plane:** CT `nh3-headscale`; headscale in a container or the .deb (prefer .deb —
|
||
fewer layers in a 512M CT); Caddy/own TLS via DNS-01; UDM forward 443; DDNS updater
|
||
(UDM → Cloudflare, or a `ddclient`/cron in the CT); DERP = Tailscale public map
|
||
initially; headplane UI optional. Backup: `/var/lib/headscale/db.sqlite` is the only
|
||
state — PBS covers the CT; add a nightly `sqlite3 .backup` to `backupStore` too.
|
||
2. **First nodes:** nh3-dev + the operator's laptop/iPad enrol → prove MagicDNS, prove
|
||
TCP-443 relay path from a UDP-blocked network.
|
||
3. **Subnet routers:** three router CTs, `--advertise-routes` per site, approve routes in
|
||
headscale, SNAT off (`--snat-subnet-routes=false`) so source IPs survive, static routes
|
||
on each site gateway pointing the other two /16s at the local router. Test
|
||
site-to-site from clientless hosts.
|
||
4. **Cut over:** move day-to-day traffic onto the mesh; **disable (not delete)** Site Magic
|
||
and the two IPsec tunnels. Running all three at once makes route precedence murky on
|
||
the UDM and produces asymmetric paths. ana-wg stays as the out-of-band WG fallback.
|
||
5. **Later:** relay (DERP) at ESH when the 2G circuit is in; OPNsense as the colo router
|
||
node; HA router pairs (Tailscale HA subnet routers) per site; tighten ACLs.
|
||
|
||
## Risks / open questions
|
||
|
||
- Control-plane outage does not drop existing tunnels but stalls new logins and key
|
||
rotation — hence the SQLite backup and the "not at the colo" rule.
|
||
- Tailscale public DERPs carry relayed (encrypted) traffic through third-party infra until
|
||
a self-hosted DERP exists. Acceptable at first; fix in phase 5.
|
||
- CGNAT at ESH: outbound-only is fine; nothing at ESH needs to be dialled inbound.
|
||
- Headscale lags Tailscale feature-wise (tailnet lock, some ACL syntax). Not relevant to
|
||
phases 0–4.
|
||
- VM 106 on pfi-pve was once named "PFI-Tailscale" (pre-2026 inventory). No config
|
||
survives; nothing to reuse.
|
||
|
||
## Status — 2026-09-06 05:35Z: phases 0–1 done, phase 3 half-done, NOT cut over
|
||
|
||
Operator inputs (05:00Z): 443 direct on the UDM; names `nh3-headscale`, `nh3-scale`,
|
||
`esh-scale`, `ana-scale`; ids my call → sequential. ACLs: flat (database policy mode,
|
||
empty policy) until told otherwise.
|
||
|
||
**Live:**
|
||
|
||
| what | where | detail |
|
||
|---|---|---|
|
||
| control plane | CT 106 `nh3-headscale` 10.100.50.45 (nh3-pve, Debian 12, 1c/512M/8G) | headscale **v0.29.3** (.deb), `https://headscale.phasefinal.com` on **443 direct**, Let's Encrypt via TLS-ALPN-01 (no Caddy, no DNS-01), SQLite at `/var/lib/headscale/db.sqlite`, DERP = Tailscale public map, MagicDNS base `mesh.phasefinal.com`, split DNS `internal` → the three AdGuards, `override_local_dns: false`. Public health check 200 with a valid cert. |
|
||
| public name | Cloudflare A `headscale.phasefinal.com` → NH3 WAN (70.230.226.88 at creation), unproxied, TTL 120 | **DDNS:** `~/.local/bin/headscale-ddns.sh` on nh3-dev, user timer `headscale-ddns.timer` every 10 min, token pulled from the vault at run time (nothing stored in the CT). No AAAA yet — NH3 LAN v6 is deliberately off. |
|
||
| UDM forward | NH3 UDM SE `headscale-443`: WAN tcp/443 → 10.100.50.45:443 | added via the classic REST API. Hairpin from inside NH3 works. |
|
||
| routers | CT 107 `nh3-scale` 10.100.50.46 (100.64.0.1) · CT 108 `esh-scale` 10.0.50.65 (100.64.0.2, Debian 13) · CT 114 `ana-scale` 10.250.50.45 (100.64.0.3) | tailscale 1.102.3, `/dev/net/tun` passed, forwarding on, each advertising its site /16, **routes approved and serving**, `--snat-subnet-routes=false`, `--accept-routes=false` (see lesson 1). |
|
||
| first client | nh3-dev (100.64.0.4) | `--accept-routes=false`, `--accept-dns=false`. Direct paths to all three routers (7–10 ms). |
|
||
| DNS | `*.internal` names for all four CTs synced to the three AdGuards | |
|
||
| backups | pfi-pve + nh3-pve jobs are `all`; esh-pve job vmid list extended with 108 | |
|
||
| vault | `headscale/preauth-router-48h-20260906`, `headscale/preauth-client-48h-20260906` (reusable, expire 2026-09-08) | mint fresh ones on the CT: `headscale preauthkeys create --user 1 --reusable --expiration 48h` |
|
||
|
||
**Lessons (do not repeat):**
|
||
|
||
1. ⚠ **`--accept-routes` on a client BEFORE the return path exists breaks that client's
|
||
LAN.** nh3-dev accepted all three /16s including its OWN site's; Linux table-52 routes
|
||
beat the main table, so every 10.100.x packet went into the tunnel to nh3-scale and
|
||
nothing came back (no SNAT → the far host has no route to 100.64/10). Took ssh to
|
||
nh3-pve, esh-pve and pfi-pve down for ~1 min until `tailscale set --accept-routes=false`.
|
||
Phase order is therefore: **return path first** (either SNAT on the routers, or static
|
||
routes for `100.64.0.0/10` → local router on every site gateway), *then* accept-routes on
|
||
clients, and exclude a node's own site route.
|
||
2. The routers currently find each other "direct" via LAN addresses (`10.0.50.65:41641`
|
||
etc.) — i.e. **the mesh is riding inside the old Site Magic / IPsec tunnels**. Public-
|
||
endpoint / relay path discovery is untested until those tunnels are disabled (phase 4).
|
||
Test that with one tunnel down before declaring cut-over ready.
|
||
3. pfi-pve (PVE 8.3.5) and nh3-pve (8.4.1) refuse the Debian 13 template
|
||
(`unsupported debian version '13.6'`); esh-pve (8.4.20) accepts it. Three CTs are
|
||
Debian 12, esh-scale is Debian 13. A failed `pct create` followed by an `>>` into the
|
||
conf leaves a stub conf that makes `pct` say "missing arch" — delete the stub. Follow-up:
|
||
bring pfi-pve/nh3-pve up to current 8.4.
|
||
4. headscale 0.29 removed `randomize_client_port` and renamed
|
||
`ephemeral_node_inactivity_timeout` → `node.ephemeral.inactivity_timeout`; with a bad
|
||
key `headscale serve` silently falls back to defaults (127.0.0.1:8080). Always
|
||
`headscale configtest` first.
|
||
|
||
**Next (in order):** (a) decide the return path — recommendation: static routes for
|
||
`100.64.0.0/10` on each site gateway → local router (keeps source IPs), SNAT as the fallback;
|
||
(b) enrol the operator's devices with the client key; (c) test a UDP-blocked path (relay over
|
||
443); (d) disable ONE old tunnel and re-test router paths; (e) cut over: static routes for the
|
||
other two /16s on each gateway, disable Site Magic + IPsec; (f) v6 AAAA once NH3 LAN v6 is
|
||
on; (g) self-hosted DERP at ESH; (h) PVE upgrades on pfi-pve/nh3-pve.
|