115 lines
7.4 KiB
Markdown
115 lines
7.4 KiB
Markdown
# Headscale overlay mesh — plan (2026-09-06)
|
||
|
||
**Status:** planning. Operator direction 2026-09-06: replace Site Magic (NH3↔ESH) and the
|
||
FortiGate IPsec tunnels (colo↔NH3, colo↔ESH) with a self-hosted Tailscale-protocol overlay
|
||
(Headscale), one ultra-light LXC per PVE host, keeping the old tunnels as an emergency
|
||
backup. This document is the plan that a `.contract.md` will be cut from; nothing here is
|
||
provisioned yet.
|
||
|
||
## Why Headscale (decided 2026-09-06)
|
||
|
||
Tailscale clients are the best in class on the devices actually used for remote access
|
||
(iPad, Mac, Linux); the future OPNsense colo edge can be a Tailscale node natively
|
||
(`os-tailscale`); Headscale is one binary + one SQLite file to self-host, versus NetBird's
|
||
six-service stack with a mandatory IdP. NetBird's built-in UI and HA routing groups are
|
||
real but not decisive for three sites. UDP-blocked networks (in-flight Wi-Fi) work because
|
||
the client falls back to a relay over TCP 443; plain WireGuard (ana-wg) cannot.
|
||
|
||
## Site facts that drive placement
|
||
|
||
| site | edge | v4 | v6 | change risk |
|
||
|---|---|---|---|---|
|
||
| ANA colo | FortiGate 80F → **OPNsense on R420 within the month** | public static | pending | HIGH — edge in flux; incident history 2026 (breaker, PSU1, WAN admin closed) |
|
||
| NH3 | UDM SE, stable | dynamic-but-stable (DDNS) | single /64, **reserved for meshing** | low |
|
||
| ESH | UDM Pro Max | CGNAT now, **static soon**; 2G symmetric soon | /56 | low, improving |
|
||
|
||
**Control plane at NH3 now** (only stable edge; static v6 + DDNS v4 → one DNS name).
|
||
**Reassess ESH** once its static v4 lands — it becomes the relay site regardless (2G
|
||
symmetric). **Never the colo:** the mesh's brain must not live in the building it exists
|
||
to reach in an emergency. Migration later = copy one SQLite file + move the DNS name;
|
||
clients follow the URL and never re-enrol.
|
||
|
||
## Topology
|
||
|
||
```
|
||
headscale.phasefinal.com (A via DDNS, AAAA static)
|
||
│ HTTPS 443 (+ DERP later)
|
||
┌─────────────────────────────┼───────────────────────────────┐
|
||
│ NH3 nh3-pve │ │
|
||
│ CT nh3-headscale ──control plane (1 core / 512M / 8G) │
|
||
│ CT nh3-mesh-rtr ──subnet router, advertises 10.100.0.0/16│
|
||
├──────────────────────────────────────────────────────────────┤
|
||
│ ESH esh-pve │
|
||
│ CT esh-mesh-rtr ──subnet router, advertises 10.0.0.0/16 │ (relay/DERP here later)
|
||
├──────────────────────────────────────────────────────────────┤
|
||
│ ANA pfi-pve │
|
||
│ CT ana-mesh-rtr ──subnet router, advertises 10.250.0.0/16│ (OPNsense takes this over later)
|
||
└──────────────────────────────────────────────────────────────┘
|
||
Per-device clients: nh3-dev, laptops/iPad, corviduo-dev, ana-ml2 … (MagicDNS names)
|
||
```
|
||
|
||
Four unprivileged Debian 13 LXCs, each 1 vCPU / 512 MB / 8 GB, `onboot=1`, backed up by
|
||
the existing `pbs-ana` job. Routers need `/dev/net/tun` passed in and `ip_forward` (both
|
||
routine for unprivileged CTs; net sysctls are namespaced).
|
||
|
||
Proposed ids/names (DHCP with reservation like every existing CT; recorded in
|
||
`dns/internal.yaml` as `<name>.<site>.internal`):
|
||
|
||
| PVE | next id | CT | note |
|
||
|---|---|---|---|
|
||
| nh3-pve (`root@10.100.250.60`, PVE 8.4.1) | 106 | `nh3-headscale` | control plane |
|
||
| nh3-pve | 107 | `nh3-mesh-rtr` | subnet router |
|
||
| esh-pve (`root@10.0.250.35`, PVE 8.4.20) | 108 | `esh-mesh-rtr` | subnet router |
|
||
| pfi-pve (`root@10.250.250.31`, PVE 8.3.5) | 114 | `ana-mesh-rtr` | subnet router; ana-wg (113) stays as independent WG fallback |
|
||
|
||
`debian-13-standard_13.6-1_amd64.tar.zst` is in `pveam available` on all three (not yet
|
||
downloaded on any).
|
||
|
||
## Access & credentials — all in hand (verified 2026-09-06)
|
||
|
||
| need | have |
|
||
|---|---|
|
||
| provision LXCs on all three PVEs | `ssh root@` works on pfi-pve, nh3-pve, esh-pve (`infra-ops@` is refused on all three PVE hosts) |
|
||
| public DNS name + DDNS | Cloudflare all-zones DNS-edit token, vault `nh3-dev/.config/cloudflare/infra-ops-dns-token` |
|
||
| NH3 UDM port-forward 443 → nh3-headscale; static routes on both UDMs | UDM API keys, vault `unifi/pfi-udmse-api-key`, `unifi/esh-udmpm-api-key` (classic `/rest/*` read+write) |
|
||
| colo static route toward ana-mesh-rtr | FortiGate infra-ops SSH pw vaulted; reachable at 10.250.0.1 via the tunnel (`execute backup config` first). Moot once OPNsense lands |
|
||
| TLS for headscale | Let's Encrypt via Cloudflare DNS-01 (same token) — no inbound 80 needed |
|
||
| secrets (pre-auth keys, API key, DB) | vault under `nh3-headscale/…` |
|
||
|
||
Nothing outstanding on credentials. Inputs still needed from the operator: confirm names/ids
|
||
above; 443 direct on the UDM vs behind the existing Caddy on nh3-dev; ACL posture (flat
|
||
"everything can reach everything" first, tighten later, is the recommendation).
|
||
|
||
## Phases
|
||
|
||
0. **Pre-flight (no changes):** confirm NH3 v6 /64 address for the AAAA; confirm DHCP
|
||
reservation ranges on the three sites; `pveam download` Debian 13 on all three.
|
||
1. **Control plane:** CT `nh3-headscale`; headscale in a container or the .deb (prefer .deb —
|
||
fewer layers in a 512M CT); Caddy/own TLS via DNS-01; UDM forward 443; DDNS updater
|
||
(UDM → Cloudflare, or a `ddclient`/cron in the CT); DERP = Tailscale public map
|
||
initially; headplane UI optional. Backup: `/var/lib/headscale/db.sqlite` is the only
|
||
state — PBS covers the CT; add a nightly `sqlite3 .backup` to `backupStore` too.
|
||
2. **First nodes:** nh3-dev + the operator's laptop/iPad enrol → prove MagicDNS, prove
|
||
TCP-443 relay path from a UDP-blocked network.
|
||
3. **Subnet routers:** three router CTs, `--advertise-routes` per site, approve routes in
|
||
headscale, SNAT off (`--snat-subnet-routes=false`) so source IPs survive, static routes
|
||
on each site gateway pointing the other two /16s at the local router. Test
|
||
site-to-site from clientless hosts.
|
||
4. **Cut over:** move day-to-day traffic onto the mesh; **disable (not delete)** Site Magic
|
||
and the two IPsec tunnels. Running all three at once makes route precedence murky on
|
||
the UDM and produces asymmetric paths. ana-wg stays as the out-of-band WG fallback.
|
||
5. **Later:** relay (DERP) at ESH when the 2G circuit is in; OPNsense as the colo router
|
||
node; HA router pairs (Tailscale HA subnet routers) per site; tighten ACLs.
|
||
|
||
## Risks / open questions
|
||
|
||
- Control-plane outage does not drop existing tunnels but stalls new logins and key
|
||
rotation — hence the SQLite backup and the "not at the colo" rule.
|
||
- Tailscale public DERPs carry relayed (encrypted) traffic through third-party infra until
|
||
a self-hosted DERP exists. Acceptable at first; fix in phase 5.
|
||
- CGNAT at ESH: outbound-only is fine; nothing at ESH needs to be dialled inbound.
|
||
- Headscale lags Tailscale feature-wise (tailnet lock, some ACL syntax). Not relevant to
|
||
phases 0–4.
|
||
- VM 106 on pfi-pve was once named "PFI-Tailscale" (pre-2026 inventory). No config
|
||
survives; nothing to reuse.
|