docs(pfi): refresh pfi-pve guest inventory from live state; headscale mesh plan
This commit is contained in:
@@ -0,0 +1,114 @@
|
||||
# Headscale overlay mesh — plan (2026-09-06)
|
||||
|
||||
**Status:** planning. Operator direction 2026-09-06: replace Site Magic (NH3↔ESH) and the
|
||||
FortiGate IPsec tunnels (colo↔NH3, colo↔ESH) with a self-hosted Tailscale-protocol overlay
|
||||
(Headscale), one ultra-light LXC per PVE host, keeping the old tunnels as an emergency
|
||||
backup. This document is the plan that a `.contract.md` will be cut from; nothing here is
|
||||
provisioned yet.
|
||||
|
||||
## Why Headscale (decided 2026-09-06)
|
||||
|
||||
Tailscale clients are the best in class on the devices actually used for remote access
|
||||
(iPad, Mac, Linux); the future OPNsense colo edge can be a Tailscale node natively
|
||||
(`os-tailscale`); Headscale is one binary + one SQLite file to self-host, versus NetBird's
|
||||
six-service stack with a mandatory IdP. NetBird's built-in UI and HA routing groups are
|
||||
real but not decisive for three sites. UDP-blocked networks (in-flight Wi-Fi) work because
|
||||
the client falls back to a relay over TCP 443; plain WireGuard (ana-wg) cannot.
|
||||
|
||||
## Site facts that drive placement
|
||||
|
||||
| site | edge | v4 | v6 | change risk |
|
||||
|---|---|---|---|---|
|
||||
| ANA colo | FortiGate 80F → **OPNsense on R420 within the month** | public static | pending | HIGH — edge in flux; incident history 2026 (breaker, PSU1, WAN admin closed) |
|
||||
| NH3 | UDM SE, stable | dynamic-but-stable (DDNS) | single /64, **reserved for meshing** | low |
|
||||
| ESH | UDM Pro Max | CGNAT now, **static soon**; 2G symmetric soon | /56 | low, improving |
|
||||
|
||||
**Control plane at NH3 now** (only stable edge; static v6 + DDNS v4 → one DNS name).
|
||||
**Reassess ESH** once its static v4 lands — it becomes the relay site regardless (2G
|
||||
symmetric). **Never the colo:** the mesh's brain must not live in the building it exists
|
||||
to reach in an emergency. Migration later = copy one SQLite file + move the DNS name;
|
||||
clients follow the URL and never re-enrol.
|
||||
|
||||
## Topology
|
||||
|
||||
```
|
||||
headscale.phasefinal.com (A via DDNS, AAAA static)
|
||||
│ HTTPS 443 (+ DERP later)
|
||||
┌─────────────────────────────┼───────────────────────────────┐
|
||||
│ NH3 nh3-pve │ │
|
||||
│ CT nh3-headscale ──control plane (1 core / 512M / 8G) │
|
||||
│ CT nh3-mesh-rtr ──subnet router, advertises 10.100.0.0/16│
|
||||
├──────────────────────────────────────────────────────────────┤
|
||||
│ ESH esh-pve │
|
||||
│ CT esh-mesh-rtr ──subnet router, advertises 10.0.0.0/16 │ (relay/DERP here later)
|
||||
├──────────────────────────────────────────────────────────────┤
|
||||
│ ANA pfi-pve │
|
||||
│ CT ana-mesh-rtr ──subnet router, advertises 10.250.0.0/16│ (OPNsense takes this over later)
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
Per-device clients: nh3-dev, laptops/iPad, corviduo-dev, ana-ml2 … (MagicDNS names)
|
||||
```
|
||||
|
||||
Four unprivileged Debian 13 LXCs, each 1 vCPU / 512 MB / 8 GB, `onboot=1`, backed up by
|
||||
the existing `pbs-ana` job. Routers need `/dev/net/tun` passed in and `ip_forward` (both
|
||||
routine for unprivileged CTs; net sysctls are namespaced).
|
||||
|
||||
Proposed ids/names (DHCP with reservation like every existing CT; recorded in
|
||||
`dns/internal.yaml` as `<name>.<site>.internal`):
|
||||
|
||||
| PVE | next id | CT | note |
|
||||
|---|---|---|---|
|
||||
| nh3-pve (`root@10.100.250.60`, PVE 8.4.1) | 106 | `nh3-headscale` | control plane |
|
||||
| nh3-pve | 107 | `nh3-mesh-rtr` | subnet router |
|
||||
| esh-pve (`root@10.0.250.35`, PVE 8.4.20) | 108 | `esh-mesh-rtr` | subnet router |
|
||||
| pfi-pve (`root@10.250.250.31`, PVE 8.3.5) | 114 | `ana-mesh-rtr` | subnet router; ana-wg (113) stays as independent WG fallback |
|
||||
|
||||
`debian-13-standard_13.6-1_amd64.tar.zst` is in `pveam available` on all three (not yet
|
||||
downloaded on any).
|
||||
|
||||
## Access & credentials — all in hand (verified 2026-09-06)
|
||||
|
||||
| need | have |
|
||||
|---|---|
|
||||
| provision LXCs on all three PVEs | `ssh root@` works on pfi-pve, nh3-pve, esh-pve (`infra-ops@` is refused on all three PVE hosts) |
|
||||
| public DNS name + DDNS | Cloudflare all-zones DNS-edit token, vault `nh3-dev/.config/cloudflare/infra-ops-dns-token` |
|
||||
| NH3 UDM port-forward 443 → nh3-headscale; static routes on both UDMs | UDM API keys, vault `unifi/pfi-udmse-api-key`, `unifi/esh-udmpm-api-key` (classic `/rest/*` read+write) |
|
||||
| colo static route toward ana-mesh-rtr | FortiGate infra-ops SSH pw vaulted; reachable at 10.250.0.1 via the tunnel (`execute backup config` first). Moot once OPNsense lands |
|
||||
| TLS for headscale | Let's Encrypt via Cloudflare DNS-01 (same token) — no inbound 80 needed |
|
||||
| secrets (pre-auth keys, API key, DB) | vault under `nh3-headscale/…` |
|
||||
|
||||
Nothing outstanding on credentials. Inputs still needed from the operator: confirm names/ids
|
||||
above; 443 direct on the UDM vs behind the existing Caddy on nh3-dev; ACL posture (flat
|
||||
"everything can reach everything" first, tighten later, is the recommendation).
|
||||
|
||||
## Phases
|
||||
|
||||
0. **Pre-flight (no changes):** confirm NH3 v6 /64 address for the AAAA; confirm DHCP
|
||||
reservation ranges on the three sites; `pveam download` Debian 13 on all three.
|
||||
1. **Control plane:** CT `nh3-headscale`; headscale in a container or the .deb (prefer .deb —
|
||||
fewer layers in a 512M CT); Caddy/own TLS via DNS-01; UDM forward 443; DDNS updater
|
||||
(UDM → Cloudflare, or a `ddclient`/cron in the CT); DERP = Tailscale public map
|
||||
initially; headplane UI optional. Backup: `/var/lib/headscale/db.sqlite` is the only
|
||||
state — PBS covers the CT; add a nightly `sqlite3 .backup` to `backupStore` too.
|
||||
2. **First nodes:** nh3-dev + the operator's laptop/iPad enrol → prove MagicDNS, prove
|
||||
TCP-443 relay path from a UDP-blocked network.
|
||||
3. **Subnet routers:** three router CTs, `--advertise-routes` per site, approve routes in
|
||||
headscale, SNAT off (`--snat-subnet-routes=false`) so source IPs survive, static routes
|
||||
on each site gateway pointing the other two /16s at the local router. Test
|
||||
site-to-site from clientless hosts.
|
||||
4. **Cut over:** move day-to-day traffic onto the mesh; **disable (not delete)** Site Magic
|
||||
and the two IPsec tunnels. Running all three at once makes route precedence murky on
|
||||
the UDM and produces asymmetric paths. ana-wg stays as the out-of-band WG fallback.
|
||||
5. **Later:** relay (DERP) at ESH when the 2G circuit is in; OPNsense as the colo router
|
||||
node; HA router pairs (Tailscale HA subnet routers) per site; tighten ACLs.
|
||||
|
||||
## Risks / open questions
|
||||
|
||||
- Control-plane outage does not drop existing tunnels but stalls new logins and key
|
||||
rotation — hence the SQLite backup and the "not at the colo" rule.
|
||||
- Tailscale public DERPs carry relayed (encrypted) traffic through third-party infra until
|
||||
a self-hosted DERP exists. Acceptable at first; fix in phase 5.
|
||||
- CGNAT at ESH: outbound-only is fine; nothing at ESH needs to be dialled inbound.
|
||||
- Headscale lags Tailscale feature-wise (tailnet lock, some ACL syntax). Not relevant to
|
||||
phases 0–4.
|
||||
- VM 106 on pfi-pve was once named "PFI-Tailscale" (pre-2026 inventory). No config
|
||||
survives; nothing to reuse.
|
||||
Reference in New Issue
Block a user