feat(mesh): headscale control plane + three subnet routers provisioned; DNS entries; plan status

2026-09-05 22:32:23 -07:00
parent e0759e41a3
commit 3ef87165df
4 changed files with 72 additions and 0 deletions
@@ -112,3 +112,53 @@ above; 443 direct on the UDM vs behind the existing Caddy on nh3-dev; ACL postur
  phases 0–4.
 - VM 106 on pfi-pve was once named "PFI-Tailscale" (pre-2026 inventory). No config
  survives; nothing to reuse.
+
+## Status — 2026-09-06 05:35Z: phases 0–1 done, phase 3 half-done, NOT cut over
+
+Operator inputs (05:00Z): 443 direct on the UDM; names `nh3-headscale`, `nh3-scale`,
+`esh-scale`, `ana-scale`; ids my call → sequential. ACLs: flat (database policy mode,
+empty policy) until told otherwise.
+
+**Live:**
+
+| what | where | detail |
+|---|---|---|
+| control plane | CT 106 `nh3-headscale` 10.100.50.45 (nh3-pve, Debian 12, 1c/512M/8G) | headscale **v0.29.3** (.deb), `https://headscale.phasefinal.com` on **443 direct**, Let's Encrypt via TLS-ALPN-01 (no Caddy, no DNS-01), SQLite at `/var/lib/headscale/db.sqlite`, DERP = Tailscale public map, MagicDNS base `mesh.phasefinal.com`, split DNS `internal` → the three AdGuards, `override_local_dns: false`. Public health check 200 with a valid cert. |
+| public name | Cloudflare A `headscale.phasefinal.com` → NH3 WAN (70.230.226.88 at creation), unproxied, TTL 120 | **DDNS:** `~/.local/bin/headscale-ddns.sh` on nh3-dev, user timer `headscale-ddns.timer` every 10 min, token pulled from the vault at run time (nothing stored in the CT). No AAAA yet — NH3 LAN v6 is deliberately off. |
+| UDM forward | NH3 UDM SE `headscale-443`: WAN tcp/443 → 10.100.50.45:443 | added via the classic REST API. Hairpin from inside NH3 works. |
+| routers | CT 107 `nh3-scale` 10.100.50.46 (100.64.0.1) · CT 108 `esh-scale` 10.0.50.65 (100.64.0.2, Debian 13) · CT 114 `ana-scale` 10.250.50.45 (100.64.0.3) | tailscale 1.102.3, `/dev/net/tun` passed, forwarding on, each advertising its site /16, **routes approved and serving**, `--snat-subnet-routes=false`, `--accept-routes=false` (see lesson 1). |
+| first client | nh3-dev (100.64.0.4) | `--accept-routes=false`, `--accept-dns=false`. Direct paths to all three routers (7–10 ms). |
+| DNS | `*.internal` names for all four CTs synced to the three AdGuards | |
+| backups | pfi-pve + nh3-pve jobs are `all`; esh-pve job vmid list extended with 108 | |
+| vault | `headscale/preauth-router-48h-20260906`, `headscale/preauth-client-48h-20260906` (reusable, expire 2026-09-08) | mint fresh ones on the CT: `headscale preauthkeys create --user 1 --reusable --expiration 48h` |
+
+**Lessons (do not repeat):**
+
+1. ⚠ **`--accept-routes` on a client BEFORE the return path exists breaks that client's
+   LAN.** nh3-dev accepted all three /16s including its OWN site's; Linux table-52 routes
+   beat the main table, so every 10.100.x packet went into the tunnel to nh3-scale and
+   nothing came back (no SNAT → the far host has no route to 100.64/10). Took ssh to
+   nh3-pve, esh-pve and pfi-pve down for ~1 min until `tailscale set --accept-routes=false`.
+   Phase order is therefore: **return path first** (either SNAT on the routers, or static
+   routes for `100.64.0.0/10` → local router on every site gateway), *then* accept-routes on
+   clients, and exclude a node's own site route.
+2. The routers currently find each other "direct" via LAN addresses (`10.0.50.65:41641`
+   etc.) — i.e. **the mesh is riding inside the old Site Magic / IPsec tunnels**. Public-
+   endpoint / relay path discovery is untested until those tunnels are disabled (phase 4).
+   Test that with one tunnel down before declaring cut-over ready.
+3. pfi-pve (PVE 8.3.5) and nh3-pve (8.4.1) refuse the Debian 13 template
+   (`unsupported debian version '13.6'`); esh-pve (8.4.20) accepts it. Three CTs are
+   Debian 12, esh-scale is Debian 13. A failed `pct create` followed by an `>>` into the
+   conf leaves a stub conf that makes `pct` say "missing arch" — delete the stub. Follow-up:
+   bring pfi-pve/nh3-pve up to current 8.4.
+4. headscale 0.29 removed `randomize_client_port` and renamed
+   `ephemeral_node_inactivity_timeout` → `node.ephemeral.inactivity_timeout`; with a bad
+   key `headscale serve` silently falls back to defaults (127.0.0.1:8080). Always
+   `headscale configtest` first.
+
+**Next (in order):** (a) decide the return path — recommendation: static routes for
+`100.64.0.0/10` on each site gateway → local router (keeps source IPs), SNAT as the fallback;
+(b) enrol the operator's devices with the client key; (c) test a UDP-blocked path (relay over
+443); (d) disable ONE old tunnel and re-test router paths; (e) cut over: static routes for the
+other two /16s on each gateway, disable Site Magic + IPsec; (f) v6 AAAA once NH3 LAN v6 is
+on; (g) self-hosted DERP at ESH; (h) PVE upgrades on pfi-pve/nh3-pve.