From 3ef87165df61e8585c984b3bbd0c71459820f07a Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 5 Sep 2026 22:32:23 -0700 Subject: [PATCH] feat(mesh): headscale control plane + three subnet routers provisioned; DNS entries; plan status --- dns/internal.yaml | 5 ++ docs/pfi/headscale-mesh-plan.md | 50 +++++++++++++++++++ .../2026-09-06-headscale-mesh-phase1.md | 16 ++++++ persistent-memory.md | 1 + 4 files changed, 72 insertions(+) create mode 100644 persistent-memory.d/2026-09-06-headscale-mesh-phase1.md diff --git a/dns/internal.yaml b/dns/internal.yaml index 8accdcb..035f431 100644 --- a/dns/internal.yaml +++ b/dns/internal.yaml @@ -101,10 +101,15 @@ hosts: - {name: jellyfin, site: esh, v4: 10.0.50.57, note: media server} - {name: brother, site: esh, v4: 10.0.90.125, note: Brother printer} - {name: slzb-mr1u, site: esh, v4: 10.0.90.10, note: SMLIGHT SLZB-MR1U Zigbee/Thread coordinator (esh-iot VLAN 90, PoE on ESH-Media p4); Home Assistant at 10.0.50.46 connects to tcp/6638} + - {name: ana-scale, site: ana, v4: 10.250.50.45, note: CT114 on pfi-pve — headscale/tailscale subnet router for 10.250.0.0/16} + - {name: nh3-headscale, site: nh3, v4: 10.100.50.45, note: CT106 on nh3-pve — headscale control plane (public headscale.phasefinal.com:443)} + - {name: nh3-scale, site: nh3, v4: 10.100.50.46, note: CT107 on nh3-pve — subnet router for 10.100.0.0/16} + - {name: esh-scale, site: esh, v4: 10.0.50.65, note: CT108 on esh-pve — subnet router for 10.0.0.0/16} # Service aliases — a name that points at whatever host currently runs it, so # consumers reference the SERVICE rather than the box. Changing where something # runs becomes a one-line edit here instead of a hunt through configs. + aliases: - {name: searxng, site: nh3, target: nh3-docker, note: moved off ana-docker 2026-09-03 — colo egress (38.120.12.42) is CAPTCHA-gated by search engines; NH3 egresses residentially} - {name: gateway, site: ana, target: ana-docker, note: LiteLLM gateway :4000} diff --git a/docs/pfi/headscale-mesh-plan.md b/docs/pfi/headscale-mesh-plan.md index 06711f2..c815e30 100644 --- a/docs/pfi/headscale-mesh-plan.md +++ b/docs/pfi/headscale-mesh-plan.md @@ -112,3 +112,53 @@ above; 443 direct on the UDM vs behind the existing Caddy on nh3-dev; ACL postur phases 0–4. - VM 106 on pfi-pve was once named "PFI-Tailscale" (pre-2026 inventory). No config survives; nothing to reuse. + +## Status — 2026-09-06 05:35Z: phases 0–1 done, phase 3 half-done, NOT cut over + +Operator inputs (05:00Z): 443 direct on the UDM; names `nh3-headscale`, `nh3-scale`, +`esh-scale`, `ana-scale`; ids my call → sequential. ACLs: flat (database policy mode, +empty policy) until told otherwise. + +**Live:** + +| what | where | detail | +|---|---|---| +| control plane | CT 106 `nh3-headscale` 10.100.50.45 (nh3-pve, Debian 12, 1c/512M/8G) | headscale **v0.29.3** (.deb), `https://headscale.phasefinal.com` on **443 direct**, Let's Encrypt via TLS-ALPN-01 (no Caddy, no DNS-01), SQLite at `/var/lib/headscale/db.sqlite`, DERP = Tailscale public map, MagicDNS base `mesh.phasefinal.com`, split DNS `internal` → the three AdGuards, `override_local_dns: false`. Public health check 200 with a valid cert. | +| public name | Cloudflare A `headscale.phasefinal.com` → NH3 WAN (70.230.226.88 at creation), unproxied, TTL 120 | **DDNS:** `~/.local/bin/headscale-ddns.sh` on nh3-dev, user timer `headscale-ddns.timer` every 10 min, token pulled from the vault at run time (nothing stored in the CT). No AAAA yet — NH3 LAN v6 is deliberately off. | +| UDM forward | NH3 UDM SE `headscale-443`: WAN tcp/443 → 10.100.50.45:443 | added via the classic REST API. Hairpin from inside NH3 works. | +| routers | CT 107 `nh3-scale` 10.100.50.46 (100.64.0.1) · CT 108 `esh-scale` 10.0.50.65 (100.64.0.2, Debian 13) · CT 114 `ana-scale` 10.250.50.45 (100.64.0.3) | tailscale 1.102.3, `/dev/net/tun` passed, forwarding on, each advertising its site /16, **routes approved and serving**, `--snat-subnet-routes=false`, `--accept-routes=false` (see lesson 1). | +| first client | nh3-dev (100.64.0.4) | `--accept-routes=false`, `--accept-dns=false`. Direct paths to all three routers (7–10 ms). | +| DNS | `*.internal` names for all four CTs synced to the three AdGuards | | +| backups | pfi-pve + nh3-pve jobs are `all`; esh-pve job vmid list extended with 108 | | +| vault | `headscale/preauth-router-48h-20260906`, `headscale/preauth-client-48h-20260906` (reusable, expire 2026-09-08) | mint fresh ones on the CT: `headscale preauthkeys create --user 1 --reusable --expiration 48h` | + +**Lessons (do not repeat):** + +1. ⚠ **`--accept-routes` on a client BEFORE the return path exists breaks that client's + LAN.** nh3-dev accepted all three /16s including its OWN site's; Linux table-52 routes + beat the main table, so every 10.100.x packet went into the tunnel to nh3-scale and + nothing came back (no SNAT → the far host has no route to 100.64/10). Took ssh to + nh3-pve, esh-pve and pfi-pve down for ~1 min until `tailscale set --accept-routes=false`. + Phase order is therefore: **return path first** (either SNAT on the routers, or static + routes for `100.64.0.0/10` → local router on every site gateway), *then* accept-routes on + clients, and exclude a node's own site route. +2. The routers currently find each other "direct" via LAN addresses (`10.0.50.65:41641` + etc.) — i.e. **the mesh is riding inside the old Site Magic / IPsec tunnels**. Public- + endpoint / relay path discovery is untested until those tunnels are disabled (phase 4). + Test that with one tunnel down before declaring cut-over ready. +3. pfi-pve (PVE 8.3.5) and nh3-pve (8.4.1) refuse the Debian 13 template + (`unsupported debian version '13.6'`); esh-pve (8.4.20) accepts it. Three CTs are + Debian 12, esh-scale is Debian 13. A failed `pct create` followed by an `>>` into the + conf leaves a stub conf that makes `pct` say "missing arch" — delete the stub. Follow-up: + bring pfi-pve/nh3-pve up to current 8.4. +4. headscale 0.29 removed `randomize_client_port` and renamed + `ephemeral_node_inactivity_timeout` → `node.ephemeral.inactivity_timeout`; with a bad + key `headscale serve` silently falls back to defaults (127.0.0.1:8080). Always + `headscale configtest` first. + +**Next (in order):** (a) decide the return path — recommendation: static routes for +`100.64.0.0/10` on each site gateway → local router (keeps source IPs), SNAT as the fallback; +(b) enrol the operator's devices with the client key; (c) test a UDP-blocked path (relay over +443); (d) disable ONE old tunnel and re-test router paths; (e) cut over: static routes for the +other two /16s on each gateway, disable Site Magic + IPsec; (f) v6 AAAA once NH3 LAN v6 is +on; (g) self-hosted DERP at ESH; (h) PVE upgrades on pfi-pve/nh3-pve. diff --git a/persistent-memory.d/2026-09-06-headscale-mesh-phase1.md b/persistent-memory.d/2026-09-06-headscale-mesh-phase1.md new file mode 100644 index 0000000..4f025e1 --- /dev/null +++ b/persistent-memory.d/2026-09-06-headscale-mesh-phase1.md @@ -0,0 +1,16 @@ +# 2026-09-06 — Headscale overlay mesh: control plane + 3 subnet routers live, not cut over + +Operator-directed (Headscale over NetBird; NH3 for the control plane, never the colo; 443 +direct; names nh3-headscale / nh3-scale / esh-scale / ana-scale). Full state, lessons and +next steps in `docs/pfi/headscale-mesh-plan.md` § Status. Headline facts: + +- `https://headscale.phasefinal.com` = CT 106 on nh3-pve (10.100.50.45), headscale v0.29.3, + LE cert via TLS-ALPN-01, UDM forward tcp/443, DDNS timer on nh3-dev (user systemd). +- Routers CT 107 nh3-scale / CT 108 esh-scale / CT 114 ana-scale advertise their /16s, + approved, SNAT off, accept-routes OFF. nh3-dev enrolled as first client (100.64.0.4). +- ⚠ Old tunnels (Site Magic, IPsec) are STILL the site-to-site path. The mesh currently + rides inside them. Nothing has been disabled. +- ⚠ Lesson: `--accept-routes` on a client before a return path for 100.64.0.0/10 exists + black-holes that client's LAN (own-site /16 included). Return path first. +- Pre-auth keys in the vault (`headscale/preauth-*-48h-20260906`, expire 09-08). +- infra-ops user now exists on all four PVE hosts (needed `apt install sudo` first). diff --git a/persistent-memory.md b/persistent-memory.md index f75414e..d4ebba2 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -473,6 +473,7 @@ below is a live commitment or a known-open risk._ ## Recent decisions +- `[2026-09-06]` **Headscale overlay mesh: control plane live at `headscale.phasefinal.com` (CT 106 nh3-pve) + subnet routers nh3-scale/esh-scale/ana-scale serving their /16s; nh3-dev enrolled. NOT cut over — Site Magic + IPsec still carry site-to-site.** ⚠ accept-routes-before-return-path black-holed nh3-dev's LAN for a minute. infra-ops user added on all four PVE hosts. → `persistent-memory.d/2026-09-06-headscale-mesh-phase1.md` - `[2026-09-06]` **pfi-pve NASPool REBUILT as six-wide raidz2 after a backplane fault killed bays 9/10** (Route C hybrid, operator-directed): parked 1.65T on ospool, destroyed, recreated, restored, backup tier back 04:03Z; guests never stopped (ALL boot disks are on ospool — the prior brief had this wrong). Legacy vzdump pruned to newest-per-guest by omission. OPEN: destroy `ospool/naspool-evac` after scrub + one backup cycle; backplane swap next visit; PSU1 still dead. → `persistent-memory.d/2026-09-06-pfi-pve-naspool-raidz2-rebuild.md` - `[2026-09-05]` **A peer's "2.7x serving-stack effect" was a coin flip — the operator rejected it on instinct and the arithmetic backed him.** Each floor was `|b0-b1|` from n=2; the ratio is half-Cauchy, P=0.452. ⚠ The disconfirming evidence sat in brokkr's own sentence, and he named *why*: the claim was his and flattering. → `persistent-memory.d/2026-09-05-floor-claim-n2-retraction.md` - `[2026-09-05]` **vLLM RUNS on sm_121 — the blocker was `ninja` off PATH, not the silicon** — and run 4 launched after two peer artifacts were rejected by reading the harness rather than accepting a "confirm this". ⚠ I declined to fill in missing `root_sha256` values I knew, because supplying both sides of a check makes it inert. → `persistent-memory.d/2026-09-05-vllm-on-sm121-and-run4.md`