feat(mesh): headscale control plane + three subnet routers provisioned; DNS entries; plan status
This commit is contained in:
@@ -101,10 +101,15 @@ hosts:
|
||||
- {name: jellyfin, site: esh, v4: 10.0.50.57, note: media server}
|
||||
- {name: brother, site: esh, v4: 10.0.90.125, note: Brother printer}
|
||||
- {name: slzb-mr1u, site: esh, v4: 10.0.90.10, note: SMLIGHT SLZB-MR1U Zigbee/Thread coordinator (esh-iot VLAN 90, PoE on ESH-Media p4); Home Assistant at 10.0.50.46 connects to tcp/6638}
|
||||
- {name: ana-scale, site: ana, v4: 10.250.50.45, note: CT114 on pfi-pve — headscale/tailscale subnet router for 10.250.0.0/16}
|
||||
- {name: nh3-headscale, site: nh3, v4: 10.100.50.45, note: CT106 on nh3-pve — headscale control plane (public headscale.phasefinal.com:443)}
|
||||
- {name: nh3-scale, site: nh3, v4: 10.100.50.46, note: CT107 on nh3-pve — subnet router for 10.100.0.0/16}
|
||||
- {name: esh-scale, site: esh, v4: 10.0.50.65, note: CT108 on esh-pve — subnet router for 10.0.0.0/16}
|
||||
|
||||
# Service aliases — a name that points at whatever host currently runs it, so
|
||||
# consumers reference the SERVICE rather than the box. Changing where something
|
||||
# runs becomes a one-line edit here instead of a hunt through configs.
|
||||
|
||||
aliases:
|
||||
- {name: searxng, site: nh3, target: nh3-docker, note: moved off ana-docker 2026-09-03 — colo egress (38.120.12.42) is CAPTCHA-gated by search engines; NH3 egresses residentially}
|
||||
- {name: gateway, site: ana, target: ana-docker, note: LiteLLM gateway :4000}
|
||||
|
||||
@@ -112,3 +112,53 @@ above; 443 direct on the UDM vs behind the existing Caddy on nh3-dev; ACL postur
|
||||
phases 0–4.
|
||||
- VM 106 on pfi-pve was once named "PFI-Tailscale" (pre-2026 inventory). No config
|
||||
survives; nothing to reuse.
|
||||
|
||||
## Status — 2026-09-06 05:35Z: phases 0–1 done, phase 3 half-done, NOT cut over
|
||||
|
||||
Operator inputs (05:00Z): 443 direct on the UDM; names `nh3-headscale`, `nh3-scale`,
|
||||
`esh-scale`, `ana-scale`; ids my call → sequential. ACLs: flat (database policy mode,
|
||||
empty policy) until told otherwise.
|
||||
|
||||
**Live:**
|
||||
|
||||
| what | where | detail |
|
||||
|---|---|---|
|
||||
| control plane | CT 106 `nh3-headscale` 10.100.50.45 (nh3-pve, Debian 12, 1c/512M/8G) | headscale **v0.29.3** (.deb), `https://headscale.phasefinal.com` on **443 direct**, Let's Encrypt via TLS-ALPN-01 (no Caddy, no DNS-01), SQLite at `/var/lib/headscale/db.sqlite`, DERP = Tailscale public map, MagicDNS base `mesh.phasefinal.com`, split DNS `internal` → the three AdGuards, `override_local_dns: false`. Public health check 200 with a valid cert. |
|
||||
| public name | Cloudflare A `headscale.phasefinal.com` → NH3 WAN (70.230.226.88 at creation), unproxied, TTL 120 | **DDNS:** `~/.local/bin/headscale-ddns.sh` on nh3-dev, user timer `headscale-ddns.timer` every 10 min, token pulled from the vault at run time (nothing stored in the CT). No AAAA yet — NH3 LAN v6 is deliberately off. |
|
||||
| UDM forward | NH3 UDM SE `headscale-443`: WAN tcp/443 → 10.100.50.45:443 | added via the classic REST API. Hairpin from inside NH3 works. |
|
||||
| routers | CT 107 `nh3-scale` 10.100.50.46 (100.64.0.1) · CT 108 `esh-scale` 10.0.50.65 (100.64.0.2, Debian 13) · CT 114 `ana-scale` 10.250.50.45 (100.64.0.3) | tailscale 1.102.3, `/dev/net/tun` passed, forwarding on, each advertising its site /16, **routes approved and serving**, `--snat-subnet-routes=false`, `--accept-routes=false` (see lesson 1). |
|
||||
| first client | nh3-dev (100.64.0.4) | `--accept-routes=false`, `--accept-dns=false`. Direct paths to all three routers (7–10 ms). |
|
||||
| DNS | `*.internal` names for all four CTs synced to the three AdGuards | |
|
||||
| backups | pfi-pve + nh3-pve jobs are `all`; esh-pve job vmid list extended with 108 | |
|
||||
| vault | `headscale/preauth-router-48h-20260906`, `headscale/preauth-client-48h-20260906` (reusable, expire 2026-09-08) | mint fresh ones on the CT: `headscale preauthkeys create --user 1 --reusable --expiration 48h` |
|
||||
|
||||
**Lessons (do not repeat):**
|
||||
|
||||
1. ⚠ **`--accept-routes` on a client BEFORE the return path exists breaks that client's
|
||||
LAN.** nh3-dev accepted all three /16s including its OWN site's; Linux table-52 routes
|
||||
beat the main table, so every 10.100.x packet went into the tunnel to nh3-scale and
|
||||
nothing came back (no SNAT → the far host has no route to 100.64/10). Took ssh to
|
||||
nh3-pve, esh-pve and pfi-pve down for ~1 min until `tailscale set --accept-routes=false`.
|
||||
Phase order is therefore: **return path first** (either SNAT on the routers, or static
|
||||
routes for `100.64.0.0/10` → local router on every site gateway), *then* accept-routes on
|
||||
clients, and exclude a node's own site route.
|
||||
2. The routers currently find each other "direct" via LAN addresses (`10.0.50.65:41641`
|
||||
etc.) — i.e. **the mesh is riding inside the old Site Magic / IPsec tunnels**. Public-
|
||||
endpoint / relay path discovery is untested until those tunnels are disabled (phase 4).
|
||||
Test that with one tunnel down before declaring cut-over ready.
|
||||
3. pfi-pve (PVE 8.3.5) and nh3-pve (8.4.1) refuse the Debian 13 template
|
||||
(`unsupported debian version '13.6'`); esh-pve (8.4.20) accepts it. Three CTs are
|
||||
Debian 12, esh-scale is Debian 13. A failed `pct create` followed by an `>>` into the
|
||||
conf leaves a stub conf that makes `pct` say "missing arch" — delete the stub. Follow-up:
|
||||
bring pfi-pve/nh3-pve up to current 8.4.
|
||||
4. headscale 0.29 removed `randomize_client_port` and renamed
|
||||
`ephemeral_node_inactivity_timeout` → `node.ephemeral.inactivity_timeout`; with a bad
|
||||
key `headscale serve` silently falls back to defaults (127.0.0.1:8080). Always
|
||||
`headscale configtest` first.
|
||||
|
||||
**Next (in order):** (a) decide the return path — recommendation: static routes for
|
||||
`100.64.0.0/10` on each site gateway → local router (keeps source IPs), SNAT as the fallback;
|
||||
(b) enrol the operator's devices with the client key; (c) test a UDP-blocked path (relay over
|
||||
443); (d) disable ONE old tunnel and re-test router paths; (e) cut over: static routes for the
|
||||
other two /16s on each gateway, disable Site Magic + IPsec; (f) v6 AAAA once NH3 LAN v6 is
|
||||
on; (g) self-hosted DERP at ESH; (h) PVE upgrades on pfi-pve/nh3-pve.
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
# 2026-09-06 — Headscale overlay mesh: control plane + 3 subnet routers live, not cut over
|
||||
|
||||
Operator-directed (Headscale over NetBird; NH3 for the control plane, never the colo; 443
|
||||
direct; names nh3-headscale / nh3-scale / esh-scale / ana-scale). Full state, lessons and
|
||||
next steps in `docs/pfi/headscale-mesh-plan.md` § Status. Headline facts:
|
||||
|
||||
- `https://headscale.phasefinal.com` = CT 106 on nh3-pve (10.100.50.45), headscale v0.29.3,
|
||||
LE cert via TLS-ALPN-01, UDM forward tcp/443, DDNS timer on nh3-dev (user systemd).
|
||||
- Routers CT 107 nh3-scale / CT 108 esh-scale / CT 114 ana-scale advertise their /16s,
|
||||
approved, SNAT off, accept-routes OFF. nh3-dev enrolled as first client (100.64.0.4).
|
||||
- ⚠ Old tunnels (Site Magic, IPsec) are STILL the site-to-site path. The mesh currently
|
||||
rides inside them. Nothing has been disabled.
|
||||
- ⚠ Lesson: `--accept-routes` on a client before a return path for 100.64.0.0/10 exists
|
||||
black-holes that client's LAN (own-site /16 included). Return path first.
|
||||
- Pre-auth keys in the vault (`headscale/preauth-*-48h-20260906`, expire 09-08).
|
||||
- infra-ops user now exists on all four PVE hosts (needed `apt install sudo` first).
|
||||
@@ -473,6 +473,7 @@ below is a live commitment or a known-open risk._
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-06]` **Headscale overlay mesh: control plane live at `headscale.phasefinal.com` (CT 106 nh3-pve) + subnet routers nh3-scale/esh-scale/ana-scale serving their /16s; nh3-dev enrolled. NOT cut over — Site Magic + IPsec still carry site-to-site.** ⚠ accept-routes-before-return-path black-holed nh3-dev's LAN for a minute. infra-ops user added on all four PVE hosts. → `persistent-memory.d/2026-09-06-headscale-mesh-phase1.md`
|
||||
- `[2026-09-06]` **pfi-pve NASPool REBUILT as six-wide raidz2 after a backplane fault killed bays 9/10** (Route C hybrid, operator-directed): parked 1.65T on ospool, destroyed, recreated, restored, backup tier back 04:03Z; guests never stopped (ALL boot disks are on ospool — the prior brief had this wrong). Legacy vzdump pruned to newest-per-guest by omission. OPEN: destroy `ospool/naspool-evac` after scrub + one backup cycle; backplane swap next visit; PSU1 still dead. → `persistent-memory.d/2026-09-06-pfi-pve-naspool-raidz2-rebuild.md`
|
||||
- `[2026-09-05]` **A peer's "2.7x serving-stack effect" was a coin flip — the operator rejected it on instinct and the arithmetic backed him.** Each floor was `|b0-b1|` from n=2; the ratio is half-Cauchy, P=0.452. ⚠ The disconfirming evidence sat in brokkr's own sentence, and he named *why*: the claim was his and flattering. → `persistent-memory.d/2026-09-05-floor-claim-n2-retraction.md`
|
||||
- `[2026-09-05]` **vLLM RUNS on sm_121 — the blocker was `ninja` off PATH, not the silicon** — and run 4 launched after two peer artifacts were rejected by reading the harness rather than accepting a "confirm this". ⚠ I declined to fill in missing `root_sha256` values I knew, because supplying both sides of a check makes it inert. → `persistent-memory.d/2026-09-05-vllm-on-sm121-and-run4.md`
|
||||
|
||||
Reference in New Issue
Block a user