From df8440e4b87d31f91d2090a90e683a380a6fd07a Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 6 Sep 2026 00:02:15 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20headscale=20cutover=20=E2=80=94=20IPs?= =?UTF-8?q?ec=20dormant,=20mesh=20primary,=20Site=20Magic=20pending=20oper?= =?UTF-8?q?ator?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../2026-09-06-headscale-cutover.md | 22 +++++++++++++++++++ persistent-memory.md | 1 + 2 files changed, 23 insertions(+) create mode 100644 persistent-memory.d/2026-09-06-headscale-cutover.md diff --git a/persistent-memory.d/2026-09-06-headscale-cutover.md b/persistent-memory.d/2026-09-06-headscale-cutover.md new file mode 100644 index 0000000..421191b --- /dev/null +++ b/persistent-memory.d/2026-09-06-headscale-cutover.md @@ -0,0 +1,22 @@ +# 2026-09-06 — Headscale cutover: IPsec dormant, mesh primary; Site Magic pending operator + +Operator goal (/goal): replace Site Magic + IPsec with headscale, tunnels dormant as backup; +"if paranoid, enable world-accessible SSH on the FortiGate first." Full detail + method + +follow-ups in `docs/pfi/headscale-mesh-plan.md` § CUTOVER EXECUTED. Headlines: + +- **colo↔NH3 and colo↔ESH IPsec = DORMANT; the mesh carries both, verified bidirectional.** + NH3 UDM `pfi-nh3-ana` + ESH UDM `esh-ana` set enabled=false (API). Mesh /16 routes added on + both UDMs and the FortiGate (→ ana-scale 10.250.50.45 / nh3-scale 10.100.50.46 / esh-scale + 10.0.50.65). Dependent flows OK over mesh: restic ESH→rest-server-ana, FortiGate mgmt. +- **NH3↔ESH Site Magic NOT cut by API** — `sdwan-mesh-tunnel` = `api.err.NoEdit` (cloud + orchestrated). Routes PRE-STAGED + shadowed; DERP path 9ms ready. **Operator disables it in + the UniFi UI**, then the mesh takes over. Told the operator "mesh is online" → he does it. +- **FortiGate WAN SSH safety net (TEMPORARY):** wan1 allowaccess ping+ssh; admin infra-ops + trusthost2/3 = NH3 70.230.226.88 + ESH 23.164.40.160 (not 0.0.0.0). Reach it at + `ssh infra-ops@38.120.12.42`. Config backed up flash `pre-wan-ssh-cutover-20260906`. Remove + when the edge (being replaced by OPNsense/R420) is retired. +- ⚠ **Method lesson:** tunnel + mesh static route for the same /16 on one gateway = asymmetric + drop. Disable the tunnel FIRST, then add the route. Broke colo once doing it tunnel-up; rolled + back. See [[incident_crowdsec_cgnat_false_ban]] (same day) and the plan doc. +- Dormancy = disabled+retained (flip UDM object back to enabled=true to restore); NO auto + failover wired. Bonus: exit nodes → free multi-location egress proxy (parked). diff --git a/persistent-memory.md b/persistent-memory.md index d4ebba2..79d704f 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -473,6 +473,7 @@ below is a live commitment or a known-open risk._ ## Recent decisions +- `[2026-09-06]` **Headscale cutover: colo↔NH3 + colo↔ESH IPsec now DORMANT, the mesh carries both (verified bidirectional); NH3↔ESH Site Magic pre-staged but needs the operator's UniFi-UI disable (`api.err.NoEdit`).** FortiGate WAN-SSH safety net added (temp, scoped to NH3+ESH egress). Method: disable tunnel FIRST then add mesh route, or the two fight (asymmetric drop). → `persistent-memory.d/2026-09-06-headscale-cutover.md` - `[2026-09-06]` **Headscale overlay mesh: control plane live at `headscale.phasefinal.com` (CT 106 nh3-pve) + subnet routers nh3-scale/esh-scale/ana-scale serving their /16s; nh3-dev enrolled. NOT cut over — Site Magic + IPsec still carry site-to-site.** ⚠ accept-routes-before-return-path black-holed nh3-dev's LAN for a minute. infra-ops user added on all four PVE hosts. → `persistent-memory.d/2026-09-06-headscale-mesh-phase1.md` - `[2026-09-06]` **pfi-pve NASPool REBUILT as six-wide raidz2 after a backplane fault killed bays 9/10** (Route C hybrid, operator-directed): parked 1.65T on ospool, destroyed, recreated, restored, backup tier back 04:03Z; guests never stopped (ALL boot disks are on ospool — the prior brief had this wrong). Legacy vzdump pruned to newest-per-guest by omission. OPEN: destroy `ospool/naspool-evac` after scrub + one backup cycle; backplane swap next visit; PSU1 still dead. → `persistent-memory.d/2026-09-06-pfi-pve-naspool-raidz2-rebuild.md` - `[2026-09-05]` **A peer's "2.7x serving-stack effect" was a coin flip — the operator rejected it on instinct and the arithmetic backed him.** Each floor was `|b0-b1|` from n=2; the ratio is half-Cauchy, P=0.452. ⚠ The disconfirming evidence sat in brokkr's own sentence, and he named *why*: the claim was his and flattering. → `persistent-memory.d/2026-09-05-floor-claim-n2-retraction.md`