From be4e7d68c91e0245bcc7449d30ef133d97a9b4f2 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 6 Sep 2026 00:58:21 -0700 Subject: [PATCH] =?UTF-8?q?feat(mesh):=20irv-ml1=20cutover=20complete=20(r?= =?UTF-8?q?emote,=205-day-safe)=20=E2=80=94=203=20recovery=20paths,=20wg0?= =?UTF-8?q?=20dormant,=20dante=20retired?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/pfi/headscale-mesh-plan.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/docs/pfi/headscale-mesh-plan.md b/docs/pfi/headscale-mesh-plan.md index 5ca2317..618386b 100644 --- a/docs/pfi/headscale-mesh-plan.md +++ b/docs/pfi/headscale-mesh-plan.md @@ -353,3 +353,35 @@ flip accept-routes on, confirm fleet reachability over the mesh, THEN down wg0. ⚠ **Lesson (repeat):** `pkill -f` / `pgrep -f ` over SSH self-matches when the pattern is in your own ssh argv — killed my session twice disarming the switch. Kill by explicit PID, or make the switch sentinel-based (which saved it here). + +## 2026-09-06 — irv-ml1 CUTOVER COMPLETE + dante retired (done remotely, 5-day no-access) + +Operator wanted the cutover finished tonight despite no physical Irvine access for 5 days, and +directed: dante proxy goes away (exit nodes instead), yt-voice-clipper may stay dead until +scoped exit-node egress is wired. + +**Three independent recovery paths built BEFORE cutting wg0** (the discipline that made it safe): +1. Mesh over DERP-LA (~12ms) — irv-ml1's own WAN (38.120.94.3, easy NAT), independent of wg0. +2. **Reverse SSH tunnel** `revtun-nh3.service` on irv-ml1 → nh3-dev via NH3 public WAN + (UDM forward `irv-revtun-ssh` tcp/47822 → 10.100.10.50:22, **src-restricted to 38.120.94.3**; + key `/root/.ssh/revtun_ed25519`, authorized on nh3-dev lkraven with `restrict,port-forwarding`). + Reach irv-ml1: `ssh -i ~/.ssh/infra-ops_ed25519 -p 2201 infra-ops@127.0.0.1` on nh3-dev. + Independent of BOTH wg0 and the mesh/control-plane. +3. **`wg0-watchdog.service`** on irv-ml1 — `wg-quick up wg0` if the fleet is unreachable over the + mesh for ~5min (auto-failover). Plus wg-quick@wg0 stays enabled → reboot restores wg0. + +**Cut:** accept-routes=true on irv-ml1 (fleet /16s → tailscale0 table 52), then `wg-quick down +wg0` over the reverse tunnel under a 12-min auto-revert dead-man (disarmed after verify). +Verified: irv-ml1 reaches nh3-docker/colo/ESH/nh3-pve over pure mesh; unreachable ONLY 10.100.10.50 +(nh3-dev's own LAN IP — a mesh-node-reaching-a-mesh-node quirk; every other host fine; moot now +dante is gone). wg0 dormant (down, enabled). Recovery via reverse tunnel + mesh IP both OK. + +**Dante retired** on nh3-dev: `danted` disabled+stopped, :1080 closed, `/etc/danted.conf` → +`.retired-20260906`. Only live consumer was YTVC (operator-exempted). searxng ref was already +commented out. + +**OPEN follow-ups (on-site or later):** wire yt-voice-clipper egress via a SCOPED exit-node path +(do NOT set irv-ml1 `--exit-node` globally — it would route the reverse tunnel through the mesh and +kill the independent lifeline; use tailscale `--socks5-server`+exit-node or a per-container netns); +then bring YTVC back. On-site: decide whether to keep the reverse tunnel + UDM forward + revtun key +or remove them (small WAN attack surface, src-restricted). Fully retire wg0 config if desired.