feat(mesh): irv-ml1 cutover complete (remote, 5-day-safe) — 3 recovery paths, wg0 dormant, dante retired

This commit is contained in:
vh
2026-09-06 00:58:21 -07:00
parent f8eb3606eb
commit be4e7d68c9
+32
View File
@@ -353,3 +353,35 @@ flip accept-routes on, confirm fleet reachability over the mesh, THEN down wg0.
⚠ **Lesson (repeat):** `pkill -f` / `pgrep -f <pattern>` over SSH self-matches when the pattern
is in your own ssh argv — killed my session twice disarming the switch. Kill by explicit PID,
or make the switch sentinel-based (which saved it here).
## 2026-09-06 — irv-ml1 CUTOVER COMPLETE + dante retired (done remotely, 5-day no-access)
Operator wanted the cutover finished tonight despite no physical Irvine access for 5 days, and
directed: dante proxy goes away (exit nodes instead), yt-voice-clipper may stay dead until
scoped exit-node egress is wired.
**Three independent recovery paths built BEFORE cutting wg0** (the discipline that made it safe):
1. Mesh over DERP-LA (~12ms) — irv-ml1's own WAN (38.120.94.3, easy NAT), independent of wg0.
2. **Reverse SSH tunnel** `revtun-nh3.service` on irv-ml1 → nh3-dev via NH3 public WAN
(UDM forward `irv-revtun-ssh` tcp/47822 → 10.100.10.50:22, **src-restricted to 38.120.94.3**;
key `/root/.ssh/revtun_ed25519`, authorized on nh3-dev lkraven with `restrict,port-forwarding`).
Reach irv-ml1: `ssh -i ~/.ssh/infra-ops_ed25519 -p 2201 infra-ops@127.0.0.1` on nh3-dev.
Independent of BOTH wg0 and the mesh/control-plane.
3. **`wg0-watchdog.service`** on irv-ml1 — `wg-quick up wg0` if the fleet is unreachable over the
mesh for ~5min (auto-failover). Plus wg-quick@wg0 stays enabled → reboot restores wg0.
**Cut:** accept-routes=true on irv-ml1 (fleet /16s → tailscale0 table 52), then `wg-quick down
wg0` over the reverse tunnel under a 12-min auto-revert dead-man (disarmed after verify).
Verified: irv-ml1 reaches nh3-docker/colo/ESH/nh3-pve over pure mesh; unreachable ONLY 10.100.10.50
(nh3-dev's own LAN IP — a mesh-node-reaching-a-mesh-node quirk; every other host fine; moot now
dante is gone). wg0 dormant (down, enabled). Recovery via reverse tunnel + mesh IP both OK.
**Dante retired** on nh3-dev: `danted` disabled+stopped, :1080 closed, `/etc/danted.conf` →
`.retired-20260906`. Only live consumer was YTVC (operator-exempted). searxng ref was already
commented out.
**OPEN follow-ups (on-site or later):** wire yt-voice-clipper egress via a SCOPED exit-node path
(do NOT set irv-ml1 `--exit-node` globally — it would route the reverse tunnel through the mesh and
kill the independent lifeline; use tailscale `--socks5-server`+exit-node or a per-container netns);
then bring YTVC back. On-site: decide whether to keep the reverse tunnel + UDM forward + revtun key
or remove them (small WAN attack surface, src-restricted). Fully retire wg0 config if desired.