feat(mesh): irv-ml1 cutover complete (remote, 5-day-safe) — 3 recovery paths, wg0 dormant, dante retired
This commit is contained in:
@@ -353,3 +353,35 @@ flip accept-routes on, confirm fleet reachability over the mesh, THEN down wg0.
|
||||
⚠ **Lesson (repeat):** `pkill -f` / `pgrep -f <pattern>` over SSH self-matches when the pattern
|
||||
is in your own ssh argv — killed my session twice disarming the switch. Kill by explicit PID,
|
||||
or make the switch sentinel-based (which saved it here).
|
||||
|
||||
## 2026-09-06 — irv-ml1 CUTOVER COMPLETE + dante retired (done remotely, 5-day no-access)
|
||||
|
||||
Operator wanted the cutover finished tonight despite no physical Irvine access for 5 days, and
|
||||
directed: dante proxy goes away (exit nodes instead), yt-voice-clipper may stay dead until
|
||||
scoped exit-node egress is wired.
|
||||
|
||||
**Three independent recovery paths built BEFORE cutting wg0** (the discipline that made it safe):
|
||||
1. Mesh over DERP-LA (~12ms) — irv-ml1's own WAN (38.120.94.3, easy NAT), independent of wg0.
|
||||
2. **Reverse SSH tunnel** `revtun-nh3.service` on irv-ml1 → nh3-dev via NH3 public WAN
|
||||
(UDM forward `irv-revtun-ssh` tcp/47822 → 10.100.10.50:22, **src-restricted to 38.120.94.3**;
|
||||
key `/root/.ssh/revtun_ed25519`, authorized on nh3-dev lkraven with `restrict,port-forwarding`).
|
||||
Reach irv-ml1: `ssh -i ~/.ssh/infra-ops_ed25519 -p 2201 infra-ops@127.0.0.1` on nh3-dev.
|
||||
Independent of BOTH wg0 and the mesh/control-plane.
|
||||
3. **`wg0-watchdog.service`** on irv-ml1 — `wg-quick up wg0` if the fleet is unreachable over the
|
||||
mesh for ~5min (auto-failover). Plus wg-quick@wg0 stays enabled → reboot restores wg0.
|
||||
|
||||
**Cut:** accept-routes=true on irv-ml1 (fleet /16s → tailscale0 table 52), then `wg-quick down
|
||||
wg0` over the reverse tunnel under a 12-min auto-revert dead-man (disarmed after verify).
|
||||
Verified: irv-ml1 reaches nh3-docker/colo/ESH/nh3-pve over pure mesh; unreachable ONLY 10.100.10.50
|
||||
(nh3-dev's own LAN IP — a mesh-node-reaching-a-mesh-node quirk; every other host fine; moot now
|
||||
dante is gone). wg0 dormant (down, enabled). Recovery via reverse tunnel + mesh IP both OK.
|
||||
|
||||
**Dante retired** on nh3-dev: `danted` disabled+stopped, :1080 closed, `/etc/danted.conf` →
|
||||
`.retired-20260906`. Only live consumer was YTVC (operator-exempted). searxng ref was already
|
||||
commented out.
|
||||
|
||||
**OPEN follow-ups (on-site or later):** wire yt-voice-clipper egress via a SCOPED exit-node path
|
||||
(do NOT set irv-ml1 `--exit-node` globally — it would route the reverse tunnel through the mesh and
|
||||
kill the independent lifeline; use tailscale `--socks5-server`+exit-node or a per-container netns);
|
||||
then bring YTVC back. On-site: decide whether to keep the reverse tunnel + UDM forward + revtun key
|
||||
or remove them (small WAN attack surface, src-restricted). Fully retire wg0 config if desired.
|
||||
|
||||
Reference in New Issue
Block a user