feat(fv): invert the watchdog to break-glass; fv-ml1 off the mesh

Operator's design, and a better one. Once the FV SNAT rules landed, fv-ml1's mesh membership was redundant for routing and its only remaining value was as a second way in. Keeping it enrolled bought a standing second door; joining on demand buys the same recovery path without one. normal tailscaled stopped + disabled; fleet reached via the gateway SNAT fault nh3-dev / nh3-docker unreachable while the WAN is up action start tailscaled + tailscale up -> reachable at its 100.64.x address fv-ml1 is now off the mesh and its node record deleted. Verified it still reaches NH3, ESH, Anaheim, Irvine and the internet on the SNAT path alone, then the break-glass fired on cue (counted 1..4, joined at 5 as 100.64.0.10), answered ping and ssh from nh3-dev, and was closed again cleanly. No auto-leave, deliberately: once open the door stays open until a human runs systemctl disable --now tailscaled. A watchdog that re-closes on recovery flaps, and a flapping recovery path is down exactly when someone finally looks. It also skips entirely when already on the mesh, which is what makes it idempotent after firing. The question exposed a hole worth more than the redesign. The stored rejoin key was one of the 2026-09-12 FV cutover keys, expiring 2026-09-19 -- a break-glass credential that dies in four days and fails silently at the only moment it matters. Replaced with a dedicated 1-year reusable key (headscale ID 8, expires 2027-09-15), vaulted as fv-ml1/headscale-breakglass-key, root:600 on the host. That also closes the standing self-join risk rather than trading it: the two stale reusable keys (IDs 5, 6) are expired, so the mesh now has exactly one live reusable key -- purpose-built, on a host we control -- instead of two orphans nobody owned. Rejoin uses --accept-routes=false and the reason is in the script: on 2026-09-14 tailscale up --accept-routes on this box accepted its OWN subnet from the gateway and black-holed it. That happened with a human watching; here it runs unattended, during an incident, on a box already in trouble.
2026-09-15 01:02:16 -07:00
parent 838132cd6b
commit 959a743256
3 changed files with 104 additions and 45 deletions
@@ -1,4 +1,4 @@
-# `[2026-09-15]` Dead man's switch on fv-ml1 — mesh watchdog
+# `[2026-09-15]` Break-glass mesh path on fv-ml1 (inverted from a restore-watchdog)

 Every path into Fountain Valley runs through equipment at FV. When fv-ml1 loses
 its way back to the fleet there is no console, no local hands, and the BMC sits
@@ -40,3 +40,42 @@ it from its own LAN: it accepted `10.251.0.0/16` from the gateway — **its own
 subnet** — and routed the local network through the tunnel. Recovery only worked
 because its mesh address happened to still answer. Same family as the
 2026-09-06 nh3-dev incident; see [[2026-09-15-fv-cross-site-snat]].
+
+## ⭐ INVERTED the same night, on the operator's suggestion
+
+The first version kept fv-ml1 permanently on the mesh and restored its Tailscale
+state when it broke. Once the FV SNAT rules landed
+([[2026-09-15-fv-cross-site-snat]]) that membership became **redundant for
+routing** — its only remaining value was as a second way in. The operator's
+question was the better design: *keep the box OFF the mesh and have the watchdog
+JOIN when it loses the fleet.* Same recovery path, no standing second door.
+
+    normal    tailscaled stopped + disabled; fleet reached via the gateway SNAT
+    fault     FLEET_ANCHORS (nh3-dev, nh3-docker) unreachable while the WAN is up
+    action    start tailscaled + `tailscale up` -> reachable at its 100.64.x address
+
+**Verified end to end, off-mesh:** fv-ml1 removed from headscale entirely, then
+confirmed it still reaches NH3/ESH/ANA/IRV/internet on the SNAT path alone; then
+the break-glass fired on cue (counted 1..4, joined at 5 as `100.64.0.10`),
+answered ping **and ssh** from nh3-dev, and was closed again cleanly.
+
+⚠ **No auto-leave, deliberately.** Once open the door stays open until a human
+runs `systemctl disable --now tailscaled`. A watchdog that re-closes on recovery
+flaps, and a flapping recovery path is down exactly when someone finally looks.
+
+⚠ **Skipped if already on the mesh** — that is what makes it idempotent after
+firing, rather than re-running `tailscale up` every minute.
+
+### The hole the operator's question exposed
+
+The stored rejoin key was `hskey-auth-g8_iQtwSHntv`, one of the 2026-09-12 FV
+cutover keys — **expiring 2026-09-19**. A break-glass credential that dies in
+four days and fails silently at the only moment it matters. Replaced with a
+dedicated **1-year reusable** key (headscale ID 8, expires 2027-09-15), vaulted
+as `fv-ml1/headscale-breakglass-key`, stored `root:600` at
+`/var/lib/fv-mesh-watchdog/authkey`.
+
+⭐ That also **closes** the standing self-join risk rather than trading it: the
+two stale reusable keys (IDs 5, 6) were expired, so the mesh now has exactly one
+live reusable key, purpose-built, on a host we control — instead of two orphans
+nobody owned.