vh
959a743256
Operator's design, and a better one. Once the FV SNAT rules landed, fv-ml1's mesh membership was redundant for routing and its only remaining value was as a second way in. Keeping it enrolled bought a standing second door; joining on demand buys the same recovery path without one. normal tailscaled stopped + disabled; fleet reached via the gateway SNAT fault nh3-dev / nh3-docker unreachable while the WAN is up action start tailscaled + tailscale up -> reachable at its 100.64.x address fv-ml1 is now off the mesh and its node record deleted. Verified it still reaches NH3, ESH, Anaheim, Irvine and the internet on the SNAT path alone, then the break-glass fired on cue (counted 1..4, joined at 5 as 100.64.0.10), answered ping and ssh from nh3-dev, and was closed again cleanly. No auto-leave, deliberately: once open the door stays open until a human runs systemctl disable --now tailscaled. A watchdog that re-closes on recovery flaps, and a flapping recovery path is down exactly when someone finally looks. It also skips entirely when already on the mesh, which is what makes it idempotent after firing. The question exposed a hole worth more than the redesign. The stored rejoin key was one of the 2026-09-12 FV cutover keys, expiring 2026-09-19 -- a break-glass credential that dies in four days and fails silently at the only moment it matters. Replaced with a dedicated 1-year reusable key (headscale ID 8, expires 2027-09-15), vaulted as fv-ml1/headscale-breakglass-key, root:600 on the host. That also closes the standing self-join risk rather than trading it: the two stale reusable keys (IDs 5, 6) are expired, so the mesh now has exactly one live reusable key -- purpose-built, on a host we control -- instead of two orphans nobody owned. Rejoin uses --accept-routes=false and the reason is in the script: on 2026-09-14 tailscale up --accept-routes on this box accepted its OWN subnet from the gateway and black-holed it. That happened with a human watching; here it runs unattended, during an incident, on a box already in trouble.
4.1 KiB
[2026-09-15] Break-glass mesh path on fv-ml1 (inverted from a restore-watchdog)
Every path into Fountain Valley runs through equipment at FV. When fv-ml1 loses its way back to the fleet there is no console, no local hands, and the BMC sits behind the same gateway. This is the net under the next routing change.
/usr/local/sbin/fv-mesh-watchdog.sh + fv-mesh-watchdog.{service,timer},
every 60 s. Canonical copies in servers/fv-ml1/. Commit 8c8559b.
Design choices that matter
- Two anchors that cannot share a failure mode — a plain-internet one
(
1.1.1.1) and a mesh-only one (100.64.0.1). If only the mesh anchor fails, the mesh is the problem and it acts. ⭐ If BOTH fail it deliberately does nothing — the site uplink is down, Tailscale cannot fix that, and thrashing tailscaled during an ISP outage turns a wait into an incident. - Threshold 5 consecutive failures, counter reset on recovery.
- Narrow remit: only
tailscale set --accept-routes=false+ re-upwith a stored key +systemctl restart tailscaled. It touches no routes, no firewall, no services — a watchdog with a wide remit is a second way to lose the box. - Disable file
/etc/fv-watchdog.disablefor planned work.
Proven, not assumed
Positive control against a black-holed anchor (MESH_ANCHOR=192.0.2.1 via the
conf file, real WAN anchor left in place so the uplink guard did not
short-circuit):
run1..run4 counted 1/5 .. 4/5, no action
run5 fired — tailscale up ran, tailscaled restarted, "restore attempt complete"
after counter reset to 0 once the real anchor returned
fv-ml1 stayed reachable throughout.
Why it exists
Earlier the same session, tailscale up --accept-routes on fv-ml1 black-holed
it from its own LAN: it accepted 10.251.0.0/16 from the gateway — its own
subnet — and routed the local network through the tunnel. Recovery only worked
because its mesh address happened to still answer. Same family as the
2026-09-06 nh3-dev incident; see 2026-09-15-fv-cross-site-snat.
⭐ INVERTED the same night, on the operator's suggestion
The first version kept fv-ml1 permanently on the mesh and restored its Tailscale state when it broke. Once the FV SNAT rules landed (2026-09-15-fv-cross-site-snat) that membership became redundant for routing — its only remaining value was as a second way in. The operator's question was the better design: keep the box OFF the mesh and have the watchdog JOIN when it loses the fleet. Same recovery path, no standing second door.
normal tailscaled stopped + disabled; fleet reached via the gateway SNAT
fault FLEET_ANCHORS (nh3-dev, nh3-docker) unreachable while the WAN is up
action start tailscaled + `tailscale up` -> reachable at its 100.64.x address
Verified end to end, off-mesh: fv-ml1 removed from headscale entirely, then
confirmed it still reaches NH3/ESH/ANA/IRV/internet on the SNAT path alone; then
the break-glass fired on cue (counted 1..4, joined at 5 as 100.64.0.10),
answered ping and ssh from nh3-dev, and was closed again cleanly.
⚠ No auto-leave, deliberately. Once open the door stays open until a human
runs systemctl disable --now tailscaled. A watchdog that re-closes on recovery
flaps, and a flapping recovery path is down exactly when someone finally looks.
⚠ Skipped if already on the mesh — that is what makes it idempotent after
firing, rather than re-running tailscale up every minute.
The hole the operator's question exposed
The stored rejoin key was hskey-auth-g8_iQtwSHntv, one of the 2026-09-12 FV
cutover keys — expiring 2026-09-19. A break-glass credential that dies in
four days and fails silently at the only moment it matters. Replaced with a
dedicated 1-year reusable key (headscale ID 8, expires 2027-09-15), vaulted
as fv-ml1/headscale-breakglass-key, stored root:600 at
/var/lib/fv-mesh-watchdog/authkey.
⭐ That also closes the standing self-join risk rather than trading it: the two stale reusable keys (IDs 5, 6) were expired, so the mesh now has exactly one live reusable key, purpose-built, on a host we control — instead of two orphans nobody owned.