vh
959a743256
Operator's design, and a better one. Once the FV SNAT rules landed, fv-ml1's mesh membership was redundant for routing and its only remaining value was as a second way in. Keeping it enrolled bought a standing second door; joining on demand buys the same recovery path without one. normal tailscaled stopped + disabled; fleet reached via the gateway SNAT fault nh3-dev / nh3-docker unreachable while the WAN is up action start tailscaled + tailscale up -> reachable at its 100.64.x address fv-ml1 is now off the mesh and its node record deleted. Verified it still reaches NH3, ESH, Anaheim, Irvine and the internet on the SNAT path alone, then the break-glass fired on cue (counted 1..4, joined at 5 as 100.64.0.10), answered ping and ssh from nh3-dev, and was closed again cleanly. No auto-leave, deliberately: once open the door stays open until a human runs systemctl disable --now tailscaled. A watchdog that re-closes on recovery flaps, and a flapping recovery path is down exactly when someone finally looks. It also skips entirely when already on the mesh, which is what makes it idempotent after firing. The question exposed a hole worth more than the redesign. The stored rejoin key was one of the 2026-09-12 FV cutover keys, expiring 2026-09-19 -- a break-glass credential that dies in four days and fails silently at the only moment it matters. Replaced with a dedicated 1-year reusable key (headscale ID 8, expires 2027-09-15), vaulted as fv-ml1/headscale-breakglass-key, root:600 on the host. That also closes the standing self-join risk rather than trading it: the two stale reusable keys (IDs 5, 6) are expired, so the mesh now has exactly one live reusable key -- purpose-built, on a host we control -- instead of two orphans nobody owned. Rejoin uses --accept-routes=false and the reason is in the script: on 2026-09-14 tailscale up --accept-routes on this box accepted its OWN subnet from the gateway and black-holed it. That happened with a human watching; here it runs unattended, during an incident, on a box already in trouble.
82 lines
4.1 KiB
Markdown
82 lines
4.1 KiB
Markdown
# `[2026-09-15]` Break-glass mesh path on fv-ml1 (inverted from a restore-watchdog)
|
|
|
|
Every path into Fountain Valley runs through equipment at FV. When fv-ml1 loses
|
|
its way back to the fleet there is no console, no local hands, and the BMC sits
|
|
behind the same gateway. This is the net under the next routing change.
|
|
|
|
`/usr/local/sbin/fv-mesh-watchdog.sh` + `fv-mesh-watchdog.{service,timer}`,
|
|
every 60 s. Canonical copies in `servers/fv-ml1/`. Commit `8c8559b`.
|
|
|
|
## Design choices that matter
|
|
|
|
- **Two anchors that cannot share a failure mode** — a plain-internet one
|
|
(`1.1.1.1`) and a mesh-only one (`100.64.0.1`). If only the mesh anchor fails,
|
|
the mesh is the problem and it acts. ⭐ **If BOTH fail it deliberately does
|
|
nothing** — the site uplink is down, Tailscale cannot fix that, and thrashing
|
|
tailscaled during an ISP outage turns a wait into an incident.
|
|
- **Threshold 5 consecutive failures**, counter reset on recovery.
|
|
- **Narrow remit**: only `tailscale set --accept-routes=false` + re-`up` with a
|
|
stored key + `systemctl restart tailscaled`. It touches no routes, no
|
|
firewall, no services — a watchdog with a wide remit is a second way to lose
|
|
the box.
|
|
- **Disable file** `/etc/fv-watchdog.disable` for planned work.
|
|
|
|
## Proven, not assumed
|
|
|
|
Positive control against a black-holed anchor (`MESH_ANCHOR=192.0.2.1` via the
|
|
conf file, real WAN anchor left in place so the uplink guard did not
|
|
short-circuit):
|
|
|
|
run1..run4 counted 1/5 .. 4/5, no action
|
|
run5 fired — tailscale up ran, tailscaled restarted, "restore attempt complete"
|
|
after counter reset to 0 once the real anchor returned
|
|
|
|
fv-ml1 stayed reachable throughout.
|
|
|
|
## Why it exists
|
|
|
|
Earlier the same session, `tailscale up --accept-routes` on fv-ml1 black-holed
|
|
it from its own LAN: it accepted `10.251.0.0/16` from the gateway — **its own
|
|
subnet** — and routed the local network through the tunnel. Recovery only worked
|
|
because its mesh address happened to still answer. Same family as the
|
|
2026-09-06 nh3-dev incident; see [[2026-09-15-fv-cross-site-snat]].
|
|
|
|
## ⭐ INVERTED the same night, on the operator's suggestion
|
|
|
|
The first version kept fv-ml1 permanently on the mesh and restored its Tailscale
|
|
state when it broke. Once the FV SNAT rules landed
|
|
([[2026-09-15-fv-cross-site-snat]]) that membership became **redundant for
|
|
routing** — its only remaining value was as a second way in. The operator's
|
|
question was the better design: *keep the box OFF the mesh and have the watchdog
|
|
JOIN when it loses the fleet.* Same recovery path, no standing second door.
|
|
|
|
normal tailscaled stopped + disabled; fleet reached via the gateway SNAT
|
|
fault FLEET_ANCHORS (nh3-dev, nh3-docker) unreachable while the WAN is up
|
|
action start tailscaled + `tailscale up` -> reachable at its 100.64.x address
|
|
|
|
**Verified end to end, off-mesh:** fv-ml1 removed from headscale entirely, then
|
|
confirmed it still reaches NH3/ESH/ANA/IRV/internet on the SNAT path alone; then
|
|
the break-glass fired on cue (counted 1..4, joined at 5 as `100.64.0.10`),
|
|
answered ping **and ssh** from nh3-dev, and was closed again cleanly.
|
|
|
|
⚠ **No auto-leave, deliberately.** Once open the door stays open until a human
|
|
runs `systemctl disable --now tailscaled`. A watchdog that re-closes on recovery
|
|
flaps, and a flapping recovery path is down exactly when someone finally looks.
|
|
|
|
⚠ **Skipped if already on the mesh** — that is what makes it idempotent after
|
|
firing, rather than re-running `tailscale up` every minute.
|
|
|
|
### The hole the operator's question exposed
|
|
|
|
The stored rejoin key was `hskey-auth-g8_iQtwSHntv`, one of the 2026-09-12 FV
|
|
cutover keys — **expiring 2026-09-19**. A break-glass credential that dies in
|
|
four days and fails silently at the only moment it matters. Replaced with a
|
|
dedicated **1-year reusable** key (headscale ID 8, expires 2027-09-15), vaulted
|
|
as `fv-ml1/headscale-breakglass-key`, stored `root:600` at
|
|
`/var/lib/fv-mesh-watchdog/authkey`.
|
|
|
|
⭐ That also **closes** the standing self-join risk rather than trading it: the
|
|
two stale reusable keys (IDs 5, 6) were expired, so the mesh now has exactly one
|
|
live reusable key, purpose-built, on a host we control — instead of two orphans
|
|
nobody owned.
|