# `[2026-09-15]` Break-glass mesh path on fv-ml1 (inverted from a restore-watchdog)

Every path into Fountain Valley runs through equipment at FV. When fv-ml1 loses
its way back to the fleet there is no console, no local hands, and the BMC sits
behind the same gateway. This is the net under the next routing change.

`/usr/local/sbin/fv-mesh-watchdog.sh` + `fv-mesh-watchdog.{service,timer}`,
every 60 s. Canonical copies in `servers/fv-ml1/`. Commit `8c8559b`.

## Design choices that matter

- **Two anchors that cannot share a failure mode** — a plain-internet one
  (`1.1.1.1`) and a mesh-only one (`100.64.0.1`). If only the mesh anchor fails,
  the mesh is the problem and it acts. ⭐ **If BOTH fail it deliberately does
  nothing** — the site uplink is down, Tailscale cannot fix that, and thrashing
  tailscaled during an ISP outage turns a wait into an incident.
- **Threshold 5 consecutive failures**, counter reset on recovery.
- **Narrow remit**: only `tailscale set --accept-routes=false` + re-`up` with a
  stored key + `systemctl restart tailscaled`. It touches no routes, no
  firewall, no services — a watchdog with a wide remit is a second way to lose
  the box.
- **Disable file** `/etc/fv-watchdog.disable` for planned work.

## Proven, not assumed

Positive control against a black-holed anchor (`MESH_ANCHOR=192.0.2.1` via the
conf file, real WAN anchor left in place so the uplink guard did not
short-circuit):

    run1..run4   counted 1/5 .. 4/5, no action
    run5         fired — tailscale up ran, tailscaled restarted, "restore attempt complete"
    after        counter reset to 0 once the real anchor returned

fv-ml1 stayed reachable throughout.

## Why it exists

Earlier the same session, `tailscale up --accept-routes` on fv-ml1 black-holed
it from its own LAN: it accepted `10.251.0.0/16` from the gateway — **its own
subnet** — and routed the local network through the tunnel. Recovery only worked
because its mesh address happened to still answer. Same family as the
2026-09-06 nh3-dev incident; see [[2026-09-15-fv-cross-site-snat]].

## ⭐ INVERTED the same night, on the operator's suggestion

The first version kept fv-ml1 permanently on the mesh and restored its Tailscale
state when it broke. Once the FV SNAT rules landed
([[2026-09-15-fv-cross-site-snat]]) that membership became **redundant for
routing** — its only remaining value was as a second way in. The operator's
question was the better design: *keep the box OFF the mesh and have the watchdog
JOIN when it loses the fleet.* Same recovery path, no standing second door.

    normal    tailscaled stopped + disabled; fleet reached via the gateway SNAT
    fault     FLEET_ANCHORS (nh3-dev, nh3-docker) unreachable while the WAN is up
    action    start tailscaled + `tailscale up` -> reachable at its 100.64.x address

**Verified end to end, off-mesh:** fv-ml1 removed from headscale entirely, then
confirmed it still reaches NH3/ESH/ANA/IRV/internet on the SNAT path alone; then
the break-glass fired on cue (counted 1..4, joined at 5 as `100.64.0.10`),
answered ping **and ssh** from nh3-dev, and was closed again cleanly.

⚠ **No auto-leave, deliberately.** Once open the door stays open until a human
runs `systemctl disable --now tailscaled`. A watchdog that re-closes on recovery
flaps, and a flapping recovery path is down exactly when someone finally looks.

⚠ **Skipped if already on the mesh** — that is what makes it idempotent after
firing, rather than re-running `tailscale up` every minute.

### The hole the operator's question exposed

The stored rejoin key was `hskey-auth-g8_iQtwSHntv`, one of the 2026-09-12 FV
cutover keys — **expiring 2026-09-19**. A break-glass credential that dies in
four days and fails silently at the only moment it matters. Replaced with a
dedicated **1-year reusable** key (headscale ID 8, expires 2027-09-15), vaulted
as `fv-ml1/headscale-breakglass-key`, stored `root:600` at
`/var/lib/fv-mesh-watchdog/authkey`.

⭐ That also **closes** the standing self-join risk rather than trading it: the
two stale reusable keys (IDs 5, 6) were expired, so the mesh now has exactly one
live reusable key, purpose-built, on a host we control — instead of two orphans
nobody owned.