feat(fv): invert the watchdog to break-glass; fv-ml1 off the mesh

Operator's design, and a better one. Once the FV SNAT rules landed, fv-ml1's
mesh membership was redundant for routing and its only remaining value was as a
second way in. Keeping it enrolled bought a standing second door; joining on
demand buys the same recovery path without one.

  normal    tailscaled stopped + disabled; fleet reached via the gateway SNAT
  fault     nh3-dev / nh3-docker unreachable while the WAN is up
  action    start tailscaled + tailscale up -> reachable at its 100.64.x address

fv-ml1 is now off the mesh and its node record deleted. Verified it still
reaches NH3, ESH, Anaheim, Irvine and the internet on the SNAT path alone, then
the break-glass fired on cue (counted 1..4, joined at 5 as 100.64.0.10),
answered ping and ssh from nh3-dev, and was closed again cleanly.

No auto-leave, deliberately: once open the door stays open until a human runs
systemctl disable --now tailscaled. A watchdog that re-closes on recovery flaps,
and a flapping recovery path is down exactly when someone finally looks. It also
skips entirely when already on the mesh, which is what makes it idempotent after
firing.

The question exposed a hole worth more than the redesign. The stored rejoin key
was one of the 2026-09-12 FV cutover keys, expiring 2026-09-19 -- a break-glass
credential that dies in four days and fails silently at the only moment it
matters. Replaced with a dedicated 1-year reusable key (headscale ID 8, expires
2027-09-15), vaulted as fv-ml1/headscale-breakglass-key, root:600 on the host.

That also closes the standing self-join risk rather than trading it: the two
stale reusable keys (IDs 5, 6) are expired, so the mesh now has exactly one live
reusable key -- purpose-built, on a host we control -- instead of two orphans
nobody owned.

Rejoin uses --accept-routes=false and the reason is in the script: on 2026-09-14
tailscale up --accept-routes on this box accepted its OWN subnet from the
gateway and black-holed it. That happened with a human watching; here it runs
unattended, during an incident, on a box already in trouble.

persistent-memory.d/2026-09-15-fv-mesh-watchdog.md

@@ -1,4 +1,4 @@
-# `[2026-09-15]` Dead man's switch on fv-ml1 — mesh watchdog
+# `[2026-09-15]` Break-glass mesh path on fv-ml1 (inverted from a restore-watchdog)
 
 Every path into Fountain Valley runs through equipment at FV. When fv-ml1 loses
 its way back to the fleet there is no console, no local hands, and the BMC sits
@@ -40,3 +40,42 @@ it from its own LAN: it accepted `10.251.0.0/16` from the gateway — **its own
 subnet** — and routed the local network through the tunnel. Recovery only worked
 because its mesh address happened to still answer. Same family as the
 2026-09-06 nh3-dev incident; see [[2026-09-15-fv-cross-site-snat]].
+
+## ⭐ INVERTED the same night, on the operator's suggestion
+
+The first version kept fv-ml1 permanently on the mesh and restored its Tailscale
+state when it broke. Once the FV SNAT rules landed
+([[2026-09-15-fv-cross-site-snat]]) that membership became **redundant for
+routing** — its only remaining value was as a second way in. The operator's
+question was the better design: *keep the box OFF the mesh and have the watchdog
+JOIN when it loses the fleet.* Same recovery path, no standing second door.
+
+    normal    tailscaled stopped + disabled; fleet reached via the gateway SNAT
+    fault     FLEET_ANCHORS (nh3-dev, nh3-docker) unreachable while the WAN is up
+    action    start tailscaled + `tailscale up` -> reachable at its 100.64.x address
+
+**Verified end to end, off-mesh:** fv-ml1 removed from headscale entirely, then
+confirmed it still reaches NH3/ESH/ANA/IRV/internet on the SNAT path alone; then
+the break-glass fired on cue (counted 1..4, joined at 5 as `100.64.0.10`),
+answered ping **and ssh** from nh3-dev, and was closed again cleanly.
+
+⚠ **No auto-leave, deliberately.** Once open the door stays open until a human
+runs `systemctl disable --now tailscaled`. A watchdog that re-closes on recovery
+flaps, and a flapping recovery path is down exactly when someone finally looks.
+
+⚠ **Skipped if already on the mesh** — that is what makes it idempotent after
+firing, rather than re-running `tailscale up` every minute.
+
+### The hole the operator's question exposed
+
+The stored rejoin key was `hskey-auth-g8_iQtwSHntv`, one of the 2026-09-12 FV
+cutover keys — **expiring 2026-09-19**. A break-glass credential that dies in
+four days and fails silently at the only moment it matters. Replaced with a
+dedicated **1-year reusable** key (headscale ID 8, expires 2027-09-15), vaulted
+as `fv-ml1/headscale-breakglass-key`, stored `root:600` at
+`/var/lib/fv-mesh-watchdog/authkey`.
+
+⭐ That also **closes** the standing self-join risk rather than trading it: the
+two stale reusable keys (IDs 5, 6) were expired, so the mesh now has exactly one
+live reusable key, purpose-built, on a host we control — instead of two orphans
+nobody owned.

persistent-memory.md

@@ -127,9 +127,9 @@ on the FV gateway, all `src=10.251.50.0/24`, dst `10.100.0.0/16` / `10.0.0.0/16`
 Anaheim only. Commits `fa04f45`, `0ab9da5`.
 
 ### Safety net at FV
-- **Dead man's switch on fv-ml1** — `fv-mesh-watchdog` timer, 1/min, two independent
+- **Break-glass mesh path on fv-ml1** — the box is now OFF the mesh; `fv-mesh-watchdog`
-  anchors, restores known-good Tailscale after 5 consecutive mesh-only failures, no-ops
+  (1/min) JOINS it when nh3-dev/nh3-docker go unreachable while the WAN is up. Verified
-  when the WAN anchor is also down. Proven by positive control. `8c8559b`.
+  end to end off-mesh: joined as `100.64.0.10`, answered ping+ssh, closed cleanly.
 - **FV firewall config in the nightly restic run** via ana-docker's pre-backup hook
   (`80d982d`); `infra-ops` now has an SSH key + `/bin/sh` on the gateway, so the
   allowlisted WAN path gives a shell rather than API-only.
@@ -142,12 +142,12 @@ from the mesh at its LAN address (`9dbd829`); ESPHome pinned/relocated/rotated
 (`d1769ed` ff); `kb` KB-search tool shipped (`68fa80f`).
 
 ### Open loose ends (none blocking)
-- **nh3-dev is still a bare mesh member.** Operator authorized retiring both it and
+- **nh3-dev is still a bare mesh member** — retirement authorized, never executed.
-  fv-ml1 from the mesh; fv-ml1 turned out **load-bearing** (removing it cut FV off from
+  **Operator's call.** (fv-ml1 IS now retired: off-mesh, node record deleted, reaching the
-  NH3/ESH) and was restored as `100.64.0.9`. nh3-dev was never touched. Now that the
+  fleet on the gateway's SNAT alone, with break-glass rejoin as its safety net.)
-  SNAT fix exists the original rationale may hold again — **unexecuted, operator's call.**
+- ✅ **Stale reusable pre-auth keys retired** (IDs 5, 6 expired). The mesh now has exactly
-- **Two reusable headscale pre-auth keys** (IDs 5, 6) live until **2026-09-19** — the one
+  one live reusable key: the dedicated fv-ml1 break-glass key, ID 8, expires 2027-09-15,
-  credential shape that lets an unknown device self-join. Vaulted, unused, FV cutover done.
+  vaulted `fv-ml1/headscale-breakglass-key`.
 - **FV WAN admin is plain HTTP**, no TLS. Allowlisted to ESH/ANA/NH3 egress — ⚠ pinned to
   **egress IPs**, so a WAN change at any site silently removes the fallback.
 - **Legacy `/32` ANA NAT rule** redundant and invisible to `source_nat/search_rule`;
@@ -161,7 +161,7 @@ from the mesh at its LAN address (`9dbd829`); ESPHome pinned/relocated/rotated
 
 - `[2026-09-15]` **FV cross-site routing fixed — one OPNsense outbound-NAT rule had been scoped to Anaheim only.** fv-ml1 now reaches NH3/ESH/IRV/ANA/mesh/internet; four rules, all `src=10.251.50.0/24`. The diagnostic signature is the valuable part: every layer looks correct and the discriminator is that *every other site pair works*. → `persistent-memory.d/2026-09-15-fv-cross-site-snat.md`
 
-- `[2026-09-15]` **Dead man's switch on fv-ml1** — mesh watchdog, two independent anchors, deliberately no-ops when the uplink itself is down. Proven by positive control, not assumed. → `persistent-memory.d/2026-09-15-fv-mesh-watchdog.md`
+- `[2026-09-15]` **Break-glass mesh path on fv-ml1** — inverted from a restore-watchdog on the operator's suggestion: the box is OFF the mesh and the watchdog JOINS it on fleet loss. Exposed a rejoin key expiring in 4 days; replaced with a dedicated 1-year key and the two stale reusable keys retired. → `persistent-memory.d/2026-09-15-fv-mesh-watchdog.md`
 
 - `[2026-09-15]` **Fleet identity/group/path conventions pinned + docker trees → `root:docker 2775` setgid on 5 hosts.** `svc-*` in 800-849, infra-ops 850, docker 851, `vh` for new hosts with no retro-renames; `0777` cleared; `linus` deleted; `llmuser` de-privileged. → `persistent-memory.d/2026-09-15-fleet-identity-conventions.md`
 

servers/fv-ml1/fv-mesh-watchdog.sh

@@ -1,26 +1,38 @@
 #!/bin/sh
-# fv-mesh-watchdog — restore fv-ml1's mesh connectivity if a change strands it.
+# fv-mesh-watchdog — BREAK-GLASS: join the mesh when fv-ml1 loses the fleet.
 #
-# WHY: every path into the Fountain Valley site runs through equipment at FV.
+# ⭐ INVERTED DESIGN (operator, 2026-09-15). fv-ml1 is normally OFF the mesh: its
-# When fv-ml1 loses its way back to the fleet there is no console, no local
+# cross-site traffic is carried by outbound SNAT on the FV OPNsense gateway
-# hands, and the BMC sits behind the same gateway. On 2026-09-14 a
+# (docs/runbooks/fv-to-ana-nat.md), so mesh membership is redundant for routing
-# `tailscale up --accept-routes` black-holed this box from its own LAN: it
+# and its only value is as a SECOND way in when the first one breaks. Keeping it
-# accepted 10.251.0.0/16 from the gateway — its OWN subnet — and started routing
+# permanently enrolled bought a standing second door; joining ON DEMAND buys the
-# the local network through the tunnel. Recovery needed its mesh address, which
+# same recovery path without one.
-# only happened to still work.
 #
-# WHAT IT DOES: probes two INDEPENDENT anchors once a minute. After
+# This is therefore not a "restore Tailscale" watchdog. It is a door that opens
-# FAIL_THRESHOLD consecutive failures it puts Tailscale back to known-good:
+# when the normal path is gone.
-# accept-routes off, re-up against headscale. It touches nothing else — not
-# routes, not firewall, not services — because a watchdog with a wide remit is
-# a second way to lose the box.
 #
-# ⚠ Two anchors on purpose, and they must not share a failure mode:
+#   normal    tailscaled stopped+disabled; fleet reached via the gateway's SNAT
-#   WAN_ANCHOR   plain internet, no mesh involved
+#   fault     FLEET_ANCHORS unreachable while the WAN is up
-#   MESH_ANCHOR  a fleet address only reachable if the mesh works
+#   action    start tailscaled and `tailscale up` -> the box is reachable at its
-# If only the mesh anchor fails, the mesh is the problem and we act. If BOTH
+#             mesh address even though its LAN path is broken
-# fail the site's uplink is down and Tailscale cannot fix that, so we do NOT
+#
-# act — thrashing tailscaled during an ISP outage turns a wait into an incident.
+# ⚠ TWO ANCHOR CLASSES, and they must not share a failure mode:
+#   WAN_ANCHOR    plain internet. If THIS is down the site uplink is gone,
+#                 Tailscale cannot fix it, and we do nothing -- thrashing
+#                 during an ISP outage turns a wait into an incident.
+#   FLEET_ANCHORS hosts reachable ONLY through the gateway's SNAT path. Their
+#                 loss is exactly the fault this exists for.
+#
+# ⚠⚠ THE REJOIN MUST USE --accept-routes=false. On 2026-09-14 a
+# `tailscale up --accept-routes` on this box accepted 10.251.0.0/16 from the
+# gateway -- its OWN subnet -- and routed the local network through the tunnel,
+# black-holing it. That happened with a human watching. Here it would run
+# unattended, during an incident, on a box already in trouble.
+#
+# ⚠ NO AUTO-LEAVE. Once the door is open it stays open until a human closes it
+# (`systemctl disable --now tailscaled`). A watchdog that re-closes on recovery
+# flaps, and a flapping recovery path is worse than none -- it would be down
+# exactly when someone finally looked.
 #
 # Disable for planned work:  touch /etc/fv-watchdog.disable
 
@@ -30,7 +42,7 @@ CONF=/etc/fv-mesh-watchdog.conf
 DISABLE=/etc/fv-watchdog.disable
 FAIL_THRESHOLD=5
 WAN_ANCHOR=1.1.1.1
-MESH_ANCHOR=100.64.0.1
+FLEET_ANCHORS="10.100.10.50 10.100.50.40"    # nh3-dev, nh3-docker
 LOGIN_SERVER=https://headscale.phasefinal.com
 [ -r "$CONF" ] && . "$CONF"
 
@@ -41,34 +53,42 @@ mkdir -p "$STATE"; COUNT_FILE="$STATE/consecutive_failures"
 [ -f "$COUNT_FILE" ] || echo 0 > "$COUNT_FILE"
 count=$(cat "$COUNT_FILE" 2>/dev/null || echo 0)
 
-ping -c 2 -W 3 "$WAN_ANCHOR"  >/dev/null 2>&1 && wan=up   || wan=down
+# Already on the mesh? Then the door is open and there is nothing to do. This is
-ping -c 2 -W 3 "$MESH_ANCHOR" >/dev/null 2>&1 && mesh=up  || mesh=down
+# also what makes the script idempotent after it has fired.
+if [ "$(systemctl is-active tailscaled 2>/dev/null)" = active ] \
+   && tailscale status >/dev/null 2>&1; then
+  [ "$count" -ne 0 ] && echo 0 > "$COUNT_FILE"
+  exit 0
+fi
 
+ping -c 2 -W 3 "$WAN_ANCHOR" >/dev/null 2>&1 && wan=up || wan=down
 if [ "$wan" = down ]; then
-  # Uplink is down. Tailscale cannot fix that and restarting it during an ISP
+  [ "$count" -ne 0 ] && log "uplink down — not a fleet-path fault, counter reset"
-  # outage only adds churn. Reset the counter so the mesh timer starts clean
-  # once the site is back.
-  [ "$count" -ne 0 ] && log "uplink down (wan=$wan mesh=$mesh) — not a mesh fault, counter reset"
   echo 0 > "$COUNT_FILE"; exit 0
 fi
 
-if [ "$mesh" = up ]; then
+# Any fleet anchor answering means the normal path works.
-  [ "$count" -ne 0 ] && log "mesh recovered after $count failure(s)"
+fleet=down
+for a in $FLEET_ANCHORS; do
+  if ping -c 2 -W 3 "$a" >/dev/null 2>&1; then fleet=up; break; fi
+done
+
+if [ "$fleet" = up ]; then
+  [ "$count" -ne 0 ] && log "fleet path recovered after $count failure(s)"
   echo 0 > "$COUNT_FILE"; exit 0
 fi
 
 count=$((count + 1)); echo "$count" > "$COUNT_FILE"
-log "mesh anchor $MESH_ANCHOR unreachable while WAN is up ($count/$FAIL_THRESHOLD)"
+log "fleet unreachable via the gateway path while WAN is up ($count/$FAIL_THRESHOLD) — anchors: $FLEET_ANCHORS"
 [ "$count" -lt "$FAIL_THRESHOLD" ] && exit 0
 
-log "THRESHOLD REACHED — restoring known-good Tailscale state"
+log "THRESHOLD REACHED — opening the break-glass mesh path"
 echo 0 > "$COUNT_FILE"
-tailscale set --accept-routes=false 2>&1 | while read -r l; do log "  set: $l"; done
+systemctl start tailscaled 2>&1 | while read -r l; do log "  start: $l"; done
 if [ -r "$STATE/authkey" ]; then
   tailscale up --login-server "$LOGIN_SERVER" --authkey "$(cat "$STATE/authkey")" \
       --accept-routes=false --hostname "$(hostname -s)" 2>&1 | while read -r l; do log "  up: $l"; done
+  log "mesh path OPEN — reach this host at its 100.64.x address; close with 'systemctl disable --now tailscaled'"
 else
-  log "  no $STATE/authkey — cannot re-register; tried accept-routes=false only"
+  log "  NO $STATE/authkey — cannot join. The break-glass path is UNARMED."
 fi
-systemctl restart tailscaled 2>&1 | while read -r l; do log "  restart: $l"; done
-log "restore attempt complete"