vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(fv): invert the watchdog to break-glass; fv-ml1 off the mesh

Browse Source 
Operator's design, and a better one. Once the FV SNAT rules landed, fv-ml1's
mesh membership was redundant for routing and its only remaining value was as a
second way in. Keeping it enrolled bought a standing second door; joining on
demand buys the same recovery path without one.

  normal    tailscaled stopped + disabled; fleet reached via the gateway SNAT
  fault     nh3-dev / nh3-docker unreachable while the WAN is up
  action    start tailscaled + tailscale up -> reachable at its 100.64.x address

fv-ml1 is now off the mesh and its node record deleted. Verified it still
reaches NH3, ESH, Anaheim, Irvine and the internet on the SNAT path alone, then
the break-glass fired on cue (counted 1..4, joined at 5 as 100.64.0.10),
answered ping and ssh from nh3-dev, and was closed again cleanly.

No auto-leave, deliberately: once open the door stays open until a human runs
systemctl disable --now tailscaled. A watchdog that re-closes on recovery flaps,
and a flapping recovery path is down exactly when someone finally looks. It also
skips entirely when already on the mesh, which is what makes it idempotent after
firing.

The question exposed a hole worth more than the redesign. The stored rejoin key
was one of the 2026-09-12 FV cutover keys, expiring 2026-09-19 -- a break-glass
credential that dies in four days and fails silently at the only moment it
matters. Replaced with a dedicated 1-year reusable key (headscale ID 8, expires
2027-09-15), vaulted as fv-ml1/headscale-breakglass-key, root:600 on the host.

That also closes the standing self-join risk rather than trading it: the two
stale reusable keys (IDs 5, 6) are expired, so the mesh now has exactly one live
reusable key -- purpose-built, on a host we control -- instead of two orphans
nobody owned.

Rejoin uses --accept-routes=false and the reason is in the script: on 2026-09-14
tailscale up --accept-routes on this box accepted its OWN subnet from the
gateway and black-holed it. That happened with a human watching; here it runs
unattended, during an incident, on a box already in trouble.
This commit is contained in:
vh
2026-09-15 01:02:16 -07:00
parent 838132cd6b
commit 959a743256
3 changed files with 104 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
servers/fv-ml1/fv-mesh-watchdog.sh
+54 -34
View File
@@ -1,26 +1,38 @@
#!/bin/sh
# fv-mesh-watchdog — restore fv-ml1's mesh connectivity if a change strands it.
# fv-mesh-watchdog — BREAK-GLASS: join the mesh when fv-ml1 loses the fleet.
#
# WHY: every path into the Fountain Valley site runs through equipment at FV.
# When fv-ml1 loses its way back to the fleet there is no console, no local
# hands, and the BMC sits behind the same gateway. On 2026-09-14 a
# `tailscale up --accept-routes` black-holed this box from its own LAN: it
# accepted 10.251.0.0/16 from the gateway — its OWN subnet — and started routing
# the local network through the tunnel. Recovery needed its mesh address, which
# only happened to still work.
# ⭐ INVERTED DESIGN (operator, 2026-09-15). fv-ml1 is normally OFF the mesh: its
# cross-site traffic is carried by outbound SNAT on the FV OPNsense gateway
# (docs/runbooks/fv-to-ana-nat.md), so mesh membership is redundant for routing
# and its only value is as a SECOND way in when the first one breaks. Keeping it
# permanently enrolled bought a standing second door; joining ON DEMAND buys the
# same recovery path without one.
#
# WHAT IT DOES: probes two INDEPENDENT anchors once a minute. After
# FAIL_THRESHOLD consecutive failures it puts Tailscale back to known-good:
# accept-routes off, re-up against headscale. It touches nothing else — not
# routes, not firewall, not services — because a watchdog with a wide remit is
# a second way to lose the box.
# This is therefore not a "restore Tailscale" watchdog. It is a door that opens
# when the normal path is gone.
#
# ⚠ Two anchors on purpose, and they must not share a failure mode:
# WAN_ANCHOR plain internet, no mesh involved
# MESH_ANCHOR a fleet address only reachable if the mesh works
# If only the mesh anchor fails, the mesh is the problem and we act. If BOTH
# fail the site's uplink is down and Tailscale cannot fix that, so we do NOT
# act — thrashing tailscaled during an ISP outage turns a wait into an incident.
# normal tailscaled stopped+disabled; fleet reached via the gateway's SNAT
# fault FLEET_ANCHORS unreachable while the WAN is up
# action start tailscaled and `tailscale up` -> the box is reachable at its
# mesh address even though its LAN path is broken
#
# ⚠ TWO ANCHOR CLASSES, and they must not share a failure mode:
# WAN_ANCHOR plain internet. If THIS is down the site uplink is gone,
# Tailscale cannot fix it, and we do nothing -- thrashing
# during an ISP outage turns a wait into an incident.
# FLEET_ANCHORS hosts reachable ONLY through the gateway's SNAT path. Their
# loss is exactly the fault this exists for.
#
# ⚠⚠ THE REJOIN MUST USE --accept-routes=false. On 2026-09-14 a
# `tailscale up --accept-routes` on this box accepted 10.251.0.0/16 from the
# gateway -- its OWN subnet -- and routed the local network through the tunnel,
# black-holing it. That happened with a human watching. Here it would run
# unattended, during an incident, on a box already in trouble.
#
# ⚠ NO AUTO-LEAVE. Once the door is open it stays open until a human closes it
# (`systemctl disable --now tailscaled`). A watchdog that re-closes on recovery
# flaps, and a flapping recovery path is worse than none -- it would be down
# exactly when someone finally looked.
#
# Disable for planned work: touch /etc/fv-watchdog.disable
@@ -30,7 +42,7 @@ CONF=/etc/fv-mesh-watchdog.conf
DISABLE=/etc/fv-watchdog.disable
FAIL_THRESHOLD=5
WAN_ANCHOR=1.1.1.1
MESH_ANCHOR=100.64.0.1
FLEET_ANCHORS="10.100.10.50 10.100.50.40" # nh3-dev, nh3-docker
LOGIN_SERVER=https://headscale.phasefinal.com
[ -r "$CONF" ] && . "$CONF"
@@ -41,34 +53,42 @@ mkdir -p "$STATE"; COUNT_FILE="$STATE/consecutive_failures"
[ -f "$COUNT_FILE" ] || echo 0 > "$COUNT_FILE"
count=$(cat "$COUNT_FILE" 2>/dev/null || echo 0)
ping -c 2 -W 3 "$WAN_ANCHOR" >/dev/null 2>&1 && wan=up || wan=down
ping -c 2 -W 3 "$MESH_ANCHOR" >/dev/null 2>&1 && mesh=up || mesh=down
# Already on the mesh? Then the door is open and there is nothing to do. This is
# also what makes the script idempotent after it has fired.
if [ "$(systemctl is-active tailscaled 2>/dev/null)" = active ] \
&& tailscale status >/dev/null 2>&1; then
[ "$count" -ne 0 ] && echo 0 > "$COUNT_FILE"
exit 0
fi
ping -c 2 -W 3 "$WAN_ANCHOR" >/dev/null 2>&1 && wan=up || wan=down
if [ "$wan" = down ]; then
# Uplink is down. Tailscale cannot fix that and restarting it during an ISP
# outage only adds churn. Reset the counter so the mesh timer starts clean
# once the site is back.
[ "$count" -ne 0 ] && log "uplink down (wan=$wan mesh=$mesh) — not a mesh fault, counter reset"
[ "$count" -ne 0 ] && log "uplink down — not a fleet-path fault, counter reset"
echo 0 > "$COUNT_FILE"; exit 0
fi
if [ "$mesh" = up ]; then
[ "$count" -ne 0 ] && log "mesh recovered after $count failure(s)"
# Any fleet anchor answering means the normal path works.
fleet=down
for a in $FLEET_ANCHORS; do
if ping -c 2 -W 3 "$a" >/dev/null 2>&1; then fleet=up; break; fi
done
if [ "$fleet" = up ]; then
[ "$count" -ne 0 ] && log "fleet path recovered after $count failure(s)"
echo 0 > "$COUNT_FILE"; exit 0
fi
count=$((count + 1)); echo "$count" > "$COUNT_FILE"
log "mesh anchor $MESH_ANCHOR unreachable while WAN is up ($count/$FAIL_THRESHOLD)"
log "fleet unreachable via the gateway path while WAN is up ($count/$FAIL_THRESHOLD) — anchors: $FLEET_ANCHORS"
[ "$count" -lt "$FAIL_THRESHOLD" ] && exit 0
log "THRESHOLD REACHED — restoring known-good Tailscale state"
log "THRESHOLD REACHED — opening the break-glass mesh path"
echo 0 > "$COUNT_FILE"
tailscale set --accept-routes=false 2>&1 | while read -r l; do log " set: $l"; done
systemctl start tailscaled 2>&1 | while read -r l; do log " start: $l"; done
if [ -r "$STATE/authkey" ]; then
tailscale up --login-server "$LOGIN_SERVER" --authkey "$(cat "$STATE/authkey")" \
--accept-routes=false --hostname "$(hostname -s)" 2>&1 | while read -r l; do log " up: $l"; done
log "mesh path OPEN — reach this host at its 100.64.x address; close with 'systemctl disable --now tailscaled'"
else
log " no $STATE/authkey — cannot re-register; tried accept-routes=false only"
log " NO $STATE/authkey — cannot join. The break-glass path is UNARMED."
fi
systemctl restart tailscaled 2>&1 | while read -r l; do log " restart: $l"; done
log "restore attempt complete"