Operator's design, and a better one. Once the FV SNAT rules landed, fv-ml1's
mesh membership was redundant for routing and its only remaining value was as a
second way in. Keeping it enrolled bought a standing second door; joining on
demand buys the same recovery path without one.
normal tailscaled stopped + disabled; fleet reached via the gateway SNAT
fault nh3-dev / nh3-docker unreachable while the WAN is up
action start tailscaled + tailscale up -> reachable at its 100.64.x address
fv-ml1 is now off the mesh and its node record deleted. Verified it still
reaches NH3, ESH, Anaheim, Irvine and the internet on the SNAT path alone, then
the break-glass fired on cue (counted 1..4, joined at 5 as 100.64.0.10),
answered ping and ssh from nh3-dev, and was closed again cleanly.
No auto-leave, deliberately: once open the door stays open until a human runs
systemctl disable --now tailscaled. A watchdog that re-closes on recovery flaps,
and a flapping recovery path is down exactly when someone finally looks. It also
skips entirely when already on the mesh, which is what makes it idempotent after
firing.
The question exposed a hole worth more than the redesign. The stored rejoin key
was one of the 2026-09-12 FV cutover keys, expiring 2026-09-19 -- a break-glass
credential that dies in four days and fails silently at the only moment it
matters. Replaced with a dedicated 1-year reusable key (headscale ID 8, expires
2027-09-15), vaulted as fv-ml1/headscale-breakglass-key, root:600 on the host.
That also closes the standing self-join risk rather than trading it: the two
stale reusable keys (IDs 5, 6) are expired, so the mesh now has exactly one live
reusable key -- purpose-built, on a host we control -- instead of two orphans
nobody owned.
Rejoin uses --accept-routes=false and the reason is in the script: on 2026-09-14
tailscale up --accept-routes on this box accepted its OWN subnet from the
gateway and black-holed it. That happened with a human watching; here it runs
unattended, during an incident, on a box already in trouble.
Session captured: the FV outbound-NAT root cause and its diagnostic signature,
the fv-ml1 dead man's switch, fleet identity/group/path conventions and the
root:docker normalization, nh3-dev's ts-input reachability fix, ESPHome
modernisation and the kb KB-search tool, and the Hermes bearer rotation
release. Six new detail files.
Tried-and-abandoned gains three: probing OPNsense endpoints by POSTing at them
(which rebooted the FV firewall), advertising a /32 from nh3-dev, and the
nh3-scale remote-site masquerade rules that fired but were not the fix.
Housekeeping: 8 Recent-decisions entries archived to archival-memory.md, and 21
oversized inline entries split into detail files per the two-tier rule -- they
had been sitting fully inline in the index, which is what the split exists to
prevent. Two pointers to a detail file archived this run were repointed at
archival-memory.md.
The index is 389 lines, still over the ~300 soft cap. The archival guards stop
it there: only 4 further entries are old enough to move and every one carries an
open deferred-work pointer. An over-cap file that keeps live decisions beats a
scannable one that lost a deferred call.
vh