WATCHDOG (done, proven). fv-mesh-watchdog probes two independent anchors every
minute and, after 5 consecutive failures, puts Tailscale back to known-good:
accept-routes off, re-up against headscale with a stored key. It touches
nothing else — a watchdog with a wide remit is a second way to lose the box.
Two anchors that cannot share a failure mode: a plain-internet one and a
mesh-only one. If BOTH fail the site uplink is down, Tailscale cannot fix that,
and it deliberately does nothing — thrashing tailscaled during an ISP outage
turns a wait into an incident. Disable file at /etc/fv-watchdog.disable for
planned work.
Proven by positive control, not assumed: counter incremented 1..4 without
acting, fired the restore at 5 (tailscale up ran, tailscaled restarted), and
reset to 0 once the real anchor returned. fv-ml1 stayed reachable throughout.
This exists because a on fv-ml1 black-holed it
from its own LAN earlier the same day: it accepted 10.251.0.0/16 from the
gateway — its OWN subnet — and routed the local network through the tunnel.
FV CROSS-SITE ROUTING (partial). Two changes landed, the path is still broken:
1. acceptSubnetRoutes 0 -> 1 on the FV gateway's tailscale plugin, via
settings/set + service/reconfigure (the documented apply, not a reboot).
The GATEWAY now has 10.0/16, 10.100/16 and 10.250/16 in its routing table
and reaches NH3 and ESH itself. It could not before.
2. Remote-site MASQUERADE rules on nh3-scale. The existing jump matched only
-s 100.64.0.0/10, so traffic from another site's LAN never entered
MESH-EXIT and kept its original source; an NH3 host then replied via its
own LAN router instead of back through nh3-scale, making the path
asymmetric. The rule is confirmed firing (counter increments on FV
traffic) but does not complete the path.
Still failing: fv-ml1 -> NH3/ESH LAN addresses. Mesh addresses work perfectly
from fv-ml1 (100.64.0.1, 100.64.0.4), Anaheim works over the metro link, and
the FV firewall log shows the outbound passing on tailscale0 with
src=10.251.50.54 and no reply ever returning. The remaining gap is forwarded
FV-LAN traffic specifically, not the gateway's own.
Full regression sweep clean: nh3-dev, ana-docker and esh-docker-vm all reach
all four sites plus the internet.
One rule on nh3-scale (CT 107): -d 10.100.10.50/32 -j MASQUERADE, above the
RFC1918 RETURNs in /usr/local/sbin/mesh-exit-masq.sh, so it survives a reboot
rather than living only in the running ruleset.
Cause. A host that runs Tailscale installs -A ts-input -s 100.64.0.0/10
! -i tailscale0 -j DROP. The fleet's subnet routers run NoSNAT: true with
RFC1918 explicitly exempted from masquerade — deliberate source preservation,
and a departure from Tailscale's own --snat-subnet-routes=true default — so a
mesh client's packet reached nh3-dev's ens18 still sourced 100.64.x and died
at the anti-spoof rule. Every NH3 host that does not run Tailscale was
unaffected, which is why this read as a DNS or routing fault rather than a
policy one. Masquerading just this destination makes it behave like every
other host and leaves source preservation absolute elsewhere.
Verified before and after against 13 targets from nh3-dev and 9 from the
MacBook Air, and again after restarting the service so the chain was rebuilt
from the script rather than from the manual insert. nh3-dev.nh3.internal now
resolves and connects from the mesh, ssh and the Booth port included, with no
script changes anywhere.
Records the failed approach prominently, because it is the attractive one:
advertising 10.100.10.50/32 from nh3-dev itself black-holed it from ESH,
Anaheim, FV and Irvine. ip rule there puts lookup 52 at priority 5270 ahead of
main at 32766, and becoming a subnet router let table 52 capture cross-site
traffic the node has no accepted route for. Its own LAN and the internet kept
working throughout, so a single-host check confirms a break it cannot see.
vh