feat(fv): mesh dead-man's switch on fv-ml1; partial progress on FV cross-site routing

WATCHDOG (done, proven). fv-mesh-watchdog probes two independent anchors every
minute and, after 5 consecutive failures, puts Tailscale back to known-good:
accept-routes off, re-up against headscale with a stored key. It touches
nothing else — a watchdog with a wide remit is a second way to lose the box.

Two anchors that cannot share a failure mode: a plain-internet one and a
mesh-only one. If BOTH fail the site uplink is down, Tailscale cannot fix that,
and it deliberately does nothing — thrashing tailscaled during an ISP outage
turns a wait into an incident. Disable file at /etc/fv-watchdog.disable for
planned work.

Proven by positive control, not assumed: counter incremented 1..4 without
acting, fired the restore at 5 (tailscale up ran, tailscaled restarted), and
reset to 0 once the real anchor returned. fv-ml1 stayed reachable throughout.

This exists because a  on fv-ml1 black-holed it
from its own LAN earlier the same day: it accepted 10.251.0.0/16 from the
gateway — its OWN subnet — and routed the local network through the tunnel.

FV CROSS-SITE ROUTING (partial). Two changes landed, the path is still broken:

  1. acceptSubnetRoutes 0 -> 1 on the FV gateway's tailscale plugin, via
     settings/set + service/reconfigure (the documented apply, not a reboot).
     The GATEWAY now has 10.0/16, 10.100/16 and 10.250/16 in its routing table
     and reaches NH3 and ESH itself. It could not before.

  2. Remote-site MASQUERADE rules on nh3-scale. The existing jump matched only
     -s 100.64.0.0/10, so traffic from another site's LAN never entered
     MESH-EXIT and kept its original source; an NH3 host then replied via its
     own LAN router instead of back through nh3-scale, making the path
     asymmetric. The rule is confirmed firing (counter increments on FV
     traffic) but does not complete the path.

Still failing: fv-ml1 -> NH3/ESH LAN addresses. Mesh addresses work perfectly
from fv-ml1 (100.64.0.1, 100.64.0.4), Anaheim works over the metro link, and
the FV firewall log shows the outbound passing on tailscale0 with
src=10.251.50.54 and no reply ever returning. The remaining gap is forwarded
FV-LAN traffic specifically, not the gateway's own.

Full regression sweep clean: nh3-dev, ana-docker and esh-docker-vm all reach
all four sites plus the internet.
This commit is contained in:
vh
2026-09-15 00:25:55 -07:00
parent 80d982d1d8
commit 8c8559b8ec
4 changed files with 109 additions and 0 deletions
+21
View File
@@ -30,3 +30,24 @@ iptables -t nat -A MESH-EXIT -d 100.64.0.0/10 -j RETURN
iptables -t nat -A MESH-EXIT -j MASQUERADE
iptables -t nat -C POSTROUTING -s 100.64.0.0/10 -o eth0 -j MESH-EXIT 2>/dev/null \
|| iptables -t nat -A POSTROUTING -s 100.64.0.0/10 -o eth0 -j MESH-EXIT
# ── Remote-SITE sources, not just mesh clients ────────────────────────────────
# The jump above matches only 100.64.0.0/10, so traffic from another site's LAN
# arriving over the mesh never enters MESH-EXIT and keeps its original source.
# An NH3 host then replies via its own LAN router instead of back through this
# node, the path is asymmetric, and the reply is lost. Measured 2026-09-15:
# fv-ml1 reached 100.64.0.1 and 100.64.0.4 fine while 10.100.50.40 failed
# outright, and the FV firewall log showed the outbound passing with
# src=10.251.50.54 and nothing ever coming back.
#
# Masquerading remote-site sources onto this node's LAN address makes the reply
# return here, where the conntrack state lives. It costs source visibility for
# cross-site traffic on the NH3 LAN — the same trade as the nh3-dev rule above,
# and the alternative is no connectivity at all.
#
# ⚠ Sites are listed explicitly rather than using 10.0.0.0/8: a blanket rule
# would also masquerade NH3-local traffic that has no business being rewritten.
for site_net in 10.251.0.0/16 10.0.0.0/16 10.250.0.0/16 10.6.110.0/24; do
iptables -t nat -C POSTROUTING -s "$site_net" -o eth0 -j MASQUERADE 2>/dev/null \
|| iptables -t nat -A POSTROUTING -s "$site_net" -o eth0 -j MASQUERADE
done