diff --git a/docs/runbooks/fv-to-ana-nat.md b/docs/runbooks/fv-to-ana-nat.md index e72621e..e52b37d 100644 --- a/docs/runbooks/fv-to-ana-nat.md +++ b/docs/runbooks/fv-to-ana-nat.md @@ -17,7 +17,52 @@ One manual IPv4 rule: No routes, access/filter rules, WAN settings, subnet advertisements, host Tailscale settings, or other FV source addresses were changed. Other remote -sites remain outside this fix's scope. Translation means Anaheim services may +sites remain outside this fix's scope. + +## ⭐ EXTENDED 2026-09-15 — NH3, ESH and Irvine now have the same rule + +The Anaheim-only scope above was the whole reason FV could not reach anywhere +else, and it cost an evening to rediscover. Three rules added, each an exact +mirror of the ANA one — same interface, same source, only the destination +differs: + +| destination | description | +|---|---| +| `10.100.0.0/16` | `fv-ml1 to NH3 via Tailscale SNAT` | +| `10.0.0.0/16` | `fv-ml1 to ESH via Tailscale SNAT` | +| `10.6.110.0/24` | `fv-ml1 to IRV via Tailscale SNAT` | + +Applied through `POST /api/firewall/source_nat/add_rule` + `.../apply`, with a +pre-change `core/backup/download/this` taken first. After: fv-ml1 reaches all +four sites, the mesh and the internet; nh3-dev, nh3-docker and esh-docker-vm +regression-swept clean. + +⚠ **The source is still `10.251.50.54/32` — fv-ml1 only.** A second host at FV +will have exactly this problem and no symptom that points at NAT. Broaden to +`10.251.50.0/24`, or add a rule per host, when FV grows. + +### What this looks like when it breaks, so the next person skips the evening + +The failure presents as a routing or Tailscale problem and is neither: + +- fv-ml1 reaches mesh addresses (`100.64.0.x`) perfectly and LAN addresses not + at all — which reads like a subnet-router fault. +- The FV firewall log shows the outbound **passing** on tailscale0 with + `src=10.251.50.54` and no reply ever returning, so nothing looks blocked. +- The far-side router genuinely receives and replies — verified with temporary + counting rules on nh3-scale: 5 packets in, 4 replies out. +- Both peers' Tailscale `AllowedIPs` are correct, so cryptokey routing is fine. +- **Every other site pair works**, which rules out a general subnet-to-subnet + limitation: `nh3-docker → esh/ana/FV` and `esh-docker-vm → FV` all succeed. + +That combination means outbound SNAT, not routing. Check +`/api/firewall/source_nat/search_rule` for a rule covering the destination +before investigating anything else. + +⚠ Also retired on the way: a set of remote-site MASQUERADE rules added to +nh3-scale's `mesh-exit-masq.sh` on the asymmetric-return theory. They fired +(counters incremented) but were not the fix, and were reverted rather than left +to accumulate. Translation means Anaheim services may see the gateway identity rather than fv-ml1's original LAN address. ## Evidence and verification diff --git a/servers/nh3-pve/mesh-exit-masq.sh b/servers/nh3-pve/mesh-exit-masq.sh index c55331b..41fd027 100644 --- a/servers/nh3-pve/mesh-exit-masq.sh +++ b/servers/nh3-pve/mesh-exit-masq.sh @@ -31,23 +31,3 @@ iptables -t nat -A MESH-EXIT -j MASQUERADE iptables -t nat -C POSTROUTING -s 100.64.0.0/10 -o eth0 -j MESH-EXIT 2>/dev/null \ || iptables -t nat -A POSTROUTING -s 100.64.0.0/10 -o eth0 -j MESH-EXIT -# ── Remote-SITE sources, not just mesh clients ──────────────────────────────── -# The jump above matches only 100.64.0.0/10, so traffic from another site's LAN -# arriving over the mesh never enters MESH-EXIT and keeps its original source. -# An NH3 host then replies via its own LAN router instead of back through this -# node, the path is asymmetric, and the reply is lost. Measured 2026-09-15: -# fv-ml1 reached 100.64.0.1 and 100.64.0.4 fine while 10.100.50.40 failed -# outright, and the FV firewall log showed the outbound passing with -# src=10.251.50.54 and nothing ever coming back. -# -# Masquerading remote-site sources onto this node's LAN address makes the reply -# return here, where the conntrack state lives. It costs source visibility for -# cross-site traffic on the NH3 LAN — the same trade as the nh3-dev rule above, -# and the alternative is no connectivity at all. -# -# ⚠ Sites are listed explicitly rather than using 10.0.0.0/8: a blanket rule -# would also masquerade NH3-local traffic that has no business being rewritten. -for site_net in 10.251.0.0/16 10.0.0.0/16 10.250.0.0/16 10.6.110.0/24; do - iptables -t nat -C POSTROUTING -s "$site_net" -o eth0 -j MASQUERADE 2>/dev/null \ - || iptables -t nat -A POSTROUTING -s "$site_net" -o eth0 -j MASQUERADE -done