# FV to Anaheim: scoped Tailscale NAT repair Applied 2026-09-13 ~06:22Z after operator approval with explicit instruction not to strand the subnet. Gateway: OPNsense at 10.251.50.1, current hostname vb-gateway.internal; tailscale0 address 100.64.0.8, assigned interface opt6/MESH. ## Persistent configuration Firewall / NAT / Outbound: **Hybrid** mode, retaining every automatic WAN rule. One manual IPv4 rule: - Interface: MESH (opt6 / tailscale0) - Source: 10.251.50.54/32 (fv-ml1 only) - Destination: 10.250.0.0/16 (Anaheim only) - Translation: interface address (100.64.0.8 currently) - Description: `fv-ml1 to ANA via Tailscale SNAT` No routes, access/filter rules, WAN settings, subnet advertisements, host Tailscale settings, or other FV source addresses were changed. Other remote sites remain outside this fix's scope. ## ⭐ EXTENDED 2026-09-15 — NH3, ESH and Irvine now have the same rule The Anaheim-only scope above was the whole reason FV could not reach anywhere else, and it cost an evening to rediscover. Three rules added, each an exact mirror of the ANA one — same interface, same source, only the destination differs: | destination | description | |---|---| | `10.100.0.0/16` | `fv-ml1 to NH3 via Tailscale SNAT` | | `10.0.0.0/16` | `fv-ml1 to ESH via Tailscale SNAT` | | `10.6.110.0/24` | `fv-ml1 to IRV via Tailscale SNAT` | Applied through `POST /api/firewall/source_nat/add_rule` + `.../apply`, with a pre-change `core/backup/download/this` taken first. After: fv-ml1 reaches all four sites, the mesh and the internet; nh3-dev, nh3-docker and esh-docker-vm regression-swept clean. ### Broadened to the whole FV LAN, same session All four rules now carry source **`10.251.50.0/24`** rather than fv-ml1's `/32`, so a second host at FV works on arrival instead of reproducing this failure with no symptom that points at NAT. Anaheim got a `/24` rule of its own through the API for the same reason — the 2026-09-13 rule was written with `write_config` and is invisible to `source_nat/search_rule`, so leaving it as the only ANA coverage would have kept one destination on a different code path from the other three. src=10.251.50.0/24 dst=10.100.0.0/16 FV LAN 10.251.50.0/24 to NH3 via Tailscale SNAT src=10.251.50.0/24 dst=10.0.0.0/16 FV LAN 10.251.50.0/24 to ESH via Tailscale SNAT src=10.251.50.0/24 dst=10.6.110.0/24 FV LAN 10.251.50.0/24 to IRV via Tailscale SNAT src=10.251.50.0/24 dst=10.250.0.0/16 FV LAN 10.251.50.0/24 to ANA via Tailscale SNAT Descriptions were rewritten to name the actual scope. A description that understates a rule's reach is the same trap as the Anaheim-only rule itself. ⚠ **The legacy `/32` ANA rule still exists** in `config.xml` and is now redundant. Harmless — it NATs identically and first-match wins — but it is the one rule the API cannot see, so delete it from the UI when convenient rather than wondering later why the counts disagree. ⚠ **`10.251.250.0/24` (BMC / management) is NOT covered.** The rules are scoped to the server VLAN only. The BMC is reachable inbound from the fleet, which is what matters for out-of-band recovery, but a management-VLAN host originating traffic to another site would hit this same wall. ### What this looks like when it breaks, so the next person skips the evening The failure presents as a routing or Tailscale problem and is neither: - fv-ml1 reaches mesh addresses (`100.64.0.x`) perfectly and LAN addresses not at all — which reads like a subnet-router fault. - The FV firewall log shows the outbound **passing** on tailscale0 with `src=10.251.50.54` and no reply ever returning, so nothing looks blocked. - The far-side router genuinely receives and replies — verified with temporary counting rules on nh3-scale: 5 packets in, 4 replies out. - Both peers' Tailscale `AllowedIPs` are correct, so cryptokey routing is fine. - **Every other site pair works**, which rules out a general subnet-to-subnet limitation: `nh3-docker → esh/ana/FV` and `esh-docker-vm → FV` all succeed. That combination means outbound SNAT, not routing. Check `/api/firewall/source_nat/search_rule` for a rule covering the destination before investigating anything else. ⚠ Also retired on the way: a set of remote-site MASQUERADE rules added to nh3-scale's `mesh-exit-masq.sh` on the asymmetric-return theory. They fired (counters incremented) but were not the fix, and were reverted rather than left to accumulate. Translation means Anaheim services may see the gateway identity rather than fv-ml1's original LAN address. ## Evidence and verification Before: requests reached ANA with source 10.251.50.54; ANA sent SYN-ACKs then received resets attributed to that address, while FV host packet captures showed neither incoming SYN-ACKs nor outgoing resets. Gateway itself got hub HTTP200. Rules contained WAN NAT only, none on tailscale0. FV->ANA SSH, Beszel, Restic and ICMP failed; ANA->FV SSH and ping worked. Temporary /32 host-to-hub NAT immediately fixed HTTP and restored Beszel. Only NAT rules were loaded for this first test. Persistent /32-to-ANA-/16 rule then saved using OPNsense write_config, followed by configctl filter reload. Both stages had independent 240-second rollback watchdogs. Before disarming, verified all preexisting NAT lines retained and pfctl -sr byte-identical. After: FV->hub HTTP200, FV->ANA PostgreSQL TCP5432 reachable, FV internet HTTPS200, ANA->FV SSH reachable, gateway management intact. Beszel **18/18 up**. Agent logs show WebSocket connected and intentional SSH listener shutdown; 45876 refusing after WebSocket connection is NOT a loss of host connectivity. A single BMC ping to 10.251.50.50 did not answer; no prior BMC baseline was taken, so BMC health is not claimed. The NAT rule excludes BMC source traffic. ## Backup and rollback Root-only directory on gateway: `/root/fv-nat-repair-20260913/` contains config.xml.before (secret-bearing, never commit), nat.before, filter.before, nat-config.before.json, nat.after, nat-config.php and verified marker. Automatic rollback disarmed only after verification; backup retained. Manual rollback, after checking for intervening NAT edits: ```sh php /root/fv-nat-repair-20260913/nat-config.php rollback configctl filter reload ``` This restores the saved NAT config section, not unrelated configuration. It will remove subsequent NAT edits too: inspect before using later. Emergency runtime NAT-only restore: `pfctl -N -f /root/fv-nat-repair-20260913/nat.before`. Never use pfctl -F all or flush all states for this repair. Gateway SSH key auth is unavailable; use vaulted fv-gateway/root-password or approved infra-ops credentials, without logging secrets. This session used /tmp/fv-elway.py (FreeBSD sh + legacy SCP + existing SSH control socket). References: [OPNsense hybrid NAT](https://docs.opnsense.org/manual/nat.html), [Tailscale FreeBSD userspace routing](https://tailscale.com/docs/reference/kernel-vs-userspace-routers).