vh
0ab9da5b89
All four outbound-NAT rules on the FV gateway now match source 10.251.50.0/24 instead of fv-ml1's single address, so a second host at FV works on arrival rather than reproducing a failure whose symptoms point at routing rather than NAT. Anaheim got a /24 rule of its own through the API. The 2026-09-13 ANA rule was written with write_config and is invisible to source_nat/search_rule, so leaving it as the only ANA coverage would have kept one destination on a different code path from the other three. The legacy /32 rule is now redundant but harmless -- it NATs identically and first-match wins -- and is noted in the runbook for deletion from the UI, since it is the one rule the API cannot see. Descriptions rewritten to name the real scope. Three of them said "fv-ml1 to X" while covering the whole subnet, and a description that understates a rule's reach is the same trap as the Anaheim-only scope that caused this. Verified after: fv-ml1 reaches NH3, nh3-dev, ESH, Anaheim, Irvine, the mesh and the internet; nh3-dev, esh-docker-vm and ana-docker all reach FV and each other; the FV BMC remains reachable inbound. Pre-change config backup taken. Also records that 10.251.250.0/24 (BMC/management) is deliberately NOT covered -- inbound reachability is what out-of-band recovery needs, but a management host originating traffic to another site would hit this same wall.
6.8 KiB
FV to Anaheim: scoped Tailscale NAT repair
Applied 2026-09-13 ~06:22Z after operator approval with explicit instruction not to strand the subnet. Gateway: OPNsense at 10.251.50.1, current hostname vb-gateway.internal; tailscale0 address 100.64.0.8, assigned interface opt6/MESH.
Persistent configuration
Firewall / NAT / Outbound: Hybrid mode, retaining every automatic WAN rule. One manual IPv4 rule:
- Interface: MESH (opt6 / tailscale0)
- Source: 10.251.50.54/32 (fv-ml1 only)
- Destination: 10.250.0.0/16 (Anaheim only)
- Translation: interface address (100.64.0.8 currently)
- Description:
fv-ml1 to ANA via Tailscale SNAT
No routes, access/filter rules, WAN settings, subnet advertisements, host Tailscale settings, or other FV source addresses were changed. Other remote sites remain outside this fix's scope.
⭐ EXTENDED 2026-09-15 — NH3, ESH and Irvine now have the same rule
The Anaheim-only scope above was the whole reason FV could not reach anywhere else, and it cost an evening to rediscover. Three rules added, each an exact mirror of the ANA one — same interface, same source, only the destination differs:
| destination | description |
|---|---|
10.100.0.0/16 |
fv-ml1 to NH3 via Tailscale SNAT |
10.0.0.0/16 |
fv-ml1 to ESH via Tailscale SNAT |
10.6.110.0/24 |
fv-ml1 to IRV via Tailscale SNAT |
Applied through POST /api/firewall/source_nat/add_rule + .../apply, with a
pre-change core/backup/download/this taken first. After: fv-ml1 reaches all
four sites, the mesh and the internet; nh3-dev, nh3-docker and esh-docker-vm
regression-swept clean.
Broadened to the whole FV LAN, same session
All four rules now carry source 10.251.50.0/24 rather than fv-ml1's /32,
so a second host at FV works on arrival instead of reproducing this failure with
no symptom that points at NAT. Anaheim got a /24 rule of its own through the
API for the same reason — the 2026-09-13 rule was written with write_config
and is invisible to source_nat/search_rule, so leaving it as the only ANA
coverage would have kept one destination on a different code path from the
other three.
src=10.251.50.0/24 dst=10.100.0.0/16 FV LAN 10.251.50.0/24 to NH3 via Tailscale SNAT
src=10.251.50.0/24 dst=10.0.0.0/16 FV LAN 10.251.50.0/24 to ESH via Tailscale SNAT
src=10.251.50.0/24 dst=10.6.110.0/24 FV LAN 10.251.50.0/24 to IRV via Tailscale SNAT
src=10.251.50.0/24 dst=10.250.0.0/16 FV LAN 10.251.50.0/24 to ANA via Tailscale SNAT
Descriptions were rewritten to name the actual scope. A description that understates a rule's reach is the same trap as the Anaheim-only rule itself.
⚠ The legacy /32 ANA rule still exists in config.xml and is now
redundant. Harmless — it NATs identically and first-match wins — but it is the
one rule the API cannot see, so delete it from the UI when convenient rather
than wondering later why the counts disagree.
⚠ 10.251.250.0/24 (BMC / management) is NOT covered. The rules are scoped
to the server VLAN only. The BMC is reachable inbound from the fleet, which is
what matters for out-of-band recovery, but a management-VLAN host originating
traffic to another site would hit this same wall.
What this looks like when it breaks, so the next person skips the evening
The failure presents as a routing or Tailscale problem and is neither:
- fv-ml1 reaches mesh addresses (
100.64.0.x) perfectly and LAN addresses not at all — which reads like a subnet-router fault. - The FV firewall log shows the outbound passing on tailscale0 with
src=10.251.50.54and no reply ever returning, so nothing looks blocked. - The far-side router genuinely receives and replies — verified with temporary counting rules on nh3-scale: 5 packets in, 4 replies out.
- Both peers' Tailscale
AllowedIPsare correct, so cryptokey routing is fine. - Every other site pair works, which rules out a general subnet-to-subnet
limitation:
nh3-docker → esh/ana/FVandesh-docker-vm → FVall succeed.
That combination means outbound SNAT, not routing. Check
/api/firewall/source_nat/search_rule for a rule covering the destination
before investigating anything else.
⚠ Also retired on the way: a set of remote-site MASQUERADE rules added to
nh3-scale's mesh-exit-masq.sh on the asymmetric-return theory. They fired
(counters incremented) but were not the fix, and were reverted rather than left
to accumulate. Translation means Anaheim services may
see the gateway identity rather than fv-ml1's original LAN address.
Evidence and verification
Before: requests reached ANA with source 10.251.50.54; ANA sent SYN-ACKs then received resets attributed to that address, while FV host packet captures showed neither incoming SYN-ACKs nor outgoing resets. Gateway itself got hub HTTP200. Rules contained WAN NAT only, none on tailscale0. FV->ANA SSH, Beszel, Restic and ICMP failed; ANA->FV SSH and ping worked.
Temporary /32 host-to-hub NAT immediately fixed HTTP and restored Beszel. Only NAT rules were loaded for this first test. Persistent /32-to-ANA-/16 rule then saved using OPNsense write_config, followed by configctl filter reload. Both stages had independent 240-second rollback watchdogs. Before disarming, verified all preexisting NAT lines retained and pfctl -sr byte-identical.
After: FV->hub HTTP200, FV->ANA PostgreSQL TCP5432 reachable, FV internet HTTPS200, ANA->FV SSH reachable, gateway management intact. Beszel 18/18 up. Agent logs show WebSocket connected and intentional SSH listener shutdown; 45876 refusing after WebSocket connection is NOT a loss of host connectivity. A single BMC ping to 10.251.50.50 did not answer; no prior BMC baseline was taken, so BMC health is not claimed. The NAT rule excludes BMC source traffic.
Backup and rollback
Root-only directory on gateway: /root/fv-nat-repair-20260913/ contains
config.xml.before (secret-bearing, never commit), nat.before, filter.before,
nat-config.before.json, nat.after, nat-config.php and verified marker.
Automatic rollback disarmed only after verification; backup retained.
Manual rollback, after checking for intervening NAT edits:
php /root/fv-nat-repair-20260913/nat-config.php rollback
configctl filter reload
This restores the saved NAT config section, not unrelated configuration.
It will remove subsequent NAT edits too: inspect before using later.
Emergency runtime NAT-only restore: pfctl -N -f /root/fv-nat-repair-20260913/nat.before.
Never use pfctl -F all or flush all states for this repair.
Gateway SSH key auth is unavailable; use vaulted fv-gateway/root-password or approved infra-ops credentials, without logging secrets. This session used /tmp/fv-elway.py (FreeBSD sh + legacy SCP + existing SSH control socket).
References: OPNsense hybrid NAT, Tailscale FreeBSD userspace routing.