vh
0ab9da5b89
All four outbound-NAT rules on the FV gateway now match source 10.251.50.0/24 instead of fv-ml1's single address, so a second host at FV works on arrival rather than reproducing a failure whose symptoms point at routing rather than NAT. Anaheim got a /24 rule of its own through the API. The 2026-09-13 ANA rule was written with write_config and is invisible to source_nat/search_rule, so leaving it as the only ANA coverage would have kept one destination on a different code path from the other three. The legacy /32 rule is now redundant but harmless -- it NATs identically and first-match wins -- and is noted in the runbook for deletion from the UI, since it is the one rule the API cannot see. Descriptions rewritten to name the real scope. Three of them said "fv-ml1 to X" while covering the whole subnet, and a description that understates a rule's reach is the same trap as the Anaheim-only scope that caused this. Verified after: fv-ml1 reaches NH3, nh3-dev, ESH, Anaheim, Irvine, the mesh and the internet; nh3-dev, esh-docker-vm and ana-docker all reach FV and each other; the FV BMC remains reachable inbound. Pre-change config backup taken. Also records that 10.251.250.0/24 (BMC/management) is deliberately NOT covered -- inbound reachability is what out-of-band recovery needs, but a management host originating traffic to another site would hit this same wall.
138 lines
6.8 KiB
Markdown
138 lines
6.8 KiB
Markdown
# FV to Anaheim: scoped Tailscale NAT repair
|
|
|
|
Applied 2026-09-13 ~06:22Z after operator approval with explicit instruction
|
|
not to strand the subnet. Gateway: OPNsense at 10.251.50.1, current hostname
|
|
vb-gateway.internal; tailscale0 address 100.64.0.8, assigned interface opt6/MESH.
|
|
|
|
## Persistent configuration
|
|
|
|
Firewall / NAT / Outbound: **Hybrid** mode, retaining every automatic WAN rule.
|
|
One manual IPv4 rule:
|
|
|
|
- Interface: MESH (opt6 / tailscale0)
|
|
- Source: 10.251.50.54/32 (fv-ml1 only)
|
|
- Destination: 10.250.0.0/16 (Anaheim only)
|
|
- Translation: interface address (100.64.0.8 currently)
|
|
- Description: `fv-ml1 to ANA via Tailscale SNAT`
|
|
|
|
No routes, access/filter rules, WAN settings, subnet advertisements, host
|
|
Tailscale settings, or other FV source addresses were changed. Other remote
|
|
sites remain outside this fix's scope.
|
|
|
|
## ⭐ EXTENDED 2026-09-15 — NH3, ESH and Irvine now have the same rule
|
|
|
|
The Anaheim-only scope above was the whole reason FV could not reach anywhere
|
|
else, and it cost an evening to rediscover. Three rules added, each an exact
|
|
mirror of the ANA one — same interface, same source, only the destination
|
|
differs:
|
|
|
|
| destination | description |
|
|
|---|---|
|
|
| `10.100.0.0/16` | `fv-ml1 to NH3 via Tailscale SNAT` |
|
|
| `10.0.0.0/16` | `fv-ml1 to ESH via Tailscale SNAT` |
|
|
| `10.6.110.0/24` | `fv-ml1 to IRV via Tailscale SNAT` |
|
|
|
|
Applied through `POST /api/firewall/source_nat/add_rule` + `.../apply`, with a
|
|
pre-change `core/backup/download/this` taken first. After: fv-ml1 reaches all
|
|
four sites, the mesh and the internet; nh3-dev, nh3-docker and esh-docker-vm
|
|
regression-swept clean.
|
|
|
|
### Broadened to the whole FV LAN, same session
|
|
|
|
All four rules now carry source **`10.251.50.0/24`** rather than fv-ml1's `/32`,
|
|
so a second host at FV works on arrival instead of reproducing this failure with
|
|
no symptom that points at NAT. Anaheim got a `/24` rule of its own through the
|
|
API for the same reason — the 2026-09-13 rule was written with `write_config`
|
|
and is invisible to `source_nat/search_rule`, so leaving it as the only ANA
|
|
coverage would have kept one destination on a different code path from the
|
|
other three.
|
|
|
|
src=10.251.50.0/24 dst=10.100.0.0/16 FV LAN 10.251.50.0/24 to NH3 via Tailscale SNAT
|
|
src=10.251.50.0/24 dst=10.0.0.0/16 FV LAN 10.251.50.0/24 to ESH via Tailscale SNAT
|
|
src=10.251.50.0/24 dst=10.6.110.0/24 FV LAN 10.251.50.0/24 to IRV via Tailscale SNAT
|
|
src=10.251.50.0/24 dst=10.250.0.0/16 FV LAN 10.251.50.0/24 to ANA via Tailscale SNAT
|
|
|
|
Descriptions were rewritten to name the actual scope. A description that
|
|
understates a rule's reach is the same trap as the Anaheim-only rule itself.
|
|
|
|
⚠ **The legacy `/32` ANA rule still exists** in `config.xml` and is now
|
|
redundant. Harmless — it NATs identically and first-match wins — but it is the
|
|
one rule the API cannot see, so delete it from the UI when convenient rather
|
|
than wondering later why the counts disagree.
|
|
|
|
⚠ **`10.251.250.0/24` (BMC / management) is NOT covered.** The rules are scoped
|
|
to the server VLAN only. The BMC is reachable inbound from the fleet, which is
|
|
what matters for out-of-band recovery, but a management-VLAN host originating
|
|
traffic to another site would hit this same wall.
|
|
|
|
### What this looks like when it breaks, so the next person skips the evening
|
|
|
|
The failure presents as a routing or Tailscale problem and is neither:
|
|
|
|
- fv-ml1 reaches mesh addresses (`100.64.0.x`) perfectly and LAN addresses not
|
|
at all — which reads like a subnet-router fault.
|
|
- The FV firewall log shows the outbound **passing** on tailscale0 with
|
|
`src=10.251.50.54` and no reply ever returning, so nothing looks blocked.
|
|
- The far-side router genuinely receives and replies — verified with temporary
|
|
counting rules on nh3-scale: 5 packets in, 4 replies out.
|
|
- Both peers' Tailscale `AllowedIPs` are correct, so cryptokey routing is fine.
|
|
- **Every other site pair works**, which rules out a general subnet-to-subnet
|
|
limitation: `nh3-docker → esh/ana/FV` and `esh-docker-vm → FV` all succeed.
|
|
|
|
That combination means outbound SNAT, not routing. Check
|
|
`/api/firewall/source_nat/search_rule` for a rule covering the destination
|
|
before investigating anything else.
|
|
|
|
⚠ Also retired on the way: a set of remote-site MASQUERADE rules added to
|
|
nh3-scale's `mesh-exit-masq.sh` on the asymmetric-return theory. They fired
|
|
(counters incremented) but were not the fix, and were reverted rather than left
|
|
to accumulate. Translation means Anaheim services may
|
|
see the gateway identity rather than fv-ml1's original LAN address.
|
|
|
|
## Evidence and verification
|
|
|
|
Before: requests reached ANA with source 10.251.50.54; ANA sent SYN-ACKs then
|
|
received resets attributed to that address, while FV host packet captures
|
|
showed neither incoming SYN-ACKs nor outgoing resets. Gateway itself got hub
|
|
HTTP200. Rules contained WAN NAT only, none on tailscale0. FV->ANA SSH, Beszel,
|
|
Restic and ICMP failed; ANA->FV SSH and ping worked.
|
|
|
|
Temporary /32 host-to-hub NAT immediately fixed HTTP and restored Beszel.
|
|
Only NAT rules were loaded for this first test. Persistent /32-to-ANA-/16 rule
|
|
then saved using OPNsense write_config, followed by configctl filter reload.
|
|
Both stages had independent 240-second rollback watchdogs. Before disarming,
|
|
verified all preexisting NAT lines retained and pfctl -sr byte-identical.
|
|
|
|
After: FV->hub HTTP200, FV->ANA PostgreSQL TCP5432 reachable, FV internet
|
|
HTTPS200, ANA->FV SSH reachable, gateway management intact. Beszel **18/18 up**.
|
|
Agent logs show WebSocket connected and intentional SSH listener shutdown;
|
|
45876 refusing after WebSocket connection is NOT a loss of host connectivity.
|
|
A single BMC ping to 10.251.50.50 did not answer; no prior BMC baseline was
|
|
taken, so BMC health is not claimed. The NAT rule excludes BMC source traffic.
|
|
|
|
## Backup and rollback
|
|
|
|
Root-only directory on gateway: `/root/fv-nat-repair-20260913/` contains
|
|
config.xml.before (secret-bearing, never commit), nat.before, filter.before,
|
|
nat-config.before.json, nat.after, nat-config.php and verified marker.
|
|
Automatic rollback disarmed only after verification; backup retained.
|
|
|
|
Manual rollback, after checking for intervening NAT edits:
|
|
|
|
```sh
|
|
php /root/fv-nat-repair-20260913/nat-config.php rollback
|
|
configctl filter reload
|
|
```
|
|
|
|
This restores the saved NAT config section, not unrelated configuration.
|
|
It will remove subsequent NAT edits too: inspect before using later.
|
|
Emergency runtime NAT-only restore: `pfctl -N -f /root/fv-nat-repair-20260913/nat.before`.
|
|
Never use pfctl -F all or flush all states for this repair.
|
|
|
|
Gateway SSH key auth is unavailable; use vaulted fv-gateway/root-password or
|
|
approved infra-ops credentials, without logging secrets. This session used
|
|
/tmp/fv-elway.py (FreeBSD sh + legacy SCP + existing SSH control socket).
|
|
|
|
References: [OPNsense hybrid NAT](https://docs.opnsense.org/manual/nat.html),
|
|
[Tailscale FreeBSD userspace routing](https://tailscale.com/docs/reference/kernel-vs-userspace-routers).
|