vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(fv): extend the Tailscale SNAT rule to NH3, ESH and Irvine

Browse Source 
FV could not reach any site but Anaheim. The cause was a single outbound-NAT
rule on the FV gateway, added 2026-09-13 and scoped to Anaheim only --
docs/runbooks/fv-to-ana-nat.md says so in as many words: "Other remote sites
remain outside this fix's scope." Three mirrors added, same interface and
source, only the destination differing: 10.100.0.0/16, 10.0.0.0/16 and
10.6.110.0/24.

After: fv-ml1 reaches NH3, ESH, Anaheim, Irvine, the mesh and the internet.
Regression sweep clean across nh3-dev, nh3-docker and esh-docker-vm.

The runbook now records what the failure looks like, because it presents as a
routing or Tailscale fault and is neither. fv-ml1 reached mesh addresses
perfectly and LAN addresses not at all; the FV firewall log showed the outbound
passing with src=10.251.50.54 and no reply returning; temporary counting rules
proved nh3-scale received 5 packets and sent 4 replies; both peers' AllowedIPs
were correct. The discriminator that settles it is that every other site pair
works -- nh3-docker to esh/ana/FV and esh-docker-vm to FV all succeed -- so a
general subnet-to-subnet limitation is ruled out and only outbound SNAT is
left.

Also reverts the remote-site MASQUERADE rules added to nh3-scale earlier on the
asymmetric-return theory. They fired but were not the fix, so they are removed
rather than left to accumulate as NAT that achieves nothing.

Applied via source_nat/add_rule + apply with a pre-change config backup taken
first. Source scope is still fv-ml1's /32, so a second FV host will hit this
again -- flagged in the runbook.
This commit is contained in:
vh
2026-09-15 00:40:28 -07:00
parent 8c8559b8ec
commit fa04f450fb
2 changed files with 46 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/runbooks/fv-to-ana-nat.md
+46 -1
View File
@@ -17,7 +17,52 @@ One manual IPv4 rule:
No routes, access/filter rules, WAN settings, subnet advertisements, host
Tailscale settings, or other FV source addresses were changed. Other remote
sites remain outside this fix's scope. Translation means Anaheim services may
sites remain outside this fix's scope.
## ⭐ EXTENDED 2026-09-15 — NH3, ESH and Irvine now have the same rule
The Anaheim-only scope above was the whole reason FV could not reach anywhere
else, and it cost an evening to rediscover. Three rules added, each an exact
mirror of the ANA one — same interface, same source, only the destination
differs:
| destination | description |
|---|---|
| `10.100.0.0/16` | `fv-ml1 to NH3 via Tailscale SNAT` |
| `10.0.0.0/16` | `fv-ml1 to ESH via Tailscale SNAT` |
| `10.6.110.0/24` | `fv-ml1 to IRV via Tailscale SNAT` |
Applied through `POST /api/firewall/source_nat/add_rule` + `.../apply`, with a
pre-change `core/backup/download/this` taken first. After: fv-ml1 reaches all
four sites, the mesh and the internet; nh3-dev, nh3-docker and esh-docker-vm
regression-swept clean.
⚠ **The source is still `10.251.50.54/32` — fv-ml1 only.** A second host at FV
will have exactly this problem and no symptom that points at NAT. Broaden to
`10.251.50.0/24`, or add a rule per host, when FV grows.
### What this looks like when it breaks, so the next person skips the evening
The failure presents as a routing or Tailscale problem and is neither:
- fv-ml1 reaches mesh addresses (`100.64.0.x`) perfectly and LAN addresses not
at all — which reads like a subnet-router fault.
- The FV firewall log shows the outbound **passing** on tailscale0 with
`src=10.251.50.54` and no reply ever returning, so nothing looks blocked.
- The far-side router genuinely receives and replies — verified with temporary
counting rules on nh3-scale: 5 packets in, 4 replies out.
- Both peers' Tailscale `AllowedIPs` are correct, so cryptokey routing is fine.
- **Every other site pair works**, which rules out a general subnet-to-subnet
limitation: `nh3-docker → esh/ana/FV` and `esh-docker-vm → FV` all succeed.
That combination means outbound SNAT, not routing. Check
`/api/firewall/source_nat/search_rule` for a rule covering the destination
before investigating anything else.
⚠ Also retired on the way: a set of remote-site MASQUERADE rules added to
nh3-scale's `mesh-exit-masq.sh` on the asymmetric-return theory. They fired
(counters incremented) but were not the fix, and were reverted rather than left
to accumulate. Translation means Anaheim services may
see the gateway identity rather than fv-ml1's original LAN address.
## Evidence and verification