# `[2026-09-15]` FV cross-site routing fixed — one NAT rule scoped to Anaheim only

fv-ml1 could reach Anaheim and the internet but **nothing else** — not NH3, not
ESH, not Irvine. Mesh addresses (`100.64.0.x`) worked perfectly from it; LAN
addresses did not. That shape reads as a routing or Tailscale fault and is
neither.

## Root cause

One outbound-NAT rule on the FV OPNsense gateway, added 2026-09-13 and scoped to
a single destination. `docs/runbooks/fv-to-ana-nat.md` says so in as many words:

    Interface: MESH (opt6 / tailscale0)
    Source: 10.251.50.54/32 (fv-ml1 only)
    Destination: 10.250.0.0/16 (Anaheim only)
    "Other remote sites remain outside this fix's scope."

FV→Anaheim worked because a rule existed for it. FV→everywhere else failed
because none did. The runbook's own "Before" section describes the exact
symptom — far site receives with `src=10.251.50.54`, replies never complete.

## Fix

Three mirrors added (NH3 `10.100.0.0/16`, ESH `10.0.0.0/16`, Irvine
`10.6.110.0/24`), then all four broadened from fv-ml1's `/32` to the FV LAN
`10.251.50.0/24`, with descriptions rewritten to name the real scope. Applied
via `POST /api/firewall/source_nat/add_rule` + `set_rule` + `apply`, pre-change
`core/backup/download/this` taken each time. Commits `fa04f45`, `0ab9da5`.

⚠ Anaheim's original rule was written with `write_config` and is **invisible to
`source_nat/search_rule`** — the API cannot see or manage it. An API-managed
ANA `/24` rule was added alongside so all four destinations sit on the same code
path; the legacy `/32` is now redundant, harmless, and wants deleting from the
UI.

## ⭐ The diagnostic signature, so the next person skips the evening

Every one of these is true while the fault is live, and each one argues *against*
NAT being the cause:

- fv-ml1 reaches mesh addresses perfectly and LAN addresses not at all.
- The FV firewall log shows the outbound **passing** on tailscale0 with
  `src=10.251.50.54` and nothing ever returning — nothing looks blocked.
- The far-side router genuinely receives and replies — proven with temporary
  counting rules on nh3-scale: **5 packets in, 4 replies out**.
- Both peers' Tailscale `AllowedIPs` are correct, so cryptokey routing is fine.
- `ts-forward` on nh3-scale accepts everything from tailscale0; its DROP rule
  shows **0 packets**.

⭐ **The discriminator that settles it: every OTHER site pair works.**
`nh3-docker → esh/ana/FV` and `esh-docker-vm → FV` all succeed, which rules out a
general subnet-to-subnet limitation and leaves outbound SNAT as the only
candidate. Check `/api/firewall/source_nat/search_rule` for a rule covering the
destination **before** investigating anything else.

## Wrong turns worth not repeating

- **Advertising `10.100.10.50/32` from nh3-dev** to make its LAN address
  mesh-reachable — black-holed nh3-dev from ESH, Anaheim, FV and Irvine while
  leaving its own LAN and the internet up. `ip rule` there puts `lookup 52` at
  priority 5270 ahead of `main` at 32766, so becoming a subnet router let table
  52 capture cross-site traffic a `RouteAll: false` node has no accepted route
  for. Reverted; the working fix is a masquerade exception on nh3-scale
  (`9dbd829`). See [[2026-09-15-nh3-dev-ts-input-masquerade]].
- **Remote-site MASQUERADE rules on nh3-scale** for the asymmetric-return
  theory. They fired (counters incremented) but were not the fix; reverted
  rather than left to accumulate.
- **`acceptSubnetRoutes` 0→1 on the FV gateway** — real and kept: the gateway
  itself could not reach NH3/ESH before it. Necessary, not sufficient.

Related: [[2026-09-15-fv-mesh-watchdog]], [[2026-09-15-opnsense-api-reboot]].