vh
838132cd6b
Session captured: the FV outbound-NAT root cause and its diagnostic signature, the fv-ml1 dead man's switch, fleet identity/group/path conventions and the root:docker normalization, nh3-dev's ts-input reachability fix, ESPHome modernisation and the kb KB-search tool, and the Hermes bearer rotation release. Six new detail files. Tried-and-abandoned gains three: probing OPNsense endpoints by POSTing at them (which rebooted the FV firewall), advertising a /32 from nh3-dev, and the nh3-scale remote-site masquerade rules that fired but were not the fix. Housekeeping: 8 Recent-decisions entries archived to archival-memory.md, and 21 oversized inline entries split into detail files per the two-tier rule -- they had been sitting fully inline in the index, which is what the split exists to prevent. Two pointers to a detail file archived this run were repointed at archival-memory.md. The index is 389 lines, still over the ~300 soft cap. The archival guards stop it there: only 4 further entries are old enough to move and every one carries an open deferred-work pointer. An over-cap file that keeps live decisions beats a scannable one that lost a deferred call.
72 lines
3.6 KiB
Markdown
72 lines
3.6 KiB
Markdown
# `[2026-09-15]` FV cross-site routing fixed — one NAT rule scoped to Anaheim only
|
|
|
|
fv-ml1 could reach Anaheim and the internet but **nothing else** — not NH3, not
|
|
ESH, not Irvine. Mesh addresses (`100.64.0.x`) worked perfectly from it; LAN
|
|
addresses did not. That shape reads as a routing or Tailscale fault and is
|
|
neither.
|
|
|
|
## Root cause
|
|
|
|
One outbound-NAT rule on the FV OPNsense gateway, added 2026-09-13 and scoped to
|
|
a single destination. `docs/runbooks/fv-to-ana-nat.md` says so in as many words:
|
|
|
|
Interface: MESH (opt6 / tailscale0)
|
|
Source: 10.251.50.54/32 (fv-ml1 only)
|
|
Destination: 10.250.0.0/16 (Anaheim only)
|
|
"Other remote sites remain outside this fix's scope."
|
|
|
|
FV→Anaheim worked because a rule existed for it. FV→everywhere else failed
|
|
because none did. The runbook's own "Before" section describes the exact
|
|
symptom — far site receives with `src=10.251.50.54`, replies never complete.
|
|
|
|
## Fix
|
|
|
|
Three mirrors added (NH3 `10.100.0.0/16`, ESH `10.0.0.0/16`, Irvine
|
|
`10.6.110.0/24`), then all four broadened from fv-ml1's `/32` to the FV LAN
|
|
`10.251.50.0/24`, with descriptions rewritten to name the real scope. Applied
|
|
via `POST /api/firewall/source_nat/add_rule` + `set_rule` + `apply`, pre-change
|
|
`core/backup/download/this` taken each time. Commits `fa04f45`, `0ab9da5`.
|
|
|
|
⚠ Anaheim's original rule was written with `write_config` and is **invisible to
|
|
`source_nat/search_rule`** — the API cannot see or manage it. An API-managed
|
|
ANA `/24` rule was added alongside so all four destinations sit on the same code
|
|
path; the legacy `/32` is now redundant, harmless, and wants deleting from the
|
|
UI.
|
|
|
|
## ⭐ The diagnostic signature, so the next person skips the evening
|
|
|
|
Every one of these is true while the fault is live, and each one argues *against*
|
|
NAT being the cause:
|
|
|
|
- fv-ml1 reaches mesh addresses perfectly and LAN addresses not at all.
|
|
- The FV firewall log shows the outbound **passing** on tailscale0 with
|
|
`src=10.251.50.54` and nothing ever returning — nothing looks blocked.
|
|
- The far-side router genuinely receives and replies — proven with temporary
|
|
counting rules on nh3-scale: **5 packets in, 4 replies out**.
|
|
- Both peers' Tailscale `AllowedIPs` are correct, so cryptokey routing is fine.
|
|
- `ts-forward` on nh3-scale accepts everything from tailscale0; its DROP rule
|
|
shows **0 packets**.
|
|
|
|
⭐ **The discriminator that settles it: every OTHER site pair works.**
|
|
`nh3-docker → esh/ana/FV` and `esh-docker-vm → FV` all succeed, which rules out a
|
|
general subnet-to-subnet limitation and leaves outbound SNAT as the only
|
|
candidate. Check `/api/firewall/source_nat/search_rule` for a rule covering the
|
|
destination **before** investigating anything else.
|
|
|
|
## Wrong turns worth not repeating
|
|
|
|
- **Advertising `10.100.10.50/32` from nh3-dev** to make its LAN address
|
|
mesh-reachable — black-holed nh3-dev from ESH, Anaheim, FV and Irvine while
|
|
leaving its own LAN and the internet up. `ip rule` there puts `lookup 52` at
|
|
priority 5270 ahead of `main` at 32766, so becoming a subnet router let table
|
|
52 capture cross-site traffic a `RouteAll: false` node has no accepted route
|
|
for. Reverted; the working fix is a masquerade exception on nh3-scale
|
|
(`9dbd829`). See [[2026-09-15-nh3-dev-ts-input-masquerade]].
|
|
- **Remote-site MASQUERADE rules on nh3-scale** for the asymmetric-return
|
|
theory. They fired (counters incremented) but were not the fix; reverted
|
|
rather than left to accumulate.
|
|
- **`acceptSubnetRoutes` 0→1 on the FV gateway** — real and kept: the gateway
|
|
itself could not reach NH3/ESH before it. Necessary, not sufficient.
|
|
|
|
Related: [[2026-09-15-fv-mesh-watchdog]], [[2026-09-15-opnsense-api-reboot]].
|