vh
838132cd6b
Session captured: the FV outbound-NAT root cause and its diagnostic signature, the fv-ml1 dead man's switch, fleet identity/group/path conventions and the root:docker normalization, nh3-dev's ts-input reachability fix, ESPHome modernisation and the kb KB-search tool, and the Hermes bearer rotation release. Six new detail files. Tried-and-abandoned gains three: probing OPNsense endpoints by POSTing at them (which rebooted the FV firewall), advertising a /32 from nh3-dev, and the nh3-scale remote-site masquerade rules that fired but were not the fix. Housekeeping: 8 Recent-decisions entries archived to archival-memory.md, and 21 oversized inline entries split into detail files per the two-tier rule -- they had been sitting fully inline in the index, which is what the split exists to prevent. Two pointers to a detail file archived this run were repointed at archival-memory.md. The index is 389 lines, still over the ~300 soft cap. The archival guards stop it there: only 4 further entries are old enough to move and every one carries an open deferred-work pointer. An over-cap file that keeps live decisions beats a scannable one that lost a deferred call.
3.6 KiB
[2026-09-15] FV cross-site routing fixed — one NAT rule scoped to Anaheim only
fv-ml1 could reach Anaheim and the internet but nothing else — not NH3, not
ESH, not Irvine. Mesh addresses (100.64.0.x) worked perfectly from it; LAN
addresses did not. That shape reads as a routing or Tailscale fault and is
neither.
Root cause
One outbound-NAT rule on the FV OPNsense gateway, added 2026-09-13 and scoped to
a single destination. docs/runbooks/fv-to-ana-nat.md says so in as many words:
Interface: MESH (opt6 / tailscale0)
Source: 10.251.50.54/32 (fv-ml1 only)
Destination: 10.250.0.0/16 (Anaheim only)
"Other remote sites remain outside this fix's scope."
FV→Anaheim worked because a rule existed for it. FV→everywhere else failed
because none did. The runbook's own "Before" section describes the exact
symptom — far site receives with src=10.251.50.54, replies never complete.
Fix
Three mirrors added (NH3 10.100.0.0/16, ESH 10.0.0.0/16, Irvine
10.6.110.0/24), then all four broadened from fv-ml1's /32 to the FV LAN
10.251.50.0/24, with descriptions rewritten to name the real scope. Applied
via POST /api/firewall/source_nat/add_rule + set_rule + apply, pre-change
core/backup/download/this taken each time. Commits fa04f45, 0ab9da5.
⚠ Anaheim's original rule was written with write_config and is invisible to
source_nat/search_rule — the API cannot see or manage it. An API-managed
ANA /24 rule was added alongside so all four destinations sit on the same code
path; the legacy /32 is now redundant, harmless, and wants deleting from the
UI.
⭐ The diagnostic signature, so the next person skips the evening
Every one of these is true while the fault is live, and each one argues against NAT being the cause:
- fv-ml1 reaches mesh addresses perfectly and LAN addresses not at all.
- The FV firewall log shows the outbound passing on tailscale0 with
src=10.251.50.54and nothing ever returning — nothing looks blocked. - The far-side router genuinely receives and replies — proven with temporary counting rules on nh3-scale: 5 packets in, 4 replies out.
- Both peers' Tailscale
AllowedIPsare correct, so cryptokey routing is fine. ts-forwardon nh3-scale accepts everything from tailscale0; its DROP rule shows 0 packets.
⭐ The discriminator that settles it: every OTHER site pair works.
nh3-docker → esh/ana/FV and esh-docker-vm → FV all succeed, which rules out a
general subnet-to-subnet limitation and leaves outbound SNAT as the only
candidate. Check /api/firewall/source_nat/search_rule for a rule covering the
destination before investigating anything else.
Wrong turns worth not repeating
- Advertising
10.100.10.50/32from nh3-dev to make its LAN address mesh-reachable — black-holed nh3-dev from ESH, Anaheim, FV and Irvine while leaving its own LAN and the internet up.ip rulethere putslookup 52at priority 5270 ahead ofmainat 32766, so becoming a subnet router let table 52 capture cross-site traffic aRouteAll: falsenode has no accepted route for. Reverted; the working fix is a masquerade exception on nh3-scale (9dbd829). See 2026-09-15-nh3-dev-ts-input-masquerade. - Remote-site MASQUERADE rules on nh3-scale for the asymmetric-return theory. They fired (counters incremented) but were not the fix; reverted rather than left to accumulate.
acceptSubnetRoutes0→1 on the FV gateway — real and kept: the gateway itself could not reach NH3/ESH before it. Necessary, not sufficient.
Related: 2026-09-15-fv-mesh-watchdog, 2026-09-15-opnsense-api-reboot.