From 8a742f59b88c1f78b206d608533cdb929862065e Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 18 Aug 2026 08:12:35 -0700 Subject: [PATCH] fix(ana-gw): restore ESH<->colo IPsec as a dialup tunnel with NAT-T The link died when ESH lost its public IP during the fiber cutover. Two independent causes, and the second would have defeated the obvious fix: - phase1 ana-to-eshudm was type static, pinned to 70.181.90.232, an address that no longer exists. - nattraversal was disable, so ESP could not have crossed NAT even with the peer IP corrected. pfi-ana-nh3 shares that setting and survives only because NH3 is publicly addressed, which is why the two tunnels diverged. FortiOS refuses `set type dynamic` on an existing tunnel -- "Cannot change tunnel type once configured" -- and rolled back cleanly, so the fix could not be an edit. Rather than delete and recreate, which cascades into the phase2, two static routes and ten policies, the replacement was built alongside: new phase1+phase2 ana-eshudm-dyn (type dynamic, ikev2, aes256-sha1, dh14, NAT-T on, PSK read from the ESH UDM API so neither side needed a new key), static route id 10 at distance 20, and two consolidated multi-zone policies 73/74. The old tunnel is left in place, dead and harmless, as rollback. Verified up: ana-eshudm-dyn_0 97.170.236.56:4500 selectors 1/1 -- the _0 suffix is a dialup child, :4500 is NAT-T, and the address is the carrier's, which is precisely what could never have been pinned. ESH reaches all four colo hosts at 40-56ms, the colo reaches all three ESH hosts, and traceroute drops from eight hops leaking into the carrier network to three hops fully encapsulated. Config was backed up before any write (1.17MB, 36903 lines, off-box). Residual fragility recorded: the UDM's ipsec_local_ip demands a literal address -- empty is rejected as api.err.InvalidPayload -- so it still needs updating when the fiber changes ESH's WAN address. The gateway end is now address-agnostic; the UniFi end is not. --- docs/pfi/ops-lessons-playbook.md | 47 ++++++++++++++++++++++++++++++++ persistent-memory.md | 2 ++ 2 files changed, 49 insertions(+) diff --git a/docs/pfi/ops-lessons-playbook.md b/docs/pfi/ops-lessons-playbook.md index 9f232f8..5f4a2b3 100644 --- a/docs/pfi/ops-lessons-playbook.md +++ b/docs/pfi/ops-lessons-playbook.md @@ -224,6 +224,53 @@ name the mechanism in the conclusion so the scope is visible to the next reader. --- +## 9. IPsec to a NAT'd site: dialup peer + NAT-T, and you cannot convert in place + +**Rule:** a site-to-site IPsec tunnel to any endpoint that might sit behind NAT +needs **`type dynamic`** (dialup responder) **and `nattraversal enable`**. Both. +Neither alone is sufficient. + +**Why:** ESH↔colo died the moment ESH stopped having a public IP. Two independent +causes, and the second was invisible until the first was investigated: + +| setting | broken tunnel | working tunnel | +|---|---|---| +| `type` | `static`, `remote-gw 70.181.90.232` (a dead address) | `ddns` | +| `nattraversal` | `disable` | `disable` — but NH3 is **publicly addressed**, so it never mattered | + +The static peer IP is the obvious failure. The subtle one is that **`nattraversal +disable` would have kept the tunnel down even with the correct peer IP**, because +ESP cannot traverse NAT without UDP-4500 encapsulation. A "just re-pin the IP" +fix would have failed and looked mysterious. + +⚠ **FortiOS refuses `set type dynamic` on an existing tunnel** — *"Cannot change +tunnel type once configured"*, with a clean rollback. So the fix is not an edit. + +**Prefer building the replacement ALONGSIDE the broken one, not recreating it.** +Deleting a phase1 cascades into its phase2, its static routes and every policy +referencing the interface — on the affected box that was 1 + 2 + 10 objects. +A new `phase1` + `phase2` + one route + two consolidated policies is additive, +leaves the old config intact as rollback, and cannot break what still works. + +**Confirming it worked** — the tunnel summary line says everything: + +``` +'ana-eshudm-dyn_0' 97.170.236.56:4500 selectors(total,up): 1/1 + ^^^ _0 = dialup child ^^^ carrier IP ^^^ :4500 = NAT-T +``` + +`_0` means the peer was accepted without being known in advance; `:4500` means +NAT-T is carrying ESP; the address is the carrier's, which could never have been +pinned. And traceroute drops from "8 hops wandering the carrier" to "gateway → +peer → destination". + +⚠ **Residual fragility on the UniFi end.** The UDM's `ipsec_local_ip` must hold a +literal address — `""` is rejected with `api.err.InvalidPayload` — so it still +needs updating whenever that site's WAN address changes. The gateway end is now +address-agnostic; the UniFi end is not. + +--- + ## Superseded claims | date | claim | correction | diff --git a/persistent-memory.md b/persistent-memory.md index fdb13de..df041e6 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -127,6 +127,8 @@ _As of 2026-08-18 — **gen seat SWAPPED to `absolute-heresy` and operator-confi - **EVIDENCE HOLD (partial):** WT #394 FILE half STILL STANDS — do NOT delete on-disk gen dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`); rex393-fiction-* + r42-gate-* KEEP. +- **🟢 ESH↔colo IPsec FIXED 2026-08-18 — dialup + NAT-T.** Broke when ESH lost its public IP (Cox unplugged for the fiber install; 5G handed out RFC1918 `192.168.200.111`). **TWO causes:** phase1 `ana-to-eshudm` was `type static` pinned to the dead `70.181.90.232`, AND **`nattraversal disable`** — so even re-pinning the IP would have failed, since ESP cannot cross NAT without UDP-4500. `pfi-ana-nh3` survived only because NH3 is publicly addressed. ⚠ **FortiOS REFUSES `set type dynamic` on an existing tunnel** ("Cannot change tunnel type once configured", clean rollback). **Fix = ADDITIVE, not a recreate:** new phase1+phase2 **`ana-eshudm-dyn`** (`type dynamic`, ikev2, aes256-sha1, dh14, NAT-T default on, PSK read from the ESH UDM API), static route id **10** (`dstaddr ana-to-eshudm_remote` → `device ana-eshudm-dyn`, distance 20), policies **73/74** (consolidated multi-zone in/out). Old `ana-to-eshudm` LEFT IN PLACE, dead and harmless, as rollback. Verified: `ana-eshudm-dyn_0 97.170.236.56:4500 selectors 1/1` (`_0`=dialup child, `:4500`=NAT-T, carrier IP), ESH→colo all hosts 40-56ms, colo→ESH OK, traceroute now 3 hops encapsulated vs 8 leaking to the carrier. FortiGate config backed up first at `nh3-dev:~/backups/ana-gw/` (1.17MB, 36903 lines). ⚠ **RESIDUAL: the UDM's `ipsec_local_ip` needs a LITERAL IP** (empty rejected `api.err.InvalidPayload`) — currently `192.168.200.111`, so **it must be updated again when the fiber lands** or the tunnel drops. Gateway end is address-agnostic now; UniFi end is not. + - **🟡 FLEET IPv6 — STILL JUSTIFIED, but the REASON was wrong and is now corrected (2026-08-18).** There are TWO inter-site links with OPPOSITE NAT behaviour, and the original note conflated them. **(a) NH3↔ESH = Site Magic = WireGuard — SURVIVES arbitrary NAT, PROVEN.** During the fiber cutover ESH failed over to 5G on `192.168.200.111` (**RFC1918 double-NAT, zero inbound path, strictly worse than CGNAT**) and it held: nh3-dev/nh3-docker reachable ~40-43ms, ssh+exec fine. Mechanism: `magic_site_to_site_vpn` on-device is only `enabled`+keypair (**no WAN binding — it rides whichever uplink is active**), and **NH3's edge is public at `70.230.226.88`**, so the NAT'd side dials out and never needs reachability. **(b) colo(Anaheim)↔ESH = IPsec on the ana-gw FortiGate — BROKEN RIGHT NOW under those same conditions.** ana-docker/pfi-pve/pbs-ana ALL FAIL from esh-pve-nas; traceroute shows 10.250.x packets leaving the UDM to the 5G modem `192.168.200.99`, then wandering the carrier (`10.180.141.194`, `172.19.3.14`) and dying — **not encapsulated at all**, so no SA is up and traffic falls to the default route. Site-to-site IPsec pins a peer IP; ESH no longer has a routable one. **⇒ IPv6 keeps its justification, on the IPsec link SPECIFICALLY.** ⚠ An earlier version of this entry claimed the premise was refuted outright — that over-generalised a WireGuard result onto an IPsec link. **Operator caught it.** See ops-lessons-playbook §8. State: **NH3 WAN live** (`2600:1700:b25:c110::48`, AT&T delegates exactly **one /64**), **colo none** (FortiGate has zero v6), **ESH none** (both WANs `wan_type_v6=disabled`). NH3 LANs all reverted to `ipv6_interface_type=none` per operator. Work when addresses land: v6 on `ana-wg` eth0 + a **v6 port-forward for UDP 31337** on the FortiGate (its WG socket is already dual-stack — no WG reconfig), flip the UDM WG server off `v4`-pinned binding, and **AAAA records** so the dynamic prefixes at all three sites don't break endpoints. Full detail + access recipes → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`. - **🟢 WT #401 (fd-leak deadlock) CLOSED 2026-08-17 — one ping still owed.** worldtree-dev closed it on our demo verify. Layers: **(a) their `e41b139`** pins `ulimits: nofile 65536/65536` in the worldtree compose anchor — **demo VERIFIED** (api + matrix recreated 22:55:34Z, `ulimit -Sn`=65536); **personal/pinned are covered-not-verified**, they inherit at their next promotion/recreate. **(b) our host floor is STAGED, NOT ACTIVE** — `/etc/docker/daemon.json` on corviduo-dev carries `default-ulimits nofile 65536/65536` but **`default-ulimits` is NOT SIGHUP-reloadable** (measured on 29.4.3: post-reload the daemon's own "Reloaded configuration" log omits it and a fresh container still reports 1024). Activation needs a full dockerd restart = bounces all 13 containers; **worldtree-dev explicitly does NOT want one**, and `live-restore:true`-then-restart is PARKED as a separate host-side improvement for the operator to rule on, never folded into #401. Playbook `playbooks/corviduo-dev-docker-default-ulimits.yaml` (verify step 3 fails BY DESIGN until a restart). Hourly fd tripwire on corviduo-dev stays armed. **⏳ OWED: ping worldtree-dev in thread `01M08QQ655XD6VKEV7MA9GX0NS` once worldtree-personal recreates and 65536 is confirmed there.** Commit `7f3f265`.