diff --git a/docs/pfi/ops-lessons-playbook.md b/docs/pfi/ops-lessons-playbook.md index 9f232f8..5f4a2b3 100644 --- a/docs/pfi/ops-lessons-playbook.md +++ b/docs/pfi/ops-lessons-playbook.md @@ -224,6 +224,53 @@ name the mechanism in the conclusion so the scope is visible to the next reader. --- +## 9. IPsec to a NAT'd site: dialup peer + NAT-T, and you cannot convert in place + +**Rule:** a site-to-site IPsec tunnel to any endpoint that might sit behind NAT +needs **`type dynamic`** (dialup responder) **and `nattraversal enable`**. Both. +Neither alone is sufficient. + +**Why:** ESH↔colo died the moment ESH stopped having a public IP. Two independent +causes, and the second was invisible until the first was investigated: + +| setting | broken tunnel | working tunnel | +|---|---|---| +| `type` | `static`, `remote-gw 70.181.90.232` (a dead address) | `ddns` | +| `nattraversal` | `disable` | `disable` — but NH3 is **publicly addressed**, so it never mattered | + +The static peer IP is the obvious failure. The subtle one is that **`nattraversal +disable` would have kept the tunnel down even with the correct peer IP**, because +ESP cannot traverse NAT without UDP-4500 encapsulation. A "just re-pin the IP" +fix would have failed and looked mysterious. + +⚠ **FortiOS refuses `set type dynamic` on an existing tunnel** — *"Cannot change +tunnel type once configured"*, with a clean rollback. So the fix is not an edit. + +**Prefer building the replacement ALONGSIDE the broken one, not recreating it.** +Deleting a phase1 cascades into its phase2, its static routes and every policy +referencing the interface — on the affected box that was 1 + 2 + 10 objects. +A new `phase1` + `phase2` + one route + two consolidated policies is additive, +leaves the old config intact as rollback, and cannot break what still works. + +**Confirming it worked** — the tunnel summary line says everything: + +``` +'ana-eshudm-dyn_0' 97.170.236.56:4500 selectors(total,up): 1/1 + ^^^ _0 = dialup child ^^^ carrier IP ^^^ :4500 = NAT-T +``` + +`_0` means the peer was accepted without being known in advance; `:4500` means +NAT-T is carrying ESP; the address is the carrier's, which could never have been +pinned. And traceroute drops from "8 hops wandering the carrier" to "gateway → +peer → destination". + +⚠ **Residual fragility on the UniFi end.** The UDM's `ipsec_local_ip` must hold a +literal address — `""` is rejected with `api.err.InvalidPayload` — so it still +needs updating whenever that site's WAN address changes. The gateway end is now +address-agnostic; the UniFi end is not. + +--- + ## Superseded claims | date | claim | correction | diff --git a/persistent-memory.md b/persistent-memory.md index fdb13de..df041e6 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -127,6 +127,8 @@ _As of 2026-08-18 — **gen seat SWAPPED to `absolute-heresy` and operator-confi - **EVIDENCE HOLD (partial):** WT #394 FILE half STILL STANDS — do NOT delete on-disk gen dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`); rex393-fiction-* + r42-gate-* KEEP. +- **🟢 ESH↔colo IPsec FIXED 2026-08-18 — dialup + NAT-T.** Broke when ESH lost its public IP (Cox unplugged for the fiber install; 5G handed out RFC1918 `192.168.200.111`). **TWO causes:** phase1 `ana-to-eshudm` was `type static` pinned to the dead `70.181.90.232`, AND **`nattraversal disable`** — so even re-pinning the IP would have failed, since ESP cannot cross NAT without UDP-4500. `pfi-ana-nh3` survived only because NH3 is publicly addressed. ⚠ **FortiOS REFUSES `set type dynamic` on an existing tunnel** ("Cannot change tunnel type once configured", clean rollback). **Fix = ADDITIVE, not a recreate:** new phase1+phase2 **`ana-eshudm-dyn`** (`type dynamic`, ikev2, aes256-sha1, dh14, NAT-T default on, PSK read from the ESH UDM API), static route id **10** (`dstaddr ana-to-eshudm_remote` → `device ana-eshudm-dyn`, distance 20), policies **73/74** (consolidated multi-zone in/out). Old `ana-to-eshudm` LEFT IN PLACE, dead and harmless, as rollback. Verified: `ana-eshudm-dyn_0 97.170.236.56:4500 selectors 1/1` (`_0`=dialup child, `:4500`=NAT-T, carrier IP), ESH→colo all hosts 40-56ms, colo→ESH OK, traceroute now 3 hops encapsulated vs 8 leaking to the carrier. FortiGate config backed up first at `nh3-dev:~/backups/ana-gw/` (1.17MB, 36903 lines). ⚠ **RESIDUAL: the UDM's `ipsec_local_ip` needs a LITERAL IP** (empty rejected `api.err.InvalidPayload`) — currently `192.168.200.111`, so **it must be updated again when the fiber lands** or the tunnel drops. Gateway end is address-agnostic now; UniFi end is not. + - **🟡 FLEET IPv6 — STILL JUSTIFIED, but the REASON was wrong and is now corrected (2026-08-18).** There are TWO inter-site links with OPPOSITE NAT behaviour, and the original note conflated them. **(a) NH3↔ESH = Site Magic = WireGuard — SURVIVES arbitrary NAT, PROVEN.** During the fiber cutover ESH failed over to 5G on `192.168.200.111` (**RFC1918 double-NAT, zero inbound path, strictly worse than CGNAT**) and it held: nh3-dev/nh3-docker reachable ~40-43ms, ssh+exec fine. Mechanism: `magic_site_to_site_vpn` on-device is only `enabled`+keypair (**no WAN binding — it rides whichever uplink is active**), and **NH3's edge is public at `70.230.226.88`**, so the NAT'd side dials out and never needs reachability. **(b) colo(Anaheim)↔ESH = IPsec on the ana-gw FortiGate — BROKEN RIGHT NOW under those same conditions.** ana-docker/pfi-pve/pbs-ana ALL FAIL from esh-pve-nas; traceroute shows 10.250.x packets leaving the UDM to the 5G modem `192.168.200.99`, then wandering the carrier (`10.180.141.194`, `172.19.3.14`) and dying — **not encapsulated at all**, so no SA is up and traffic falls to the default route. Site-to-site IPsec pins a peer IP; ESH no longer has a routable one. **⇒ IPv6 keeps its justification, on the IPsec link SPECIFICALLY.** ⚠ An earlier version of this entry claimed the premise was refuted outright — that over-generalised a WireGuard result onto an IPsec link. **Operator caught it.** See ops-lessons-playbook §8. State: **NH3 WAN live** (`2600:1700:b25:c110::48`, AT&T delegates exactly **one /64**), **colo none** (FortiGate has zero v6), **ESH none** (both WANs `wan_type_v6=disabled`). NH3 LANs all reverted to `ipv6_interface_type=none` per operator. Work when addresses land: v6 on `ana-wg` eth0 + a **v6 port-forward for UDP 31337** on the FortiGate (its WG socket is already dual-stack — no WG reconfig), flip the UDM WG server off `v4`-pinned binding, and **AAAA records** so the dynamic prefixes at all three sites don't break endpoints. Full detail + access recipes → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`. - **🟢 WT #401 (fd-leak deadlock) CLOSED 2026-08-17 — one ping still owed.** worldtree-dev closed it on our demo verify. Layers: **(a) their `e41b139`** pins `ulimits: nofile 65536/65536` in the worldtree compose anchor — **demo VERIFIED** (api + matrix recreated 22:55:34Z, `ulimit -Sn`=65536); **personal/pinned are covered-not-verified**, they inherit at their next promotion/recreate. **(b) our host floor is STAGED, NOT ACTIVE** — `/etc/docker/daemon.json` on corviduo-dev carries `default-ulimits nofile 65536/65536` but **`default-ulimits` is NOT SIGHUP-reloadable** (measured on 29.4.3: post-reload the daemon's own "Reloaded configuration" log omits it and a fresh container still reports 1024). Activation needs a full dockerd restart = bounces all 13 containers; **worldtree-dev explicitly does NOT want one**, and `live-restore:true`-then-restart is PARKED as a separate host-side improvement for the operator to rule on, never folded into #401. Playbook `playbooks/corviduo-dev-docker-default-ulimits.yaml` (verify step 3 fails BY DESIGN until a restart). Hourly fd tripwire on corviduo-dev stays armed. **⏳ OWED: ping worldtree-dev in thread `01M08QQ655XD6VKEV7MA9GX0NS` once worldtree-personal recreates and 65536 is confirmed there.** Commit `7f3f265`.