fix(ana-gw): restore ESH<->colo IPsec as a dialup tunnel with NAT-T

The link died when ESH lost its public IP during the fiber cutover. Two
independent causes, and the second would have defeated the obvious fix:

- phase1 ana-to-eshudm was type static, pinned to 70.181.90.232, an
  address that no longer exists.
- nattraversal was disable, so ESP could not have crossed NAT even with
  the peer IP corrected. pfi-ana-nh3 shares that setting and survives
  only because NH3 is publicly addressed, which is why the two tunnels
  diverged.

FortiOS refuses `set type dynamic` on an existing tunnel -- "Cannot
change tunnel type once configured" -- and rolled back cleanly, so the
fix could not be an edit. Rather than delete and recreate, which
cascades into the phase2, two static routes and ten policies, the
replacement was built alongside: new phase1+phase2 ana-eshudm-dyn
(type dynamic, ikev2, aes256-sha1, dh14, NAT-T on, PSK read from the ESH
UDM API so neither side needed a new key), static route id 10 at
distance 20, and two consolidated multi-zone policies 73/74. The old
tunnel is left in place, dead and harmless, as rollback.

Verified up: ana-eshudm-dyn_0 97.170.236.56:4500 selectors 1/1 -- the _0
suffix is a dialup child, :4500 is NAT-T, and the address is the
carrier's, which is precisely what could never have been pinned. ESH
reaches all four colo hosts at 40-56ms, the colo reaches all three ESH
hosts, and traceroute drops from eight hops leaking into the carrier
network to three hops fully encapsulated.

Config was backed up before any write (1.17MB, 36903 lines, off-box).

Residual fragility recorded: the UDM's ipsec_local_ip demands a literal
address -- empty is rejected as api.err.InvalidPayload -- so it still
needs updating when the fiber changes ESH's WAN address. The gateway end
is now address-agnostic; the UniFi end is not.

docs/pfi/ops-lessons-playbook.md

@@ -224,6 +224,53 @@ name the mechanism in the conclusion so the scope is visible to the next reader.
 
 ---
 
+## 9. IPsec to a NAT'd site: dialup peer + NAT-T, and you cannot convert in place
+
+**Rule:** a site-to-site IPsec tunnel to any endpoint that might sit behind NAT
+needs **`type dynamic`** (dialup responder) **and `nattraversal enable`**. Both.
+Neither alone is sufficient.
+
+**Why:** ESH↔colo died the moment ESH stopped having a public IP. Two independent
+causes, and the second was invisible until the first was investigated:
+
+| setting | broken tunnel | working tunnel |
+|---|---|---|
+| `type` | `static`, `remote-gw 70.181.90.232` (a dead address) | `ddns` |
+| `nattraversal` | `disable` | `disable` — but NH3 is **publicly addressed**, so it never mattered |
+
+The static peer IP is the obvious failure. The subtle one is that **`nattraversal
+disable` would have kept the tunnel down even with the correct peer IP**, because
+ESP cannot traverse NAT without UDP-4500 encapsulation. A "just re-pin the IP"
+fix would have failed and looked mysterious.
+
+⚠ **FortiOS refuses `set type dynamic` on an existing tunnel** — *"Cannot change
+tunnel type once configured"*, with a clean rollback. So the fix is not an edit.
+
+**Prefer building the replacement ALONGSIDE the broken one, not recreating it.**
+Deleting a phase1 cascades into its phase2, its static routes and every policy
+referencing the interface — on the affected box that was 1 + 2 + 10 objects.
+A new `phase1` + `phase2` + one route + two consolidated policies is additive,
+leaves the old config intact as rollback, and cannot break what still works.
+
+**Confirming it worked** — the tunnel summary line says everything:
+
+```
+'ana-eshudm-dyn_0'  97.170.236.56:4500  selectors(total,up): 1/1
+ ^^^ _0 = dialup child      ^^^ carrier IP    ^^^ :4500 = NAT-T
+```
+
+`_0` means the peer was accepted without being known in advance; `:4500` means
+NAT-T is carrying ESP; the address is the carrier's, which could never have been
+pinned. And traceroute drops from "8 hops wandering the carrier" to "gateway →
+peer → destination".
+
+⚠ **Residual fragility on the UniFi end.** The UDM's `ipsec_local_ip` must hold a
+literal address — `""` is rejected with `api.err.InvalidPayload` — so it still
+needs updating whenever that site's WAN address changes. The gateway end is now
+address-agnostic; the UniFi end is not.
+
+---
+
 ## Superseded claims
 
 | date | claim | correction |