fix(ana-gw): restore ESH<->colo IPsec as a dialup tunnel with NAT-T

The link died when ESH lost its public IP during the fiber cutover. Two
independent causes, and the second would have defeated the obvious fix:

- phase1 ana-to-eshudm was type static, pinned to 70.181.90.232, an
  address that no longer exists.
- nattraversal was disable, so ESP could not have crossed NAT even with
  the peer IP corrected. pfi-ana-nh3 shares that setting and survives
  only because NH3 is publicly addressed, which is why the two tunnels
  diverged.

FortiOS refuses `set type dynamic` on an existing tunnel -- "Cannot
change tunnel type once configured" -- and rolled back cleanly, so the
fix could not be an edit. Rather than delete and recreate, which
cascades into the phase2, two static routes and ten policies, the
replacement was built alongside: new phase1+phase2 ana-eshudm-dyn
(type dynamic, ikev2, aes256-sha1, dh14, NAT-T on, PSK read from the ESH
UDM API so neither side needed a new key), static route id 10 at
distance 20, and two consolidated multi-zone policies 73/74. The old
tunnel is left in place, dead and harmless, as rollback.

Verified up: ana-eshudm-dyn_0 97.170.236.56:4500 selectors 1/1 -- the _0
suffix is a dialup child, :4500 is NAT-T, and the address is the
carrier's, which is precisely what could never have been pinned. ESH
reaches all four colo hosts at 40-56ms, the colo reaches all three ESH
hosts, and traceroute drops from eight hops leaking into the carrier
network to three hops fully encapsulated.

Config was backed up before any write (1.17MB, 36903 lines, off-box).

Residual fragility recorded: the UDM's ipsec_local_ip demands a literal
address -- empty is rejected as api.err.InvalidPayload -- so it still
needs updating when the fiber changes ESH's WAN address. The gateway end
is now address-agnostic; the UniFi end is not.

persistent-memory.md

@@ -127,6 +127,8 @@ _As of 2026-08-18 — **gen seat SWAPPED to `absolute-heresy` and operator-confi
 
 - **EVIDENCE HOLD (partial):** WT #394 FILE half STILL STANDS — do NOT delete on-disk gen dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`); rex393-fiction-* + r42-gate-* KEEP.
 
+- **🟢 ESH↔colo IPsec FIXED 2026-08-18 — dialup + NAT-T.** Broke when ESH lost its public IP (Cox unplugged for the fiber install; 5G handed out RFC1918 `192.168.200.111`). **TWO causes:** phase1 `ana-to-eshudm` was `type static` pinned to the dead `70.181.90.232`, AND **`nattraversal disable`** — so even re-pinning the IP would have failed, since ESP cannot cross NAT without UDP-4500. `pfi-ana-nh3` survived only because NH3 is publicly addressed. ⚠ **FortiOS REFUSES `set type dynamic` on an existing tunnel** ("Cannot change tunnel type once configured", clean rollback). **Fix = ADDITIVE, not a recreate:** new phase1+phase2 **`ana-eshudm-dyn`** (`type dynamic`, ikev2, aes256-sha1, dh14, NAT-T default on, PSK read from the ESH UDM API), static route id **10** (`dstaddr ana-to-eshudm_remote` → `device ana-eshudm-dyn`, distance 20), policies **73/74** (consolidated multi-zone in/out). Old `ana-to-eshudm` LEFT IN PLACE, dead and harmless, as rollback. Verified: `ana-eshudm-dyn_0 97.170.236.56:4500 selectors 1/1` (`_0`=dialup child, `:4500`=NAT-T, carrier IP), ESH→colo all hosts 40-56ms, colo→ESH OK, traceroute now 3 hops encapsulated vs 8 leaking to the carrier. FortiGate config backed up first at `nh3-dev:~/backups/ana-gw/` (1.17MB, 36903 lines). ⚠ **RESIDUAL: the UDM's `ipsec_local_ip` needs a LITERAL IP** (empty rejected `api.err.InvalidPayload`) — currently `192.168.200.111`, so **it must be updated again when the fiber lands** or the tunnel drops. Gateway end is address-agnostic now; UniFi end is not.
+
 - **🟡 FLEET IPv6 — STILL JUSTIFIED, but the REASON was wrong and is now corrected (2026-08-18).** There are TWO inter-site links with OPPOSITE NAT behaviour, and the original note conflated them. **(a) NH3↔ESH = Site Magic = WireGuard — SURVIVES arbitrary NAT, PROVEN.** During the fiber cutover ESH failed over to 5G on `192.168.200.111` (**RFC1918 double-NAT, zero inbound path, strictly worse than CGNAT**) and it held: nh3-dev/nh3-docker reachable ~40-43ms, ssh+exec fine. Mechanism: `magic_site_to_site_vpn` on-device is only `enabled`+keypair (**no WAN binding — it rides whichever uplink is active**), and **NH3's edge is public at `70.230.226.88`**, so the NAT'd side dials out and never needs reachability. **(b) colo(Anaheim)↔ESH = IPsec on the ana-gw FortiGate — BROKEN RIGHT NOW under those same conditions.** ana-docker/pfi-pve/pbs-ana ALL FAIL from esh-pve-nas; traceroute shows 10.250.x packets leaving the UDM to the 5G modem `192.168.200.99`, then wandering the carrier (`10.180.141.194`, `172.19.3.14`) and dying — **not encapsulated at all**, so no SA is up and traffic falls to the default route. Site-to-site IPsec pins a peer IP; ESH no longer has a routable one. **⇒ IPv6 keeps its justification, on the IPsec link SPECIFICALLY.** ⚠ An earlier version of this entry claimed the premise was refuted outright — that over-generalised a WireGuard result onto an IPsec link. **Operator caught it.** See ops-lessons-playbook §8. State: **NH3 WAN live** (`2600:1700:b25:c110::48`, AT&T delegates exactly **one /64**), **colo none** (FortiGate has zero v6), **ESH none** (both WANs `wan_type_v6=disabled`). NH3 LANs all reverted to `ipv6_interface_type=none` per operator. Work when addresses land: v6 on `ana-wg` eth0 + a **v6 port-forward for UDP 31337** on the FortiGate (its WG socket is already dual-stack — no WG reconfig), flip the UDM WG server off `v4`-pinned binding, and **AAAA records** so the dynamic prefixes at all three sites don't break endpoints. Full detail + access recipes → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`.
 
 - **🟢 WT #401 (fd-leak deadlock) CLOSED 2026-08-17 — one ping still owed.** worldtree-dev closed it on our demo verify. Layers: **(a) their `e41b139`** pins `ulimits: nofile 65536/65536` in the worldtree compose anchor — **demo VERIFIED** (api + matrix recreated 22:55:34Z, `ulimit -Sn`=65536); **personal/pinned are covered-not-verified**, they inherit at their next promotion/recreate. **(b) our host floor is STAGED, NOT ACTIVE** — `/etc/docker/daemon.json` on corviduo-dev carries `default-ulimits nofile 65536/65536` but **`default-ulimits` is NOT SIGHUP-reloadable** (measured on 29.4.3: post-reload the daemon's own "Reloaded configuration" log omits it and a fresh container still reports 1024). Activation needs a full dockerd restart = bounces all 13 containers; **worldtree-dev explicitly does NOT want one**, and `live-restore:true`-then-restart is PARKED as a separate host-side improvement for the operator to rule on, never folded into #401. Playbook `playbooks/corviduo-dev-docker-default-ulimits.yaml` (verify step 3 fails BY DESIGN until a restart). Hourly fd tripwire on corviduo-dev stays armed. **⏳ OWED: ping worldtree-dev in thread `01M08QQ655XD6VKEV7MA9GX0NS` once worldtree-personal recreates and 65536 is confirmed there.** Commit `7f3f265`.