docs: park the ipsec_local_ip watcher until ESH fiber is up

The ESH<->colo tunnel is restored and the FortiGate end is permanently
address-agnostic, but the UniFi end still needs a literal ipsec_local_ip
and so drops on any ESH WAN change -- Cox reclaiming WAN1, the fiber
cutover, or a DHCP renewal.

Operator's call not to build the self-healing watcher yet, which is
right: it would be written against the 5G failover address, which is
about to be replaced, and the fiber may reshape the topology anyway.
Parked as self-healing-ipsec-local-ip-watcher-for-the-esh with the
trigger recorded, plus the follow-up to retire the old ana-to-eshudm
tunnel whose distance-10 route would otherwise reclaim traffic if Cox
returned on the old address.

Records the manual stopgap in persistent memory so the gap is cheap to
cover by hand in the meantime: read wan_ip from the UDM's health
endpoint, PUT it into esh-ana.ipsec_local_ip.

persistent-memory.md

@@ -127,7 +127,7 @@ _As of 2026-08-18 — **gen seat SWAPPED to `absolute-heresy` and operator-confi
 
 - **EVIDENCE HOLD (partial):** WT #394 FILE half STILL STANDS — do NOT delete on-disk gen dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`); rex393-fiction-* + r42-gate-* KEEP.
 
-- **🟢 ESH↔colo IPsec FIXED 2026-08-18 — dialup + NAT-T.** Broke when ESH lost its public IP (Cox unplugged for the fiber install; 5G handed out RFC1918 `192.168.200.111`). **TWO causes:** phase1 `ana-to-eshudm` was `type static` pinned to the dead `70.181.90.232`, AND **`nattraversal disable`** — so even re-pinning the IP would have failed, since ESP cannot cross NAT without UDP-4500. `pfi-ana-nh3` survived only because NH3 is publicly addressed. ⚠ **FortiOS REFUSES `set type dynamic` on an existing tunnel** ("Cannot change tunnel type once configured", clean rollback). **Fix = ADDITIVE, not a recreate:** new phase1+phase2 **`ana-eshudm-dyn`** (`type dynamic`, ikev2, aes256-sha1, dh14, NAT-T default on, PSK read from the ESH UDM API), static route id **10** (`dstaddr ana-to-eshudm_remote` → `device ana-eshudm-dyn`, distance 20), policies **73/74** (consolidated multi-zone in/out). Old `ana-to-eshudm` LEFT IN PLACE, dead and harmless, as rollback. Verified: `ana-eshudm-dyn_0 97.170.236.56:4500 selectors 1/1` (`_0`=dialup child, `:4500`=NAT-T, carrier IP), ESH→colo all hosts 40-56ms, colo→ESH OK, traceroute now 3 hops encapsulated vs 8 leaking to the carrier. FortiGate config backed up first at `nh3-dev:~/backups/ana-gw/` (1.17MB, 36903 lines). ⚠ **RESIDUAL: the UDM's `ipsec_local_ip` needs a LITERAL IP** (empty rejected `api.err.InvalidPayload`) — currently `192.168.200.111`, so **it must be updated again when the fiber lands** or the tunnel drops. Gateway end is address-agnostic now; UniFi end is not.
+- **🟢 ESH↔colo IPsec FIXED 2026-08-18 — dialup + NAT-T.** Broke when ESH lost its public IP (Cox unplugged for the fiber install; 5G handed out RFC1918 `192.168.200.111`). **TWO causes:** phase1 `ana-to-eshudm` was `type static` pinned to the dead `70.181.90.232`, AND **`nattraversal disable`** — so even re-pinning the IP would have failed, since ESP cannot cross NAT without UDP-4500. `pfi-ana-nh3` survived only because NH3 is publicly addressed. ⚠ **FortiOS REFUSES `set type dynamic` on an existing tunnel** ("Cannot change tunnel type once configured", clean rollback). **Fix = ADDITIVE, not a recreate:** new phase1+phase2 **`ana-eshudm-dyn`** (`type dynamic`, ikev2, aes256-sha1, dh14, NAT-T default on, PSK read from the ESH UDM API), static route id **10** (`dstaddr ana-to-eshudm_remote` → `device ana-eshudm-dyn`, distance 20), policies **73/74** (consolidated multi-zone in/out). Old `ana-to-eshudm` LEFT IN PLACE, dead and harmless, as rollback. Verified: `ana-eshudm-dyn_0 97.170.236.56:4500 selectors 1/1` (`_0`=dialup child, `:4500`=NAT-T, carrier IP), ESH→colo all hosts 40-56ms, colo→ESH OK, traceroute now 3 hops encapsulated vs 8 leaking to the carrier. FortiGate config backed up first at `nh3-dev:~/backups/ana-gw/` (1.17MB, 36903 lines). ⚠ **RESIDUAL: the UDM's `ipsec_local_ip` needs a LITERAL IP** (empty rejected `api.err.InvalidPayload`) — currently `192.168.200.111`, so **it must be updated again when the fiber lands** or the tunnel drops. Gateway end is address-agnostic now; UniFi end is not — **it will drop on ANY ESH WAN change: Cox reclaiming WAN1 (still `failover_priority=1`), the fiber cutover, or a DHCP renewal.** **MANUAL FIX (seconds):** read the UDM's live `wan_ip` from `/stat/health` and PUT it into `esh-ana.ipsec_local_ip` via `rest/networkconf/<id>` using `unifi/esh-udmpm-api-key`. A self-healing watcher is **PARKED (`self-healing-ipsec-local-ip-watcher-for-the-esh`), trigger = once the new ESH fiber is live and its addressing has settled** — operator's call 2026-08-18, deliberately not built now because it would target an address that is about to change. Park item also carries the follow-up to delete the old `ana-to-eshudm` tunnel, whose distance-10 route would silently reclaim traffic if Cox ever returns on `70.181.90.232`.
 
 - **🟡 FLEET IPv6 — STILL JUSTIFIED, but the REASON was wrong and is now corrected (2026-08-18).** There are TWO inter-site links with OPPOSITE NAT behaviour, and the original note conflated them. **(a) NH3↔ESH = Site Magic = WireGuard — SURVIVES arbitrary NAT, PROVEN.** During the fiber cutover ESH failed over to 5G on `192.168.200.111` (**RFC1918 double-NAT, zero inbound path, strictly worse than CGNAT**) and it held: nh3-dev/nh3-docker reachable ~40-43ms, ssh+exec fine. Mechanism: `magic_site_to_site_vpn` on-device is only `enabled`+keypair (**no WAN binding — it rides whichever uplink is active**), and **NH3's edge is public at `70.230.226.88`**, so the NAT'd side dials out and never needs reachability. **(b) colo(Anaheim)↔ESH = IPsec on the ana-gw FortiGate — BROKEN RIGHT NOW under those same conditions.** ana-docker/pfi-pve/pbs-ana ALL FAIL from esh-pve-nas; traceroute shows 10.250.x packets leaving the UDM to the 5G modem `192.168.200.99`, then wandering the carrier (`10.180.141.194`, `172.19.3.14`) and dying — **not encapsulated at all**, so no SA is up and traffic falls to the default route. Site-to-site IPsec pins a peer IP; ESH no longer has a routable one. **⇒ IPv6 keeps its justification, on the IPsec link SPECIFICALLY.** ⚠ An earlier version of this entry claimed the premise was refuted outright — that over-generalised a WireGuard result onto an IPsec link. **Operator caught it.** See ops-lessons-playbook §8. State: **NH3 WAN live** (`2600:1700:b25:c110::48`, AT&T delegates exactly **one /64**), **colo none** (FortiGate has zero v6), **ESH none** (both WANs `wan_type_v6=disabled`). NH3 LANs all reverted to `ipv6_interface_type=none` per operator. Work when addresses land: v6 on `ana-wg` eth0 + a **v6 port-forward for UDP 31337** on the FortiGate (its WG socket is already dual-stack — no WG reconfig), flip the UDM WG server off `v4`-pinned binding, and **AAAA records** so the dynamic prefixes at all three sites don't break endpoints. Full detail + access recipes → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`.