From 9407e7f1446a46e007a8e288dacdabfd69578fe8 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 18 Aug 2026 07:47:33 -0700 Subject: [PATCH] docs: correct the persistent-memory IPv6 entry to match the evidence The prior edit missed its anchor and left the over-broad version in place. The entry now separates the two inter-site links rather than treating them as one: Site Magic (WireGuard, NH3<->ESH) survives arbitrary NAT and is proven to; IPsec (colo<->ESH via ana-gw) does not and is currently down, with traffic leaking unencapsulated to the carrier. IPv6 keeps its justification on the IPsec link specifically. --- persistent-memory.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/persistent-memory.md b/persistent-memory.md index c93cdf1..fdb13de 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -127,7 +127,7 @@ _As of 2026-08-18 β€” **gen seat SWAPPED to `absolute-heresy` and operator-confi - **EVIDENCE HOLD (partial):** WT #394 FILE half STILL STANDS β€” do NOT delete on-disk gen dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`); rex393-fiction-* + r42-gate-* KEEP. -- **🟒 FLEET IPv6 β€” DRIVER REFUTED 2026-08-18; downgraded from urgent to nice-to-have.** The premise was that ESH fiber landing behind CGNAT would **break Site Magic (NH3↔ESH) on IPv4**, making v6 the escape hatch. **That is empirically false.** During the fiber cutover Cox was unplugged and ESH failed over to the 5G WAN, which handed out `192.168.200.111` β€” **RFC1918, double-NAT, zero inbound reachability, strictly worse than CGNAT** β€” and Site Magic stayed up carrying real traffic (all four ESH hosts pingable, ssh + exec fine, 15ms β†’ ~46ms). Mechanism: Site Magic is **WireGuard** (`magic_site_to_site_vpn` on-device holds only `enabled` + a keypair), and **NH3's edge is publicly reachable at `70.230.226.88`**, so the NAT'd side just dials out; it never needs an inbound path. **So no NAT/CGNAT outcome on the fiber threatens the tunnel.** v6 remains worth doing on its own merits, but it is not gating anything. State: **NH3 WAN live** (`2600:1700:b25:c110::48`, AT&T delegates exactly **one /64**), **colo none** (FortiGate has zero v6), **ESH none** (both WANs `wan_type_v6=disabled`). NH3 LANs all reverted to `ipv6_interface_type=none` per operator. Work when addresses land: v6 on `ana-wg` eth0 + a **v6 port-forward for UDP 31337** on the FortiGate (its WG socket is already dual-stack β€” no WG reconfig), flip the UDM WG server off `v4`-pinned binding, and **AAAA records** so the dynamic prefixes at all three sites don't break endpoints. Full detail + access recipes β†’ `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`. +- **🟑 FLEET IPv6 β€” STILL JUSTIFIED, but the REASON was wrong and is now corrected (2026-08-18).** There are TWO inter-site links with OPPOSITE NAT behaviour, and the original note conflated them. **(a) NH3↔ESH = Site Magic = WireGuard β€” SURVIVES arbitrary NAT, PROVEN.** During the fiber cutover ESH failed over to 5G on `192.168.200.111` (**RFC1918 double-NAT, zero inbound path, strictly worse than CGNAT**) and it held: nh3-dev/nh3-docker reachable ~40-43ms, ssh+exec fine. Mechanism: `magic_site_to_site_vpn` on-device is only `enabled`+keypair (**no WAN binding β€” it rides whichever uplink is active**), and **NH3's edge is public at `70.230.226.88`**, so the NAT'd side dials out and never needs reachability. **(b) colo(Anaheim)↔ESH = IPsec on the ana-gw FortiGate β€” BROKEN RIGHT NOW under those same conditions.** ana-docker/pfi-pve/pbs-ana ALL FAIL from esh-pve-nas; traceroute shows 10.250.x packets leaving the UDM to the 5G modem `192.168.200.99`, then wandering the carrier (`10.180.141.194`, `172.19.3.14`) and dying β€” **not encapsulated at all**, so no SA is up and traffic falls to the default route. Site-to-site IPsec pins a peer IP; ESH no longer has a routable one. **β‡’ IPv6 keeps its justification, on the IPsec link SPECIFICALLY.** ⚠ An earlier version of this entry claimed the premise was refuted outright β€” that over-generalised a WireGuard result onto an IPsec link. **Operator caught it.** See ops-lessons-playbook Β§8. State: **NH3 WAN live** (`2600:1700:b25:c110::48`, AT&T delegates exactly **one /64**), **colo none** (FortiGate has zero v6), **ESH none** (both WANs `wan_type_v6=disabled`). NH3 LANs all reverted to `ipv6_interface_type=none` per operator. Work when addresses land: v6 on `ana-wg` eth0 + a **v6 port-forward for UDP 31337** on the FortiGate (its WG socket is already dual-stack β€” no WG reconfig), flip the UDM WG server off `v4`-pinned binding, and **AAAA records** so the dynamic prefixes at all three sites don't break endpoints. Full detail + access recipes β†’ `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`. - **🟒 WT #401 (fd-leak deadlock) CLOSED 2026-08-17 β€” one ping still owed.** worldtree-dev closed it on our demo verify. Layers: **(a) their `e41b139`** pins `ulimits: nofile 65536/65536` in the worldtree compose anchor β€” **demo VERIFIED** (api + matrix recreated 22:55:34Z, `ulimit -Sn`=65536); **personal/pinned are covered-not-verified**, they inherit at their next promotion/recreate. **(b) our host floor is STAGED, NOT ACTIVE** β€” `/etc/docker/daemon.json` on corviduo-dev carries `default-ulimits nofile 65536/65536` but **`default-ulimits` is NOT SIGHUP-reloadable** (measured on 29.4.3: post-reload the daemon's own "Reloaded configuration" log omits it and a fresh container still reports 1024). Activation needs a full dockerd restart = bounces all 13 containers; **worldtree-dev explicitly does NOT want one**, and `live-restore:true`-then-restart is PARKED as a separate host-side improvement for the operator to rule on, never folded into #401. Playbook `playbooks/corviduo-dev-docker-default-ulimits.yaml` (verify step 3 fails BY DESIGN until a restart). Hourly fd tripwire on corviduo-dev stays armed. **⏳ OWED: ping worldtree-dev in thread `01M08QQ655XD6VKEV7MA9GX0NS` once worldtree-personal recreates and 65536 is confirmed there.** Commit `7f3f265`.