# Fleet IPv6 state + the real VPN topology (verified 2026-08-17) Written because the operator expects to reference this "before too long" — the driver is an **ESH fiber install landing 2026-08-18 that puts the house behind CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than a nice-to-have. ## Why IPv6 suddenly matters: CGNAT at ESH New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic — the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch: a global v6 address on each UDM restores a routable endpoint pair without depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the most likely first consumer of fleet IPv6. Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**. ## The topology — as VERIFIED, not as assumed Three transports, three different technologies. Do not describe this as "a WireGuard mesh"; a prior session did and was corrected. | Link | Transport | Evidence | |---|---|---| | NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets | | Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse | | Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt | | Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below | **WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.** It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`, **UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`, `vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`: `38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1. **The FortiGate never terminates WireGuard — it port-forwards to the host that does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a session that reads "colo + WireGuard" and concludes the edge must be upgraded is chasing a non-problem. Do not re-derive this. ## Per-site IPv6 state (2026-08-17) | Site | Edge | IPv6 | |---|---|---| | **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` | | **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` | | **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only | ## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed `2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests. The proof matters because the naive read is wrong: the WAN sits at `c110::48` and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning `c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at **`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID settings yielding the same /64 is the signature of a single-/64 delegation. **Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a routable address on the *WAN*, not a LAN prefix. The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false` with no size field alongside), so the prefix-ID test is the only read-only-ish way to establish it from the API. ## What a v6 mesh actually requires (and what it does NOT) **Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both Site Magic and WireGuard need a routable address on the router/host WAN side, plus inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and then reverted. **ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both `0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard reconfiguration** once (a) the host holds a routable v6 address (today: link-local `fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337 over v6 — the existing VIP is v4-only (`extip 38.120.12.42`). **NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`, one field to flip when wanted. **Inbound v6 is default-deny and that held without intervention.** The UDM runs the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule` endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6 rules exist," which is wrong and alarming. Use `v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic` for both families with `Allow Return Traffic`; the only v6-specific allows are link-local plumbing (ND solicit/advert, RA, DHCPv6). ## The stability problem — design around it up front All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA, not a static assignment). A three-way mesh where every node can move is fragile; WireGuard tolerates one roaming end, not all of them. The fleet already solves this on the v4 side — IPsec peers use **hostnames** (`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet Cloudflare DNS-edit token, so this is self-serve. ## Access recipes (cost a prior session real time) - **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`, `unifi/esh-udmpm-api-key`) against `https:///proxy/network/…`, `curl -sk`. Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here. Writes are `PUT …/rest/networkconf/<_id>` with the **full** object. - **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused (`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says `root@10.250.50.252`; believe it. - **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev), password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration `drain()` hangs the session**; read until the `ana-gw #` prompt and answer `--More--` with a space. Two invocations timed out at 3 min before this was fixed. ## Changes made and reverted this session - **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`, verified. Pre-change snapshots kept in the session scratchpad only (ephemeral). - **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf` (client configs carry private keys) were all mode **644**. Now **600**, and `keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact — WireGuard holds keys in kernel memory, so no restart was needed. The parent `/etc/wireguard` was already 700, which capped the real exposure to root-capable contexts inside the LXC — but the modes were still wrong. --- ## CORRECTION (recorded 2026-08-24): "AT&T delegates exactly ONE /64" is the ## per-REQUEST truth, not the total — eight /64s exist and are unclaimed The section above concludes AT&T hands out a single `/64` and that the `c110`/`c11f` pattern reading as a `/60` was a misread. **That conclusion was itself superseded later in the same session, and the correction never made it into memory** — it survived only in the session transcript, and was recovered 2026-08-24 while assessing a proposal to grab more prefixes. Reading the **BGW's own LAN statistics page** gave the whole picture: ``` BGW WAN v6 2001:506:70b2:8958::1 <- AT&T's transit prefix BGW LAN v6 2600:1700:b25:c110::/64 <- the BGW keeps this for itself Delegated 2600:1700:b25:c11f::/64 <- what the UDM got ``` **The BGW holds the `/60` and rations it**, keeping `c110`–`c117` for itself and delegating from the top down — the UDM got `c11f`, the last one. So `c118`–`c11f` are **eight delegatable /64s that genuinely exist and are yours**, sitting unclaimed. Both observations are compatible, which is why the first one looked conclusive: the prefix-ID test only carves *within* a delegation already held, so a UDM holding one `/64` cannot move it no matter what prefix-ID you set. The BGW issues **one `/64` per IA_PD request**, and **UniFi solicits exactly once**. **Consequence — the ceiling is the requester, not the carrier.** More prefixes need more IA_PD requests (multiple IAIDs, or multiple client DUIDs), which the UDM will not do. That is what makes a separate DHCPv6-PD client viable, and it is why "ask AT&T for a bigger delegation" may be aimed at the wrong party: this looks like BGW rationing rather than a provisioning-profile limit. Live state at correction time: `wan_dhcpv6_pd_size: 64`, `wan1 v6 2600:1700:b25:c110::48`, all 5 NH3 LANs still `ipv6_interface_type: none`. ### ⛔ CLOSED 2026-08-24 — operator ruling, do not re-raise The seven unclaimed `/64`s stay unclaimed. Two facts close it: - **The BGW has no IP-passthrough mode.** Operator confirmed, and we hold admin on it — so the cheap path (let the UDM take the `/60` directly and carve it natively, as it already does at ESH) does not exist here. - **The only remaining route is a multi-DUID DHCPv6 client on a VM**, which requires re-cabling to reach the BGW's DHCPv6 server, split-stack routing (UDM for v4, VM for v6), and — the actual cost — **rebuilding the whole IPv6 firewall policy in nftables on that VM**, because routing v6 around the UDM bypasses its zone firewall entirely and would leave every LAN host globally reachable. Operator's call: not worth it. **NH3 LANs stay `ipv6_interface_type: none`.** Do not re-propose on the strength of "there are seven free prefixes" — the prefixes are real, the firewall rebuild is why nobody wants them.