Files

T

vh d419b11d43 docs(ipv6): close the NH3 multi-prefix question by operator ruling

The gateway has no IP-passthrough mode, confirmed by the operator with
admin access to it, so the inexpensive path of letting the UDM take the
delegation whole and carve it natively is unavailable at this site.

What remains is a separate DHCPv6 client presenting several identities,
which needs recabling to reach the gateway, splits routing so that one
device handles v4 and another v6, and above all moves IPv6 off the
UDM's zone firewall, leaving the entire policy to be rebuilt elsewhere
before any host could safely hold a globally routable address. The
operator has declined it and the LANs stay without v6.

The index entry also still asserted the superseded single-delegation
conclusion as fact; it now carries the corrected reading alongside the
ruling, so the finding and the decision not to act on it arrive
together rather than the first inviting a retry of the second.

2026-08-24 07:05:49 -07:00

10 KiB

Raw Permalink Blame History

Fleet IPv6 state + the real VPN topology (verified 2026-08-17)

Written because the operator expects to reference this "before too long" — the driver is an ESH fiber install landing 2026-08-18 that puts the house behind CGNAT, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than a nice-to-have.

Why IPv6 suddenly matters: CGNAT at ESH

New ESH fiber (installing 2026-08-18) hands out a CGNAT IPv4. Site Magic — the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a reachable endpoint, and a CGNAT address is not one. IPv6 is the escape hatch: a global v6 address on each UDM restores a routable endpoint pair without depending on the ISP's v4 at all. That, not the WireGuard RA mesh, is the most likely first consumer of fleet IPv6.

Operator expects addresses at Anaheim shortly and ESH 2026-08-18.

The topology — as VERIFIED, not as assumed

Three transports, three different technologies. Do not describe this as "a WireGuard mesh"; a prior session did and was corrected.

Link	Transport	Evidence
NH3 UDM ↔ ESH UDM	Site Magic (`vpn_type: sdwan-mesh-tunnel`)	UDM `networkconf`, carries all 7 ESH subnets
Colo FortiGate ↔ NH3 UDM	IPsec IKEv2	FG `pfi-ana-nh3` → 70.230.226.88, 158M pkt rx / 165M tx — the fleet workhorse
Colo FortiGate ↔ ESH UDM	IPsec IKEv2	FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt
Remote-access VPN	WireGuard, host-based on `ana-wg`	see below

WireGuard is an RA (remote-access) convention only — it is NOT the site mesh. It runs on ana-wg (LXC 113, Debian 12, 10.250.50.252), interface wg0, UDP 31337, tunnel subnet 10.30.10.0/24, 3 peers (tc2-mac, vh-iphone, vh-mba26). Reached from outside via a FortiGate VIP wg-to-ana-wg: 38.120.12.42:31337/udp → 10.250.50.252:31337 on wan1.

The FortiGate never terminates WireGuard — it port-forwards to the host that does. FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a session that reads "colo + WireGuard" and concludes the edge must be upgraded is chasing a non-problem. Do not re-derive this.

Per-site IPv6 state (2026-08-17)

Site	Edge	IPv6
NH3	UDM SE	WAN live — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none`
Anaheim colo	FortiGate-80F, FortiOS 7.2.10	None. `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0`
ESH home	UDM Pro Max	None. Both WANs `wan_type_v6=disabled`; link-local only

AT&T delegates exactly ONE /64 at NH3 — proven, not assumed

2600:1700:b25:c11f::/64. One. Not the /60 the addressing pattern suggests.

The proof matters because the naive read is wrong: the WAN sits at c110::48 and the LAN got c11f::1/64, which looks exactly like slot 15 of a /60 spanning c110–c11f. It isn't. Forcing the prefix ID from auto to a manual 0 — which on a real /60 would relocate the LAN to c110::1/64 — left the subnet at c11f::1/64, stable across a 4-minute settle. Two different prefix-ID settings yielding the same /64 is the signature of a single-/64 delegation.

Consequence: exactly one VLAN can have IPv6 at NH3, unless AT&T enlarges the delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a routable address on the WAN, not a LAN prefix.

The controller never exposes the PD size directly (wan_dhcpv6_pd_size_auto:false with no size field alongside), so the prefix-ID test is the only read-only-ish way to establish it from the API.

What a v6 mesh actually requires (and what it does NOT)

Does NOT require prefix delegation. PD hands addresses to LAN clients. Both Site Magic and WireGuard need a routable address on the router/host WAN side, plus inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and then reverted.

ana-wg's WireGuard socket is ALREADY dual-stack — ss shows both 0.0.0.0:31337 and [::]:31337. It will accept IPv6 peers with no WireGuard reconfiguration once (a) the host holds a routable v6 address (today: link-local fe80::be24:11ff:fed7:e4b7 only) and (b) the FortiGate passes inbound UDP 31337 over v6 — the existing VIP is v4-only (extip 38.120.12.42).

NH3 UDM's own WG server is v4-pinned — wireguard_interface_binding_mode_ip_version: 'v4', one field to flip when wanted.

Inbound v6 is default-deny and that held without intervention. The UDM runs the zone-based firewall (66 policies). ⚠ The legacy rest/firewallrule endpoint returns 0 rules on this box — a quick check there reads as "no IPv6 rules exist," which is wrong and alarming. Use v2/api/site/default/firewall-policies. WAN→LAN default is Block All Traffic for both families with Allow Return Traffic; the only v6-specific allows are link-local plumbing (ND solicit/advert, RA, DHCPv6).

The stability problem — design around it up front

All three endpoints will hold dynamic addresses (NH3's came via DHCPv6 IA_NA, not a static assignment). A three-way mesh where every node can move is fragile; WireGuard tolerates one roaming end, not all of them.

The fleet already solves this on the v4 side — IPsec peers use hostnames (ana-fw.phasefinal.com, nh3.phasefinal.com), not raw IPs. Extend that to AAAA records and dynamic prefixes stop mattering. infra-ops holds the fleet Cloudflare DNS-edit token, so this is self-serve.

Access recipes (cost a prior session real time)

UniFi UDMs — X-API-KEY from the vault (secret get unifi/pfi-udmse-api-key, unifi/esh-udmpm-api-key) against https://<ip>/proxy/network/…, curl -sk. Classic api/s/default/rest/networkconf + stat/device carry everything here. Writes are PUT …/rest/networkconf/<_id> with the full object.
ana-wg is root@, NOT infra-ops@ — the shared infra-ops key is refused (Permission denied (publickey,password)). servers/ana-wg/ssh-target says root@10.250.50.252; believe it.
FortiGate — paramiko via uv run --with paramiko (no sshpass on nh3-dev), password secret get fortigate/ana-gw-infra-ops-password. ⚠ A fixed-duration drain() hangs the session; read until the ana-gw # prompt and answer --More-- with a space. Two invocations timed out at 3 min before this was fixed.

Changes made and reverted this session

Enabled PD on nh3-iot (VLAN 90) to measure the delegation, then REVERTED on operator instruction — all 5 NH3 LANs are back to ipv6_interface_type=none, verified. Pre-change snapshots kept in the session scratchpad only (ephemeral).
ana-wg WireGuard key material was world-readable — wg0.conf (server private key + 2 peer PSKs), keys/*_priv, keys/*_psk, and configs/*.conf (client configs carry private keys) were all mode 644. Now 600, and keys/ + configs/ dirs 700. wg-quick@wg0 stayed active, 3 peers intact — WireGuard holds keys in kernel memory, so no restart was needed. The parent /etc/wireguard was already 700, which capped the real exposure to root-capable contexts inside the LXC — but the modes were still wrong.

CORRECTION (recorded 2026-08-24): "AT&T delegates exactly ONE /64" is the

per-REQUEST truth, not the total — eight /64s exist and are unclaimed

The section above concludes AT&T hands out a single /64 and that the c110/c11f pattern reading as a /60 was a misread. That conclusion was itself superseded later in the same session, and the correction never made it into memory — it survived only in the session transcript, and was recovered 2026-08-24 while assessing a proposal to grab more prefixes.

Reading the BGW's own LAN statistics page gave the whole picture:

BGW WAN v6   2001:506:70b2:8958::1      <- AT&T's transit prefix
BGW LAN v6   2600:1700:b25:c110::/64    <- the BGW keeps this for itself
Delegated    2600:1700:b25:c11f::/64    <- what the UDM got

The BGW holds the /60 and rations it, keeping c110–c117 for itself and delegating from the top down — the UDM got c11f, the last one. So c118–c11f are eight delegatable /64s that genuinely exist and are yours, sitting unclaimed.

Both observations are compatible, which is why the first one looked conclusive: the prefix-ID test only carves within a delegation already held, so a UDM holding one /64 cannot move it no matter what prefix-ID you set. The BGW issues one /64 per IA_PD request, and UniFi solicits exactly once.

Consequence — the ceiling is the requester, not the carrier. More prefixes need more IA_PD requests (multiple IAIDs, or multiple client DUIDs), which the UDM will not do. That is what makes a separate DHCPv6-PD client viable, and it is why "ask AT&T for a bigger delegation" may be aimed at the wrong party: this looks like BGW rationing rather than a provisioning-profile limit.

Live state at correction time: wan_dhcpv6_pd_size: 64, wan1 v6 2600:1700:b25:c110::48, all 5 NH3 LANs still ipv6_interface_type: none.

⛔ CLOSED 2026-08-24 — operator ruling, do not re-raise

The seven unclaimed /64s stay unclaimed. Two facts close it:

The BGW has no IP-passthrough mode. Operator confirmed, and we hold admin on it — so the cheap path (let the UDM take the /60 directly and carve it natively, as it already does at ESH) does not exist here.
The only remaining route is a multi-DUID DHCPv6 client on a VM, which requires re-cabling to reach the BGW's DHCPv6 server, split-stack routing (UDM for v4, VM for v6), and — the actual cost — rebuilding the whole IPv6 firewall policy in nftables on that VM, because routing v6 around the UDM bypasses its zone firewall entirely and would leave every LAN host globally reachable.

Operator's call: not worth it. NH3 LANs stay ipv6_interface_type: none. Do not re-propose on the strength of "there are seven free prefixes" — the prefixes are real, the firewall rebuild is why nobody wants them.

10 KiB Raw Permalink Blame History Unescape Escape