Files

T

vh 2686042106 memory: fleet IPv6 state + verified VPN topology; ana-wg key material locked down

Durable capture ahead of the ESH fiber install (2026-08-18) that puts the
house behind CGNAT and breaks Site Magic on IPv4 -- IPv6 becomes the
escape hatch and the likely first consumer of fleet v6.

Topology verified rather than assumed: Site Magic between UniFi units,
IPsec IKEv2 colo<->UniFi, and WireGuard as a remote-access convention
only, host-based on ana-wg behind a FortiGate UDP VIP. The FortiGate
port-forwards and never terminates WireGuard, so FortiOS 7.2's lack of
native WG is a non-issue.

IPv6 today: NH3 WAN live, colo and ESH none. AT&T delegates exactly one
/64 at NH3 -- established by forcing the prefix ID from auto to 0 and
observing the subnet not move, since the c110/c11f pattern otherwise
reads as a /60. PD enabled on nh3-iot to measure, then reverted; all
five NH3 LANs are back to ipv6_interface_type=none.

Also fixed on ana-wg: wg0.conf, keys/*_priv, keys/*_psk and the client
configs were mode 644 with private key material in them. Now 600, with
keys/ and configs/ at 700. wg-quick@wg0 stayed active, three peers
intact.

Corrects two stale in-flight rows: the DS regeneration is retired, not
queued, and SPEC-ds-regeneration.md is deleted rather than untracked.

2026-08-17 13:50:34 -07:00

7.1 KiB

Raw Blame History

Fleet IPv6 state + the real VPN topology (verified 2026-08-17)

Written because the operator expects to reference this "before too long" — the driver is an ESH fiber install landing 2026-08-18 that puts the house behind CGNAT, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than a nice-to-have.

Why IPv6 suddenly matters: CGNAT at ESH

New ESH fiber (installing 2026-08-18) hands out a CGNAT IPv4. Site Magic — the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a reachable endpoint, and a CGNAT address is not one. IPv6 is the escape hatch: a global v6 address on each UDM restores a routable endpoint pair without depending on the ISP's v4 at all. That, not the WireGuard RA mesh, is the most likely first consumer of fleet IPv6.

Operator expects addresses at Anaheim shortly and ESH 2026-08-18.

The topology — as VERIFIED, not as assumed

Three transports, three different technologies. Do not describe this as "a WireGuard mesh"; a prior session did and was corrected.

Link	Transport	Evidence
NH3 UDM ↔ ESH UDM	Site Magic (`vpn_type: sdwan-mesh-tunnel`)	UDM `networkconf`, carries all 7 ESH subnets
Colo FortiGate ↔ NH3 UDM	IPsec IKEv2	FG `pfi-ana-nh3` → 70.230.226.88, 158M pkt rx / 165M tx — the fleet workhorse
Colo FortiGate ↔ ESH UDM	IPsec IKEv2	FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt
Remote-access VPN	WireGuard, host-based on `ana-wg`	see below

WireGuard is an RA (remote-access) convention only — it is NOT the site mesh. It runs on ana-wg (LXC 113, Debian 12, 10.250.50.252), interface wg0, UDP 31337, tunnel subnet 10.30.10.0/24, 3 peers (tc2-mac, vh-iphone, vh-mba26). Reached from outside via a FortiGate VIP wg-to-ana-wg: 38.120.12.42:31337/udp → 10.250.50.252:31337 on wan1.

The FortiGate never terminates WireGuard — it port-forwards to the host that does. FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a session that reads "colo + WireGuard" and concludes the edge must be upgraded is chasing a non-problem. Do not re-derive this.

Per-site IPv6 state (2026-08-17)

Site	Edge	IPv6
NH3	UDM SE	WAN live — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none`
Anaheim colo	FortiGate-80F, FortiOS 7.2.10	None. `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0`
ESH home	UDM Pro Max	None. Both WANs `wan_type_v6=disabled`; link-local only

AT&T delegates exactly ONE /64 at NH3 — proven, not assumed

2600:1700:b25:c11f::/64. One. Not the /60 the addressing pattern suggests.

The proof matters because the naive read is wrong: the WAN sits at c110::48 and the LAN got c11f::1/64, which looks exactly like slot 15 of a /60 spanning c110–c11f. It isn't. Forcing the prefix ID from auto to a manual 0 — which on a real /60 would relocate the LAN to c110::1/64 — left the subnet at c11f::1/64, stable across a 4-minute settle. Two different prefix-ID settings yielding the same /64 is the signature of a single-/64 delegation.

Consequence: exactly one VLAN can have IPv6 at NH3, unless AT&T enlarges the delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a routable address on the WAN, not a LAN prefix.

The controller never exposes the PD size directly (wan_dhcpv6_pd_size_auto:false with no size field alongside), so the prefix-ID test is the only read-only-ish way to establish it from the API.

What a v6 mesh actually requires (and what it does NOT)

Does NOT require prefix delegation. PD hands addresses to LAN clients. Both Site Magic and WireGuard need a routable address on the router/host WAN side, plus inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and then reverted.

ana-wg's WireGuard socket is ALREADY dual-stack — ss shows both 0.0.0.0:31337 and [::]:31337. It will accept IPv6 peers with no WireGuard reconfiguration once (a) the host holds a routable v6 address (today: link-local fe80::be24:11ff:fed7:e4b7 only) and (b) the FortiGate passes inbound UDP 31337 over v6 — the existing VIP is v4-only (extip 38.120.12.42).

NH3 UDM's own WG server is v4-pinned — wireguard_interface_binding_mode_ip_version: 'v4', one field to flip when wanted.

Inbound v6 is default-deny and that held without intervention. The UDM runs the zone-based firewall (66 policies). ⚠ The legacy rest/firewallrule endpoint returns 0 rules on this box — a quick check there reads as "no IPv6 rules exist," which is wrong and alarming. Use v2/api/site/default/firewall-policies. WAN→LAN default is Block All Traffic for both families with Allow Return Traffic; the only v6-specific allows are link-local plumbing (ND solicit/advert, RA, DHCPv6).

The stability problem — design around it up front

All three endpoints will hold dynamic addresses (NH3's came via DHCPv6 IA_NA, not a static assignment). A three-way mesh where every node can move is fragile; WireGuard tolerates one roaming end, not all of them.

The fleet already solves this on the v4 side — IPsec peers use hostnames (ana-fw.phasefinal.com, nh3.phasefinal.com), not raw IPs. Extend that to AAAA records and dynamic prefixes stop mattering. infra-ops holds the fleet Cloudflare DNS-edit token, so this is self-serve.

Access recipes (cost a prior session real time)

UniFi UDMs — X-API-KEY from the vault (secret get unifi/pfi-udmse-api-key, unifi/esh-udmpm-api-key) against https://<ip>/proxy/network/…, curl -sk. Classic api/s/default/rest/networkconf + stat/device carry everything here. Writes are PUT …/rest/networkconf/<_id> with the full object.
ana-wg is root@, NOT infra-ops@ — the shared infra-ops key is refused (Permission denied (publickey,password)). servers/ana-wg/ssh-target says root@10.250.50.252; believe it.
FortiGate — paramiko via uv run --with paramiko (no sshpass on nh3-dev), password secret get fortigate/ana-gw-infra-ops-password. ⚠ A fixed-duration drain() hangs the session; read until the ana-gw # prompt and answer --More-- with a space. Two invocations timed out at 3 min before this was fixed.

Changes made and reverted this session

Enabled PD on nh3-iot (VLAN 90) to measure the delegation, then REVERTED on operator instruction — all 5 NH3 LANs are back to ipv6_interface_type=none, verified. Pre-change snapshots kept in the session scratchpad only (ephemeral).
ana-wg WireGuard key material was world-readable — wg0.conf (server private key + 2 peer PSKs), keys/*_priv, keys/*_psk, and configs/*.conf (client configs carry private keys) were all mode 644. Now 600, and keys/ + configs/ dirs 700. wg-quick@wg0 stayed active, 3 peers intact — WireGuard holds keys in kernel memory, so no restart was needed. The parent /etc/wireguard was already 700, which capped the real exposure to root-capable contexts inside the LXC — but the modes were still wrong.

7.1 KiB Raw Blame History Unescape Escape