vh
2686042106
Durable capture ahead of the ESH fiber install (2026-08-18) that puts the house behind CGNAT and breaks Site Magic on IPv4 -- IPv6 becomes the escape hatch and the likely first consumer of fleet v6. Topology verified rather than assumed: Site Magic between UniFi units, IPsec IKEv2 colo<->UniFi, and WireGuard as a remote-access convention only, host-based on ana-wg behind a FortiGate UDP VIP. The FortiGate port-forwards and never terminates WireGuard, so FortiOS 7.2's lack of native WG is a non-issue. IPv6 today: NH3 WAN live, colo and ESH none. AT&T delegates exactly one /64 at NH3 -- established by forcing the prefix ID from auto to 0 and observing the subnet not move, since the c110/c11f pattern otherwise reads as a /60. PD enabled on nh3-iot to measure, then reverted; all five NH3 LANs are back to ipv6_interface_type=none. Also fixed on ana-wg: wg0.conf, keys/*_priv, keys/*_psk and the client configs were mode 644 with private key material in them. Now 600, with keys/ and configs/ at 700. wg-quick@wg0 stayed active, three peers intact. Corrects two stale in-flight rows: the DS regeneration is retired, not queued, and SPEC-ds-regeneration.md is deleted rather than untracked.
130 lines
7.1 KiB
Markdown
130 lines
7.1 KiB
Markdown
# Fleet IPv6 state + the real VPN topology (verified 2026-08-17)
|
||
|
||
Written because the operator expects to reference this "before too long" — the
|
||
driver is an **ESH fiber install landing 2026-08-18 that puts the house behind
|
||
CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than
|
||
a nice-to-have.
|
||
|
||
## Why IPv6 suddenly matters: CGNAT at ESH
|
||
|
||
New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic —
|
||
the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a
|
||
reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch:
|
||
a global v6 address on each UDM restores a routable endpoint pair without
|
||
depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the
|
||
most likely first consumer of fleet IPv6.
|
||
|
||
Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**.
|
||
|
||
## The topology — as VERIFIED, not as assumed
|
||
|
||
Three transports, three different technologies. Do not describe this as "a
|
||
WireGuard mesh"; a prior session did and was corrected.
|
||
|
||
| Link | Transport | Evidence |
|
||
|---|---|---|
|
||
| NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets |
|
||
| Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse |
|
||
| Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt |
|
||
| Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below |
|
||
|
||
**WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.**
|
||
It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`,
|
||
**UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`,
|
||
`vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`:
|
||
`38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1.
|
||
|
||
**The FortiGate never terminates WireGuard — it port-forwards to the host that
|
||
does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a
|
||
session that reads "colo + WireGuard" and concludes the edge must be upgraded is
|
||
chasing a non-problem. Do not re-derive this.
|
||
|
||
## Per-site IPv6 state (2026-08-17)
|
||
|
||
| Site | Edge | IPv6 |
|
||
|---|---|---|
|
||
| **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` |
|
||
| **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` |
|
||
| **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only |
|
||
|
||
## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed
|
||
|
||
`2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests.
|
||
|
||
The proof matters because the naive read is wrong: the WAN sits at `c110::48`
|
||
and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning
|
||
`c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which
|
||
on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at
|
||
**`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID
|
||
settings yielding the same /64 is the signature of a single-/64 delegation.
|
||
|
||
**Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the
|
||
delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a
|
||
routable address on the *WAN*, not a LAN prefix.
|
||
|
||
The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false`
|
||
with no size field alongside), so the prefix-ID test is the only read-only-ish way
|
||
to establish it from the API.
|
||
|
||
## What a v6 mesh actually requires (and what it does NOT)
|
||
|
||
**Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both
|
||
Site Magic and WireGuard need a routable address on the router/host WAN side, plus
|
||
inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and
|
||
then reverted.
|
||
|
||
**ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both
|
||
`0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard
|
||
reconfiguration** once (a) the host holds a routable v6 address (today: link-local
|
||
`fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337
|
||
over v6 — the existing VIP is v4-only (`extip 38.120.12.42`).
|
||
|
||
**NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`,
|
||
one field to flip when wanted.
|
||
|
||
**Inbound v6 is default-deny and that held without intervention.** The UDM runs
|
||
the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule`
|
||
endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6
|
||
rules exist," which is wrong and alarming. Use
|
||
`v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic`
|
||
for both families with `Allow Return Traffic`; the only v6-specific allows are
|
||
link-local plumbing (ND solicit/advert, RA, DHCPv6).
|
||
|
||
## The stability problem — design around it up front
|
||
|
||
All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA,
|
||
not a static assignment). A three-way mesh where every node can move is fragile;
|
||
WireGuard tolerates one roaming end, not all of them.
|
||
|
||
The fleet already solves this on the v4 side — IPsec peers use **hostnames**
|
||
(`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to
|
||
AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet
|
||
Cloudflare DNS-edit token, so this is self-serve.
|
||
|
||
## Access recipes (cost a prior session real time)
|
||
|
||
- **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`,
|
||
`unifi/esh-udmpm-api-key`) against `https://<ip>/proxy/network/…`, `curl -sk`.
|
||
Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here.
|
||
Writes are `PUT …/rest/networkconf/<_id>` with the **full** object.
|
||
- **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused
|
||
(`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says
|
||
`root@10.250.50.252`; believe it.
|
||
- **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev),
|
||
password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration
|
||
`drain()` hangs the session**; read until the `ana-gw #` prompt and answer
|
||
`--More--` with a space. Two invocations timed out at 3 min before this was fixed.
|
||
|
||
## Changes made and reverted this session
|
||
|
||
- **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED
|
||
on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`,
|
||
verified. Pre-change snapshots kept in the session scratchpad only (ephemeral).
|
||
- **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server
|
||
private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf`
|
||
(client configs carry private keys) were all mode **644**. Now **600**, and
|
||
`keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact —
|
||
WireGuard holds keys in kernel memory, so no restart was needed. The parent
|
||
`/etc/wireguard` was already 700, which capped the real exposure to root-capable
|
||
contexts inside the LXC — but the modes were still wrong.
|