# Fleet IPv6 state + the real VPN topology (verified 2026-08-17)

Written because the operator expects to reference this "before too long" — the
driver is an **ESH fiber install landing 2026-08-18 that puts the house behind
CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than
a nice-to-have.

## Why IPv6 suddenly matters: CGNAT at ESH

New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic —
the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a
reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch:
a global v6 address on each UDM restores a routable endpoint pair without
depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the
most likely first consumer of fleet IPv6.

Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**.

## The topology — as VERIFIED, not as assumed

Three transports, three different technologies. Do not describe this as "a
WireGuard mesh"; a prior session did and was corrected.

| Link | Transport | Evidence |
|---|---|---|
| NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets |
| Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse |
| Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt |
| Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below |

**WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.**
It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`,
**UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`,
`vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`:
`38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1.

**The FortiGate never terminates WireGuard — it port-forwards to the host that
does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a
session that reads "colo + WireGuard" and concludes the edge must be upgraded is
chasing a non-problem. Do not re-derive this.

## Per-site IPv6 state (2026-08-17)

| Site | Edge | IPv6 |
|---|---|---|
| **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` |
| **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` |
| **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only |

## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed

`2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests.

The proof matters because the naive read is wrong: the WAN sits at `c110::48`
and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning
`c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which
on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at
**`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID
settings yielding the same /64 is the signature of a single-/64 delegation.

**Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the
delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a
routable address on the *WAN*, not a LAN prefix.

The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false`
with no size field alongside), so the prefix-ID test is the only read-only-ish way
to establish it from the API.

## What a v6 mesh actually requires (and what it does NOT)

**Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both
Site Magic and WireGuard need a routable address on the router/host WAN side, plus
inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and
then reverted.

**ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both
`0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard
reconfiguration** once (a) the host holds a routable v6 address (today: link-local
`fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337
over v6 — the existing VIP is v4-only (`extip 38.120.12.42`).

**NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`,
one field to flip when wanted.

**Inbound v6 is default-deny and that held without intervention.** The UDM runs
the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule`
endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6
rules exist," which is wrong and alarming. Use
`v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic`
for both families with `Allow Return Traffic`; the only v6-specific allows are
link-local plumbing (ND solicit/advert, RA, DHCPv6).

## The stability problem — design around it up front

All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA,
not a static assignment). A three-way mesh where every node can move is fragile;
WireGuard tolerates one roaming end, not all of them.

The fleet already solves this on the v4 side — IPsec peers use **hostnames**
(`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to
AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet
Cloudflare DNS-edit token, so this is self-serve.

## Access recipes (cost a prior session real time)

- **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`,
  `unifi/esh-udmpm-api-key`) against `https://<ip>/proxy/network/…`, `curl -sk`.
  Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here.
  Writes are `PUT …/rest/networkconf/<_id>` with the **full** object.
- **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused
  (`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says
  `root@10.250.50.252`; believe it.
- **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev),
  password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration
  `drain()` hangs the session**; read until the `ana-gw #` prompt and answer
  `--More--` with a space. Two invocations timed out at 3 min before this was fixed.

## Changes made and reverted this session

- **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED
  on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`,
  verified. Pre-change snapshots kept in the session scratchpad only (ephemeral).
- **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server
  private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf`
  (client configs carry private keys) were all mode **644**. Now **600**, and
  `keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact —
  WireGuard holds keys in kernel memory, so no restart was needed. The parent
  `/etc/wireguard` was already 700, which capped the real exposure to root-capable
  contexts inside the LXC — but the modes were still wrong.

---

## CORRECTION (recorded 2026-08-24): "AT&T delegates exactly ONE /64" is the
## per-REQUEST truth, not the total — eight /64s exist and are unclaimed

The section above concludes AT&T hands out a single `/64` and that the
`c110`/`c11f` pattern reading as a `/60` was a misread. **That conclusion was
itself superseded later in the same session, and the correction never made it
into memory** — it survived only in the session transcript, and was recovered
2026-08-24 while assessing a proposal to grab more prefixes.

Reading the **BGW's own LAN statistics page** gave the whole picture:

```
BGW WAN v6   2001:506:70b2:8958::1      <- AT&T's transit prefix
BGW LAN v6   2600:1700:b25:c110::/64    <- the BGW keeps this for itself
Delegated    2600:1700:b25:c11f::/64    <- what the UDM got
```

**The BGW holds the `/60` and rations it**, keeping `c110`–`c117` for itself and
delegating from the top down — the UDM got `c11f`, the last one. So
`c118`–`c11f` are **eight delegatable /64s that genuinely exist and are yours**,
sitting unclaimed.

Both observations are compatible, which is why the first one looked conclusive:
the prefix-ID test only carves *within* a delegation already held, so a UDM
holding one `/64` cannot move it no matter what prefix-ID you set. The BGW
issues **one `/64` per IA_PD request**, and **UniFi solicits exactly once**.

**Consequence — the ceiling is the requester, not the carrier.** More prefixes
need more IA_PD requests (multiple IAIDs, or multiple client DUIDs), which the
UDM will not do. That is what makes a separate DHCPv6-PD client viable, and it
is why "ask AT&T for a bigger delegation" may be aimed at the wrong party: this
looks like BGW rationing rather than a provisioning-profile limit.

Live state at correction time: `wan_dhcpv6_pd_size: 64`, `wan1 v6
2600:1700:b25:c110::48`, all 5 NH3 LANs still `ipv6_interface_type: none`.

### ⛔ CLOSED 2026-08-24 — operator ruling, do not re-raise

The seven unclaimed `/64`s stay unclaimed. Two facts close it:

- **The BGW has no IP-passthrough mode.** Operator confirmed, and we hold admin
  on it — so the cheap path (let the UDM take the `/60` directly and carve it
  natively, as it already does at ESH) does not exist here.
- **The only remaining route is a multi-DUID DHCPv6 client on a VM**, which
  requires re-cabling to reach the BGW's DHCPv6 server, split-stack routing
  (UDM for v4, VM for v6), and — the actual cost — **rebuilding the whole IPv6
  firewall policy in nftables on that VM**, because routing v6 around the UDM
  bypasses its zone firewall entirely and would leave every LAN host globally
  reachable.

Operator's call: not worth it. **NH3 LANs stay `ipv6_interface_type: none`.**
Do not re-propose on the strength of "there are seven free prefixes" — the
prefixes are real, the firewall rebuild is why nobody wants them.
