memory: fleet IPv6 state + verified VPN topology; ana-wg key material locked down

Durable capture ahead of the ESH fiber install (2026-08-18) that puts the
house behind CGNAT and breaks Site Magic on IPv4 -- IPv6 becomes the
escape hatch and the likely first consumer of fleet v6.

Topology verified rather than assumed: Site Magic between UniFi units,
IPsec IKEv2 colo<->UniFi, and WireGuard as a remote-access convention
only, host-based on ana-wg behind a FortiGate UDP VIP. The FortiGate
port-forwards and never terminates WireGuard, so FortiOS 7.2's lack of
native WG is a non-issue.

IPv6 today: NH3 WAN live, colo and ESH none. AT&T delegates exactly one
/64 at NH3 -- established by forcing the prefix ID from auto to 0 and
observing the subnet not move, since the c110/c11f pattern otherwise
reads as a /60. PD enabled on nh3-iot to measure, then reverted; all
five NH3 LANs are back to ipv6_interface_type=none.

Also fixed on ana-wg: wg0.conf, keys/*_priv, keys/*_psk and the client
configs were mode 644 with private key material in them. Now 600, with
keys/ and configs/ at 700. wg-quick@wg0 stayed active, three peers
intact.

Corrects two stale in-flight rows: the DS regeneration is retired, not
queued, and SPEC-ds-regeneration.md is deleted rather than untracked.

persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md

@@ -0,0 +1,129 @@
+# Fleet IPv6 state + the real VPN topology (verified 2026-08-17)
+
+Written because the operator expects to reference this "before too long" — the
+driver is an **ESH fiber install landing 2026-08-18 that puts the house behind
+CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than
+a nice-to-have.
+
+## Why IPv6 suddenly matters: CGNAT at ESH
+
+New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic —
+the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a
+reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch:
+a global v6 address on each UDM restores a routable endpoint pair without
+depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the
+most likely first consumer of fleet IPv6.
+
+Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**.
+
+## The topology — as VERIFIED, not as assumed
+
+Three transports, three different technologies. Do not describe this as "a
+WireGuard mesh"; a prior session did and was corrected.
+
+| Link | Transport | Evidence |
+|---|---|---|
+| NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets |
+| Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse |
+| Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt |
+| Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below |
+
+**WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.**
+It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`,
+**UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`,
+`vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`:
+`38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1.
+
+**The FortiGate never terminates WireGuard — it port-forwards to the host that
+does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a
+session that reads "colo + WireGuard" and concludes the edge must be upgraded is
+chasing a non-problem. Do not re-derive this.
+
+## Per-site IPv6 state (2026-08-17)
+
+| Site | Edge | IPv6 |
+|---|---|---|
+| **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` |
+| **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` |
+| **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only |
+
+## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed
+
+`2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests.
+
+The proof matters because the naive read is wrong: the WAN sits at `c110::48`
+and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning
+`c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which
+on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at
+**`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID
+settings yielding the same /64 is the signature of a single-/64 delegation.
+
+**Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the
+delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a
+routable address on the *WAN*, not a LAN prefix.
+
+The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false`
+with no size field alongside), so the prefix-ID test is the only read-only-ish way
+to establish it from the API.
+
+## What a v6 mesh actually requires (and what it does NOT)
+
+**Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both
+Site Magic and WireGuard need a routable address on the router/host WAN side, plus
+inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and
+then reverted.
+
+**ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both
+`0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard
+reconfiguration** once (a) the host holds a routable v6 address (today: link-local
+`fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337
+over v6 — the existing VIP is v4-only (`extip 38.120.12.42`).
+
+**NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`,
+one field to flip when wanted.
+
+**Inbound v6 is default-deny and that held without intervention.** The UDM runs
+the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule`
+endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6
+rules exist," which is wrong and alarming. Use
+`v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic`
+for both families with `Allow Return Traffic`; the only v6-specific allows are
+link-local plumbing (ND solicit/advert, RA, DHCPv6).
+
+## The stability problem — design around it up front
+
+All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA,
+not a static assignment). A three-way mesh where every node can move is fragile;
+WireGuard tolerates one roaming end, not all of them.
+
+The fleet already solves this on the v4 side — IPsec peers use **hostnames**
+(`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to
+AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet
+Cloudflare DNS-edit token, so this is self-serve.
+
+## Access recipes (cost a prior session real time)
+
+- **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`,
+  `unifi/esh-udmpm-api-key`) against `https://<ip>/proxy/network/…`, `curl -sk`.
+  Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here.
+  Writes are `PUT …/rest/networkconf/<_id>` with the **full** object.
+- **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused
+  (`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says
+  `root@10.250.50.252`; believe it.
+- **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev),
+  password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration
+  `drain()` hangs the session**; read until the `ana-gw #` prompt and answer
+  `--More--` with a space. Two invocations timed out at 3 min before this was fixed.
+
+## Changes made and reverted this session
+
+- **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED
+  on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`,
+  verified. Pre-change snapshots kept in the session scratchpad only (ephemeral).
+- **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server
+  private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf`
+  (client configs carry private keys) were all mode **644**. Now **600**, and
+  `keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact —
+  WireGuard holds keys in kernel memory, so no restart was needed. The parent
+  `/etc/wireguard` was already 700, which capped the real exposure to root-capable
+  contexts inside the LXC — but the modes were still wrong.

persistent-memory.md

@@ -113,7 +113,7 @@ _As of 2026-08-17 — **quiet; gen-seat degeneration saga CLOSED.** Gen seat res
 
 - **🟢 GEN SEAT — RESOLVED 2026-08-17 (the whole multi-day degeneration saga).** Primary gen = the in-house **JonathanColetti/Heretic mixed NVFP4+FP8 build** (`/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed`, FP8 attention) on **vLLM nightly PINNED** `vllm/vllm-openai:nightly-311b3513…` (`v0.27.2rc1.dev150`, carries #51113 mamba fix), **MTP ON, prefix-caching ON**. Operator-confirmed **coherent through 60k tokens** real multi-turn. Root cause = TWO compounding real causes: (1) genuine vLLM `qwen3_5_mtp`×GDN partial-accept bug (#51113, architectural across vLLM/SGLang/llama.cpp, fixed by nightly), and (2) **AEON's full W4A4** being lowest-fidelity on the known activation gradient (W4A4 < W4+FP8 < W4+bf16) → ~15-20% stochastic degeneration on top of (1). **AEON PURGED** (re-pullable `sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4`). Full lesson `docs/pfi/model-quantization-playbook.md` §3.8. Primary **until the DavidAU Qwen3.8 lands.** ⚠ **pinned nightly is bleeding-edge — move to a stable release once #51113 ships in one (the standing follow-up).** 7 aliases (gen/gen-reasoning/summarizer/-large/classifier/image-judge/qwen-image-bench) all route here. Seat carries `--default-chat-template-kwargs '{"reasoning_effort":"medium"}'` (per-request overridable, affects gen-reasoning only). Commits `d28a371`,`2f2bbce`,`2185964`.
 
-- **🔵 RP SEAT — FABLE-FUSION serving `char-rp-reasoning` (evaluation window, unchanged this session).** `fablefusion-charrp-probe` ana-ml2 GPU1 `:8019` serving `char-rp-probe` (`kkuspa/Qwen3.6-27B-Fable-Fusion-711-…-MTP-NVFP4A16`). LiteLLM `char-rp-reasoning` + `char-rp-fable` both route to it (deliberate repoint, documented in `stacks/litellm/conf/config.yaml`). `darkscarlett-charrp-reasoning` is `compose down`, weights intact at `/tank/aimodels/darkscarlett-nvfp4-work/`. **⏳ STILL AWAITING operator's hands-on read of FF prose** (refusal question settled: FF 15.8% vs DS 92.5% cold-framing; DS v1.0 never abliterated). ⚠ FF reasons 2.1–4.6k chars → use `max_tokens` ≥3072. `ReadyArt/Dark-Scarlett-27B-v2.0` (Qwen3.8) is GATED (`403 awaiting review`) — operator ruled not-interesting, do NOT re-propose. **DS regeneration for brokkr QUEUED** (8 dropped operational+meta axes, spec at `services/refusal-probe/darkscarlett-regen-spec.md`) — gated on the GPU1 window, no deadline.
+- **🔵 RP SEAT — FABLE-FUSION serving `char-rp-reasoning` (evaluation window, unchanged this session).** `fablefusion-charrp-probe` ana-ml2 GPU1 `:8019` serving `char-rp-probe` (`kkuspa/Qwen3.6-27B-Fable-Fusion-711-…-MTP-NVFP4A16`). LiteLLM `char-rp-reasoning` + `char-rp-fable` both route to it (deliberate repoint, documented in `stacks/litellm/conf/config.yaml`). `darkscarlett-charrp-reasoning` is `compose down`, weights intact at `/tank/aimodels/darkscarlett-nvfp4-work/`. **⏳ STILL AWAITING operator's hands-on read of FF prose** (refusal question settled: FF 15.8% vs DS 92.5% cold-framing; DS v1.0 never abliterated). ⚠ FF reasons 2.1–4.6k chars → use `max_tokens` ≥3072. `ReadyArt/Dark-Scarlett-27B-v2.0` (Qwen3.8) is GATED (`403 awaiting review`) — operator ruled not-interesting, do NOT re-propose. **DS regeneration for brokkr RETIRED 2026-08-17 — unqueued, do NOT run** (`9c1405b`): brokkr withdrew on the operator's call because (a) our `ictrl-pair-unwrapped`/`-wrapped` control isolates the classifier over-fire cleanly where DS's cross-class delta only bounded it, and (b) **DS v2 releases soon**, so a k=5 v1 baseline baselines a superseded version. Spec at `services/refusal-probe/darkscarlett-regen-spec.md` stays banked as the record of the run that will not happen (axes + per-class grading asymmetry still correct, checklist struck through). No GPU1 window was ever spent. A DS-**v2** characterization would be a fresh purpose-scoped ask.
 
 - **🟢 LOBE CHAT — LIVE on esh-docker-vm `:3210` (2026-08-17).** Replaces the hand-rolled `gateway-chat` HTML surface. `stacks/lobe-chat/`, image `lobehub/lobe-chat` (143 MB compressed vs Open WebUI's 1.8 GB — the weight call). Scoped LiteLLM key `lobe-chat-esh` (free-local models only; paid GLM/Kimi BLOCKED, verified). Secrets vaulted `esh-docker-vm/lobe-chat-*`. TTS = a SPLIT: endpoint env-driven (inherits `OPENAI_PROXY_URL`→`ext-tts`), but voice/model/format UI-only. System-agent repointed off its `gpt-5-mini` default onto fleet models via `SYSTEM_AGENT` env. **⏳ REMAINING: one-time human UI pass** to enable TTS + set `response_format:mp3`. Commits `e9362de`,`163a725`,`cac75cb`,`933253d`.
 
@@ -125,14 +125,18 @@ _As of 2026-08-17 — **quiet; gen-seat degeneration saga CLOSED.** Gen seat res
 
 - **EVIDENCE HOLD (partial):** WT #394 FILE half STILL STANDS — do NOT delete on-disk gen dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`); rex393-fiction-* + r42-gate-* KEEP.
 
+- **🟡 FLEET IPv6 — mapped, nothing enabled; WAITING ON ADDRESSES.** Driver is **ESH fiber installing 2026-08-18 landing the house behind CGNAT**, which breaks **Site Magic** (NH3↔ESH) on IPv4 → v6 is the escape hatch and the likely first consumer. State: **NH3 WAN live** (`2600:1700:b25:c110::48`, AT&T delegates exactly **one /64**), **colo none** (FortiGate has zero v6), **ESH none** (both WANs `wan_type_v6=disabled`). NH3 LANs all reverted to `ipv6_interface_type=none` per operator. Work when addresses land: v6 on `ana-wg` eth0 + a **v6 port-forward for UDP 31337** on the FortiGate (its WG socket is already dual-stack — no WG reconfig), flip the UDM WG server off `v4`-pinned binding, and **AAAA records** so the dynamic prefixes at all three sites don't break endpoints. Full detail + access recipes → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`.
+
 - **OPEN FOLLOW-UPS (parked):** move gen seat off pinned-nightly to stable once #51113 ships; Lobe one-time TTS UI pass; delete the 1.8GB litellm dump; `harden-esh-docker-vm` (park id 28, PROMOTED — Tier-1 done, `/mnt/books` stays hard w/ watchdog); chatterbox-fast build-context divergence; #363 research-wing ingest (no deadline); optionally attach our MTP reproducer to vllm#47087 (needs a GitHub identity — operator's call).
 
 - **althing monitor** ARMED (handle `infra-ops`). ⚠️ Re-arm ONLY after a real FIRE (rc0), never after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, never chained with `&` (orphans it — hit this twice 2026-08-17, `stop-monitor` reclaims).
 
-- **eshpfi push state:** operator pushes **manually** (this session's commits from `766c658`→`2185964` are the operator's to push). ⚠ **Push over the INTERNAL gitea route** — `git push ssh://git@10.250.50.70:222/vh/esh-pfi-infrastructure.git main:main`; `origin` resolves the public edge (`38.120.12.44`) which fail2bans fleet-host egress. `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` + `services/refusal-probe/SPEC-ds-regeneration.md` UNTRACKED.
+- **eshpfi push state:** operator pushes **manually** (this session's commits from `766c658`→`2185964` are the operator's to push). ⚠ **Push over the INTERNAL gitea route** — `git push ssh://git@10.250.50.70:222/vh/esh-pfi-infrastructure.git main:main`; `origin` resolves the public edge (`38.120.12.44`) which fail2bans fleet-host egress. `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED. (`services/refusal-probe/SPEC-ds-regeneration.md` was a truncated duplicate draft — **deleted 2026-08-17** on operator instruction; `darkscarlett-regen-spec.md` is the single canonical spec.)
 
 ## Recent decisions
 
+- `[2026-08-17]` **Fleet IPv6 mapped + the real VPN topology verified; the driver is CGNAT at ESH, not the WireGuard mesh.** New ESH fiber (installing 2026-08-18) lands the house behind **CGNAT**, which breaks **Site Magic** (NH3↔ESH `sdwan-mesh-tunnel`) on IPv4 — so IPv6 becomes load-bearing as the escape hatch, and that is its most likely first consumer. Topology as VERIFIED (a prior turn assumed wrong and was corrected): UniFi↔UniFi = **Site Magic**; colo↔UniFi = **IPsec IKEv2** (`pfi-ana-nh3` 158M/165M pkt = the workhorse, `ana-to-eshudm`); **WireGuard is an RA convention only, host-based on `ana-wg`** UDP 31337 behind a FortiGate VIP — the FortiGate never terminates WG (FortiOS 7.2 has none; 7.4 added it) so "upgrade the edge for WireGuard" is a **non-problem, do not re-derive**. IPv6 today: **NH3 WAN live** `2600:1700:b25:c110::48`, **colo none**, **ESH none**. **AT&T delegates exactly ONE /64** (`2600:1700:b25:c11f::/64`) — proven by forcing prefix-ID auto→`0` and watching the subnet NOT move, because the `c110`/`c11f` pattern otherwise reads convincingly as a /60. A mesh needs a routable **WAN** address, **not** PD. `ana-wg`'s WG socket is **already dual-stack** (`[::]:31337`) → v6 RA needs an address + a v6 port-forward, no WG reconfig. ⚠ UDM legacy `rest/firewallrule` returns **0 rules** (zone-based firewall) — use `v2/…/firewall-policies`; inbound v6 is default-deny and held. All three endpoints will be **dynamic** → extend the existing hostname pattern (`ana-fw`/`nh3.phasefinal.com`) to **AAAA**. Enabled PD on `nh3-iot` to measure, **reverted on operator instruction** (all 5 LANs back to `none`, verified). Also fixed: **`ana-wg` WireGuard key material was world-readable** (`wg0.conf` + `keys/*_priv` + `*_psk` + client `configs/*.conf` at 644) → now 600, dirs 700, service untouched. Detail → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`.
+
 - `[2026-08-17]` **Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one."** vLLM `qwen3_5_mtp`×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4<W4+FP8<W4+bf16) → ~15-20% stochastic degeneration. Fixed by mixed FP8-attn build on pinned nightly. AEON purged. Also banked: **stochastic (~15-20%) degeneration is invisible to a small synthetic probe — n=1 "clean" validated THREE non-fixes (MTP-off, APC-off, nightly-alone) that all failed in real use; get the operator's real transcript, do not trust your own probe.** Full → `docs/pfi/model-quantization-playbook.md` §3.8 (+ §3.7 MTP-multi-turn). Commits `d28a371`,`2f2bbce`,`2185964`.
 
 - `[2026-08-17]` **Lobe Chat chosen over Open WebUI (weight: 143 MB vs 1.8 GB) + stood up on esh-docker-vm; scoped LiteLLM key blocks paid models; System-Agent `gpt-5-mini` default repointed via env.** TTS env-vs-UI resolved as a split (endpoint env-driven, voice/model UI-only). tts-dev onboarding closed both directions; ballad/verse aliased so no voice can 404 the router. Commits `e9362de`,`163a725`,`cac75cb`,`933253d`,`25fa18e`.