From 2686042106e2953dbea616ebe103a9e4550c464b Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 17 Aug 2026 13:50:34 -0700 Subject: [PATCH] memory: fleet IPv6 state + verified VPN topology; ana-wg key material locked down Durable capture ahead of the ESH fiber install (2026-08-18) that puts the house behind CGNAT and breaks Site Magic on IPv4 -- IPv6 becomes the escape hatch and the likely first consumer of fleet v6. Topology verified rather than assumed: Site Magic between UniFi units, IPsec IKEv2 colo<->UniFi, and WireGuard as a remote-access convention only, host-based on ana-wg behind a FortiGate UDP VIP. The FortiGate port-forwards and never terminates WireGuard, so FortiOS 7.2's lack of native WG is a non-issue. IPv6 today: NH3 WAN live, colo and ESH none. AT&T delegates exactly one /64 at NH3 -- established by forcing the prefix ID from auto to 0 and observing the subnet not move, since the c110/c11f pattern otherwise reads as a /60. PD enabled on nh3-iot to measure, then reverted; all five NH3 LANs are back to ipv6_interface_type=none. Also fixed on ana-wg: wg0.conf, keys/*_priv, keys/*_psk and the client configs were mode 644 with private key material in them. Now 600, with keys/ and configs/ at 700. wg-quick@wg0 stayed active, three peers intact. Corrects two stale in-flight rows: the DS regeneration is retired, not queued, and SPEC-ds-regeneration.md is deleted rather than untracked. --- .../2026-08-17-fleet-ipv6-mesh.md | 129 ++++++++++++++++++ persistent-memory.md | 8 +- 2 files changed, 135 insertions(+), 2 deletions(-) create mode 100644 persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md diff --git a/persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md b/persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md new file mode 100644 index 0000000..b248c49 --- /dev/null +++ b/persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md @@ -0,0 +1,129 @@ +# Fleet IPv6 state + the real VPN topology (verified 2026-08-17) + +Written because the operator expects to reference this "before too long" — the +driver is an **ESH fiber install landing 2026-08-18 that puts the house behind +CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than +a nice-to-have. + +## Why IPv6 suddenly matters: CGNAT at ESH + +New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic — +the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a +reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch: +a global v6 address on each UDM restores a routable endpoint pair without +depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the +most likely first consumer of fleet IPv6. + +Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**. + +## The topology — as VERIFIED, not as assumed + +Three transports, three different technologies. Do not describe this as "a +WireGuard mesh"; a prior session did and was corrected. + +| Link | Transport | Evidence | +|---|---|---| +| NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets | +| Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse | +| Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt | +| Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below | + +**WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.** +It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`, +**UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`, +`vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`: +`38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1. + +**The FortiGate never terminates WireGuard — it port-forwards to the host that +does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a +session that reads "colo + WireGuard" and concludes the edge must be upgraded is +chasing a non-problem. Do not re-derive this. + +## Per-site IPv6 state (2026-08-17) + +| Site | Edge | IPv6 | +|---|---|---| +| **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` | +| **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` | +| **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only | + +## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed + +`2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests. + +The proof matters because the naive read is wrong: the WAN sits at `c110::48` +and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning +`c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which +on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at +**`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID +settings yielding the same /64 is the signature of a single-/64 delegation. + +**Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the +delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a +routable address on the *WAN*, not a LAN prefix. + +The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false` +with no size field alongside), so the prefix-ID test is the only read-only-ish way +to establish it from the API. + +## What a v6 mesh actually requires (and what it does NOT) + +**Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both +Site Magic and WireGuard need a routable address on the router/host WAN side, plus +inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and +then reverted. + +**ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both +`0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard +reconfiguration** once (a) the host holds a routable v6 address (today: link-local +`fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337 +over v6 — the existing VIP is v4-only (`extip 38.120.12.42`). + +**NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`, +one field to flip when wanted. + +**Inbound v6 is default-deny and that held without intervention.** The UDM runs +the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule` +endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6 +rules exist," which is wrong and alarming. Use +`v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic` +for both families with `Allow Return Traffic`; the only v6-specific allows are +link-local plumbing (ND solicit/advert, RA, DHCPv6). + +## The stability problem — design around it up front + +All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA, +not a static assignment). A three-way mesh where every node can move is fragile; +WireGuard tolerates one roaming end, not all of them. + +The fleet already solves this on the v4 side — IPsec peers use **hostnames** +(`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to +AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet +Cloudflare DNS-edit token, so this is self-serve. + +## Access recipes (cost a prior session real time) + +- **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`, + `unifi/esh-udmpm-api-key`) against `https:///proxy/network/…`, `curl -sk`. + Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here. + Writes are `PUT …/rest/networkconf/<_id>` with the **full** object. +- **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused + (`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says + `root@10.250.50.252`; believe it. +- **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev), + password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration + `drain()` hangs the session**; read until the `ana-gw #` prompt and answer + `--More--` with a space. Two invocations timed out at 3 min before this was fixed. + +## Changes made and reverted this session + +- **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED + on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`, + verified. Pre-change snapshots kept in the session scratchpad only (ephemeral). +- **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server + private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf` + (client configs carry private keys) were all mode **644**. Now **600**, and + `keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact — + WireGuard holds keys in kernel memory, so no restart was needed. The parent + `/etc/wireguard` was already 700, which capped the real exposure to root-capable + contexts inside the LXC — but the modes were still wrong. diff --git a/persistent-memory.md b/persistent-memory.md index d6354e4..ba0d134 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -113,7 +113,7 @@ _As of 2026-08-17 — **quiet; gen-seat degeneration saga CLOSED.** Gen seat res - **🟢 GEN SEAT — RESOLVED 2026-08-17 (the whole multi-day degeneration saga).** Primary gen = the in-house **JonathanColetti/Heretic mixed NVFP4+FP8 build** (`/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed`, FP8 attention) on **vLLM nightly PINNED** `vllm/vllm-openai:nightly-311b3513…` (`v0.27.2rc1.dev150`, carries #51113 mamba fix), **MTP ON, prefix-caching ON**. Operator-confirmed **coherent through 60k tokens** real multi-turn. Root cause = TWO compounding real causes: (1) genuine vLLM `qwen3_5_mtp`×GDN partial-accept bug (#51113, architectural across vLLM/SGLang/llama.cpp, fixed by nightly), and (2) **AEON's full W4A4** being lowest-fidelity on the known activation gradient (W4A4 < W4+FP8 < W4+bf16) → ~15-20% stochastic degeneration on top of (1). **AEON PURGED** (re-pullable `sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4`). Full lesson `docs/pfi/model-quantization-playbook.md` §3.8. Primary **until the DavidAU Qwen3.8 lands.** ⚠ **pinned nightly is bleeding-edge — move to a stable release once #51113 ships in one (the standing follow-up).** 7 aliases (gen/gen-reasoning/summarizer/-large/classifier/image-judge/qwen-image-bench) all route here. Seat carries `--default-chat-template-kwargs '{"reasoning_effort":"medium"}'` (per-request overridable, affects gen-reasoning only). Commits `d28a371`,`2f2bbce`,`2185964`. -- **🔵 RP SEAT — FABLE-FUSION serving `char-rp-reasoning` (evaluation window, unchanged this session).** `fablefusion-charrp-probe` ana-ml2 GPU1 `:8019` serving `char-rp-probe` (`kkuspa/Qwen3.6-27B-Fable-Fusion-711-…-MTP-NVFP4A16`). LiteLLM `char-rp-reasoning` + `char-rp-fable` both route to it (deliberate repoint, documented in `stacks/litellm/conf/config.yaml`). `darkscarlett-charrp-reasoning` is `compose down`, weights intact at `/tank/aimodels/darkscarlett-nvfp4-work/`. **⏳ STILL AWAITING operator's hands-on read of FF prose** (refusal question settled: FF 15.8% vs DS 92.5% cold-framing; DS v1.0 never abliterated). ⚠ FF reasons 2.1–4.6k chars → use `max_tokens` ≥3072. `ReadyArt/Dark-Scarlett-27B-v2.0` (Qwen3.8) is GATED (`403 awaiting review`) — operator ruled not-interesting, do NOT re-propose. **DS regeneration for brokkr QUEUED** (8 dropped operational+meta axes, spec at `services/refusal-probe/darkscarlett-regen-spec.md`) — gated on the GPU1 window, no deadline. +- **🔵 RP SEAT — FABLE-FUSION serving `char-rp-reasoning` (evaluation window, unchanged this session).** `fablefusion-charrp-probe` ana-ml2 GPU1 `:8019` serving `char-rp-probe` (`kkuspa/Qwen3.6-27B-Fable-Fusion-711-…-MTP-NVFP4A16`). LiteLLM `char-rp-reasoning` + `char-rp-fable` both route to it (deliberate repoint, documented in `stacks/litellm/conf/config.yaml`). `darkscarlett-charrp-reasoning` is `compose down`, weights intact at `/tank/aimodels/darkscarlett-nvfp4-work/`. **⏳ STILL AWAITING operator's hands-on read of FF prose** (refusal question settled: FF 15.8% vs DS 92.5% cold-framing; DS v1.0 never abliterated). ⚠ FF reasons 2.1–4.6k chars → use `max_tokens` ≥3072. `ReadyArt/Dark-Scarlett-27B-v2.0` (Qwen3.8) is GATED (`403 awaiting review`) — operator ruled not-interesting, do NOT re-propose. **DS regeneration for brokkr RETIRED 2026-08-17 — unqueued, do NOT run** (`9c1405b`): brokkr withdrew on the operator's call because (a) our `ictrl-pair-unwrapped`/`-wrapped` control isolates the classifier over-fire cleanly where DS's cross-class delta only bounded it, and (b) **DS v2 releases soon**, so a k=5 v1 baseline baselines a superseded version. Spec at `services/refusal-probe/darkscarlett-regen-spec.md` stays banked as the record of the run that will not happen (axes + per-class grading asymmetry still correct, checklist struck through). No GPU1 window was ever spent. A DS-**v2** characterization would be a fresh purpose-scoped ask. - **🟢 LOBE CHAT — LIVE on esh-docker-vm `:3210` (2026-08-17).** Replaces the hand-rolled `gateway-chat` HTML surface. `stacks/lobe-chat/`, image `lobehub/lobe-chat` (143 MB compressed vs Open WebUI's 1.8 GB — the weight call). Scoped LiteLLM key `lobe-chat-esh` (free-local models only; paid GLM/Kimi BLOCKED, verified). Secrets vaulted `esh-docker-vm/lobe-chat-*`. TTS = a SPLIT: endpoint env-driven (inherits `OPENAI_PROXY_URL`→`ext-tts`), but voice/model/format UI-only. System-agent repointed off its `gpt-5-mini` default onto fleet models via `SYSTEM_AGENT` env. **⏳ REMAINING: one-time human UI pass** to enable TTS + set `response_format:mp3`. Commits `e9362de`,`163a725`,`cac75cb`,`933253d`. @@ -125,14 +125,18 @@ _As of 2026-08-17 — **quiet; gen-seat degeneration saga CLOSED.** Gen seat res - **EVIDENCE HOLD (partial):** WT #394 FILE half STILL STANDS — do NOT delete on-disk gen dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`); rex393-fiction-* + r42-gate-* KEEP. +- **🟡 FLEET IPv6 — mapped, nothing enabled; WAITING ON ADDRESSES.** Driver is **ESH fiber installing 2026-08-18 landing the house behind CGNAT**, which breaks **Site Magic** (NH3↔ESH) on IPv4 → v6 is the escape hatch and the likely first consumer. State: **NH3 WAN live** (`2600:1700:b25:c110::48`, AT&T delegates exactly **one /64**), **colo none** (FortiGate has zero v6), **ESH none** (both WANs `wan_type_v6=disabled`). NH3 LANs all reverted to `ipv6_interface_type=none` per operator. Work when addresses land: v6 on `ana-wg` eth0 + a **v6 port-forward for UDP 31337** on the FortiGate (its WG socket is already dual-stack — no WG reconfig), flip the UDM WG server off `v4`-pinned binding, and **AAAA records** so the dynamic prefixes at all three sites don't break endpoints. Full detail + access recipes → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`. + - **OPEN FOLLOW-UPS (parked):** move gen seat off pinned-nightly to stable once #51113 ships; Lobe one-time TTS UI pass; delete the 1.8GB litellm dump; `harden-esh-docker-vm` (park id 28, PROMOTED — Tier-1 done, `/mnt/books` stays hard w/ watchdog); chatterbox-fast build-context divergence; #363 research-wing ingest (no deadline); optionally attach our MTP reproducer to vllm#47087 (needs a GitHub identity — operator's call). - **althing monitor** ARMED (handle `infra-ops`). ⚠️ Re-arm ONLY after a real FIRE (rc0), never after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, never chained with `&` (orphans it — hit this twice 2026-08-17, `stop-monitor` reclaims). -- **eshpfi push state:** operator pushes **manually** (this session's commits from `766c658`→`2185964` are the operator's to push). ⚠ **Push over the INTERNAL gitea route** — `git push ssh://git@10.250.50.70:222/vh/esh-pfi-infrastructure.git main:main`; `origin` resolves the public edge (`38.120.12.44`) which fail2bans fleet-host egress. `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` + `services/refusal-probe/SPEC-ds-regeneration.md` UNTRACKED. +- **eshpfi push state:** operator pushes **manually** (this session's commits from `766c658`→`2185964` are the operator's to push). ⚠ **Push over the INTERNAL gitea route** — `git push ssh://git@10.250.50.70:222/vh/esh-pfi-infrastructure.git main:main`; `origin` resolves the public edge (`38.120.12.44`) which fail2bans fleet-host egress. `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED. (`services/refusal-probe/SPEC-ds-regeneration.md` was a truncated duplicate draft — **deleted 2026-08-17** on operator instruction; `darkscarlett-regen-spec.md` is the single canonical spec.) ## Recent decisions +- `[2026-08-17]` **Fleet IPv6 mapped + the real VPN topology verified; the driver is CGNAT at ESH, not the WireGuard mesh.** New ESH fiber (installing 2026-08-18) lands the house behind **CGNAT**, which breaks **Site Magic** (NH3↔ESH `sdwan-mesh-tunnel`) on IPv4 — so IPv6 becomes load-bearing as the escape hatch, and that is its most likely first consumer. Topology as VERIFIED (a prior turn assumed wrong and was corrected): UniFi↔UniFi = **Site Magic**; colo↔UniFi = **IPsec IKEv2** (`pfi-ana-nh3` 158M/165M pkt = the workhorse, `ana-to-eshudm`); **WireGuard is an RA convention only, host-based on `ana-wg`** UDP 31337 behind a FortiGate VIP — the FortiGate never terminates WG (FortiOS 7.2 has none; 7.4 added it) so "upgrade the edge for WireGuard" is a **non-problem, do not re-derive**. IPv6 today: **NH3 WAN live** `2600:1700:b25:c110::48`, **colo none**, **ESH none**. **AT&T delegates exactly ONE /64** (`2600:1700:b25:c11f::/64`) — proven by forcing prefix-ID auto→`0` and watching the subnet NOT move, because the `c110`/`c11f` pattern otherwise reads convincingly as a /60. A mesh needs a routable **WAN** address, **not** PD. `ana-wg`'s WG socket is **already dual-stack** (`[::]:31337`) → v6 RA needs an address + a v6 port-forward, no WG reconfig. ⚠ UDM legacy `rest/firewallrule` returns **0 rules** (zone-based firewall) — use `v2/…/firewall-policies`; inbound v6 is default-deny and held. All three endpoints will be **dynamic** → extend the existing hostname pattern (`ana-fw`/`nh3.phasefinal.com`) to **AAAA**. Enabled PD on `nh3-iot` to measure, **reverted on operator instruction** (all 5 LANs back to `none`, verified). Also fixed: **`ana-wg` WireGuard key material was world-readable** (`wg0.conf` + `keys/*_priv` + `*_psk` + client `configs/*.conf` at 644) → now 600, dirs 700, service untouched. Detail → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`. + - `[2026-08-17]` **Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one."** vLLM `qwen3_5_mtp`×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4