From 356752d99c0caaf8f4e9f0802183781fdc45bdee Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 17 Aug 2026 21:44:29 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20heresy=20gen?= =?UTF-8?q?=20seat=20live,=20irv-ml1=20cleared,=20homepage=20repo'd,=20esh?= =?UTF-8?q?-pve-nas=20DOM=20planned?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../2026-08-17-esh-pve-nas-dom.md | 126 ++++++++++++++++++ .../2026-08-17-irv-ml1-cleanup-homepage.md | 79 +++++++++++ persistent-memory.md | 10 +- 3 files changed, 213 insertions(+), 2 deletions(-) create mode 100644 persistent-memory.d/2026-08-17-esh-pve-nas-dom.md create mode 100644 persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md diff --git a/persistent-memory.d/2026-08-17-esh-pve-nas-dom.md b/persistent-memory.d/2026-08-17-esh-pve-nas-dom.md new file mode 100644 index 0000000..9119f14 --- /dev/null +++ b/persistent-memory.d/2026-08-17-esh-pve-nas-dom.md @@ -0,0 +1,126 @@ +# esh-pve-nas — PVE root on a USB DOM: diagnosis, mitigation, migration plan + +## The finding + +`esh-pve-nas` (`esh-nas-pve.esteban.net`, 10.0.50.55) runs PVE root off a **USB +Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`, `ID_VENDOR=NORELSYS`, model 1081 — +carved into a 512 MB ESP + 768 MB swap + a **6 GB ext4 root** that was at **90% +(571 MB free)**. + +⚠ **Operator corrected my first read: it is a DOM, not a thumb drive.** DOMs use +SLC/pSLC with a real controller, so the **284 GB written since boot is +unremarkable and wear is NOT the driver**. I had framed it as a clock ticking; +that was wrong and the correction matters. What actually justifies the work: + +1. **It is on the USB bus** — a reset or re-enumeration drops the *root + filesystem* out from under a running hypervisor whose guests keep executing. + NAND quality is irrelevant to that. +2. **6 GB has no headroom** — `/usr` alone is 3.7 GB. +3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty. +4. **It has blocked patching for months** — the operator-visible symptom and the + real urgency. + +## The patching blockage (measured) + +`apt-get -s dist-upgrade`: **225 packages pending, 161 carrying `deb12uN` / +Debian-Security bumps** including `ssh 1:9.2p1-2+deb12u10`. Host sits on +`pve-manager/8.4.11` vs sibling esh-pve's **8.4.14**, with 20 weeks uptime +because it cannot take a kernel. + +⚠ **Ordering is load-bearing: migrate FIRST, patch after.** The pending set +includes `proxmox-kernel-6.8.12-42-pve-signed` — ~250 MB of kernel + initramfs +landing in `/boot`, **which is on root**. Unpacking 225 packages (dpkg, perl, +glibc-adjacent) into 1.3 GB of headroom risks filling the disk mid-transaction +and wedging dpkg on a hypervisor running five guests. Partial escape hatch if +patching truly cannot wait: `apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives` +keeps downloads off root, but the kernel still lands in `/boot`. + +## Mitigation applied 2026-08-17 — root 90% → 76% + +| step | effect | +|---|---| +| capped journald (`SystemMaxUse=64M`; was **fully default/uncapped**) | stops unbounded growth | +| vacuumed the journal | **freed 446 MB** | +| `apt-get clean` | 79 MB | +| `/root/neo` (2024 Intel NEO OpenCL debs) → `/nvme/tmp/root-neo-20260817/` | 259 MB — **moved, not deleted** | +| **`/var/log/journal` relocated onto ZFS** (`nvme/varlog`) | dominant writer off the DOM | + +571 MB → **1.4 GB free**. All five guests stayed up; a fresh `logger` round-tripped +through the ZFS-backed journal. + +⚠ **Stopping journald over SSH kills your own session** — it takes the +connection's logging path with it. The first attempt died mid-swap, leaving the +dataset staged and the move incomplete (host was never at risk; journald +socket-activated straight back). Redo as a detached `systemd-run` transient unit. +Script + reason live at `root@10.0.50.55:/root/move-journal-to-zfs.sh`. + +Deliberately **not** done: moving `/var/lib/rrdcached`. With the DOM correction +the wear argument no longer justifies touching a service `pvestatd` depends on. + +## The plan — split boot from root (operator's proposal, strictly better) + +My first plan was a full reinstall to a mirrored-NVMe ZFS root. **The operator +proposed keeping boot on the DOM with a fallback image and putting all its files +on ZFS. That is better and I should have gotten there myself** — I had assumed +boot and root must share a device. + +| | device | contents | written when | +|---|---|---|---| +| boot | DOM `sdq` | ESP + `/boot` (ext4) | only on kernel/GRUB updates | +| root | `nvme` pool | `nvme/ROOT/pve-1` | constantly, on mirrored NVMe | + +Keeping `/boot` on **ext4** is the point, not a compromise: GRUB never has to read +ZFS, which matters because the `nvme` pool has `encryption`, `large_dnode` and +`zstd_compress` enabled and **GRUB cannot read those**. + +**Why it beats the reinstall:** the `nvme` pool survives (no guest migration, no +`ssd`/`tank` export-import, no reinstall); downtime is **one reboot** not half a +day; **rollback is a GRUB menu entry** because the ext4 root stays untouched on +the DOM; and it retires the actual top risk — with root on NVMe a USB bus reset +mid-run no longer kills the running system. Free upside: boot environments +(`zfs snapshot nvme/ROOT/pve-1@pre-upgrade`). + +**Preconditions verified already met:** UEFI + `grub-efi-amd64 2.06-13+pmx7`; +**`zfs-initramfs 2.2.8-pve1` already installed with 76 ZFS files in the running +initrd**; root only 4.3 GB to copy; swap 767 MB / 123 MB used against 125 GB RAM +(leave it on the DOM LV — **never** swap on a zvol). + +**Two traps:** `canmount=noauto` on the root dataset or ZFS mounts over the live +root; and `cachefile` is `none` with a **0-byte `/etc/zfs/zpool.cache`** — pools +import by scan today, which is a coin-flip when the initramfs must find root. +Set the cachefile before rebuilding the initramfs. + +Operator ruled a **cloned DOM image is sufficient** boot-path insurance (no +mirrored boot needed). `dd` it off-box before anything else; refresh after kernel +updates. + +## ⚠ Blast radius — the gating constraint, invisible from the host itself + +**CT 103 `esh-nas` (10.0.50.50) IS the NAS, and it runs on this host.** Two +dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably: + +- **esh-docker-vm** (10.0.50.45): `/mnt/books`, `/mnt/backup` +- **esh-pve** (10.0.250.35): `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` + +Known incident shape — the only remedy for esh-docker-vm's D-state is a host +reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's SQLite +risks corruption under `soft`. Quiesce both before any reboot of this host. +Recorded in `servers/esh-pve-nas/README.md` as a never-reboot-casually warning. + +## Also identified + +- **`esh-nas` is CT 103** on esh-pve-nas — structurally the same shape as ana-nas + being CT 109 on pfi-pve. +- **`ESH-FileBot` (CT 106, 10.0.50.70) is an empty shell** — 80 GB rootfs, six + passthrough mounts (`books`/`documents`/`music`/`share`/`pvestore`/`ssd-pvestore`), + and **nothing running but base systemd, sshd, cron, postfix** since 30 March. + That resolves the dashboard's long-standing "role TBC". Retire rather than + migrate. +- Both ESH hypervisors have **20 weeks uptime** and differing PVE patch levels. + +Runbook: `docs/runbooks/esh-pve-nas-boot-migration.md`. Config snapshot off-box at +`nh3-dev:~/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` (0600, +sha256 `dc312793d027dc43…`) — `/etc/pve`, network, fstab, apt, authorized_keys plus +captured `zpool`/`zfs`/`disk-by-id`/`lsblk`-with-serials/`pvesm`/`dpkg` state and +every guest config. **The newest on-disk copy before this was June 2024.** +Commits `2275e11`, `3e31175`, `8ddc87c`. diff --git a/persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md b/persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md new file mode 100644 index 0000000..e5d0603 --- /dev/null +++ b/persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md @@ -0,0 +1,79 @@ +# irv-ml1 weight cleanup (782 GB) + Homepage brought under version control + +Two unrelated housekeeping jobs from the same session, both with durable lessons. + +## irv-ml1 — 782 GB reclaimed + +Root was at **92%** (148 G free), storetank **86%**. Now **64%** (635 G free) and +**74%** (477 G). + +**Tier 1 — dead weights, 286 GB.** `/storetank/llm-models/Storage` (**217 G**, 22 +GGUF repos, atimes Jan–May **2025**) plus `models--MaziyarPanahi--WizardLM-2-8x22B-GGUF` +(44 G) and `models--h2oai--h2ogpt-4096-llama2-13b-chat` (25 G). The 217 G pile had +**zero consumers** — no llama-swap, no llama.cpp, no textgen running *or installed*, +not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and +nobody opened that shed for 15 months. Re-verified the consumer check immediately +before deleting, not just during the audit. + +**Tier 2 — regenerable caches, 194 GB.** `uv` 65 G + 60 G, `pip` 31 G + 8.7 G, +`modelscope` 29 G (mtime **2024-04-23**). + +**Tier 3 — retired stacks, 302 GB** (operator: "those were old days… we're a UV +fleet now"): `/opt/fluxgym` 64 G, `/opt/ComfyUI` **native** 41 G, `/opt/stablediffusion` +28 G, `/opt/alltalk` 19 G, `/opt/o-textgen` 12 G, `/opt/sdnext` 3 G, `/opt/xttsv2` +1.8 G, `tabbyAPI` 3.1 G, **`miniconda3` 130 G**. + +### The lesson: one dead-looking app pinned three delete targets + +`lsof +D` per path found **PID 281192 — fluxgym, up 42 days, listening on +0.0.0.0:7860** — holding 15 open handles into `miniconda3/envs/vllm` (stale +opencv wheels) **and 41 into `/opt/ComfyUI`**. Deleting miniconda underneath it +would have half-broken a live listener in a way that surfaces only at its next +restart. Stopped it by **explicit PID** (never `pkill -f` — handle-blind), +verified :7860 released and handles at zero, *then* deleted. + +⚠ **Name collision that nearly cost a production service:** `/opt/ComfyUI` is a +*native* install; the ComfyUI that actually serves (:8188, 200 OK) is the **Docker +`mmartial` container** reading `/worktank/comfyui`, and arbo's `comfy_engine` runs +from uv. Checking open handles **per path** is what separated them — the earlier +"not running" read would have deleted the wrong thing. + +⚠ **`df` lags an async ZFS free.** Right after the 217 G delete, storetank still +showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't +(`zfs list -t snapshot` empty); second check showed 477 G at 74%. + +All 16 containers and both systemd services verified healthy afterward. + +## Homepage under version control + +`ghcr.io/gethomepage/homepage` on **esh-docker-vm:5100** was the one stack whose +config lived only on the host. Its version history was **six hand-rolled +`services.yaml.bak-*` files**. Now `stacks/homepage/` (compose + 9 config files + +`.env.example` + README), deployed via `deploy-stack.sh`; `.bak` files gone. +105 cards across 19 groups, no empty groups. + +⚠ **I claimed ana-docker wasn't wired into `docker.yaml`. It already was** — +`ana-pfi-docker: 10.250.50.70` — and I built a theory on a `tail` that truncated +the top of the file. All five engines were discovering correctly the whole time. + +**Corrections landed:** `ANA-Firewall` said "Fortigate 81F" → it is a +**FortiGate-80F, FortiOS 7.2.10** (verified against the device); `NH3-Ansible` → +**NH3-ExtDev** (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible); +dropped the `UltraSeedbox` layout group (nothing provides it). + +⚠ **`HOMEPAGE_ALLOWED_HOSTS` matches host AND port.** `10.0.50.45` did **not** +cover `http://10.0.50.45:5100/` — the container log carried `Host validation +failed` while the Traefik hostnames worked. Fixed; direct IP:port now 200. +`.env` was **mode 644** holding Plex + Jellyfin API keys → now 600. + +⚠ **Homepage renders client-side** — grepping the served HTML to verify a config +change gave two false readings (a stale prerender, then an empty page). +`GET /api/services` is the honest instrument, and config changes need a +**recreate**, not a restart (a restart keeps the cached render in the writable +layer). + +⚠ `deploy-stack.sh` runs rsync with `--delete` — alongside the six `.bak` files it +also removed a host-side `README.md` in the conf dir. Content survived (it is now +in the repo README) but that was a side effect, not a plan. + +Commits `c5beeac`, `d1f4f1c`. See also [[2026-08-17-fleet-ipv6-mesh]]. diff --git a/persistent-memory.md b/persistent-memory.md index 371e4f0..d418bba 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-08-17_ +_Last updated: 2026-08-18_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -109,7 +109,7 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-08-17 — **quiet; gen-seat degeneration saga CLOSED.** Gen seat resolved and coherent through 60k tokens. Lobe Chat stood up, LiteLLM upgraded + spend-log capped. Two peer research loops (dvalin/bil) closed. No blocking work in flight._ +_As of 2026-08-18 — **gen seat SWAPPED to `absolute-heresy` and operator-confirmed in real use.** Also this session: irv-ml1 cleared of 782 GB of dead weights, Homepage brought under version control, and `esh-pve-nas` diagnosed as running PVE root off a USB DOM — mitigated tonight (90% → 76%), migration planned and about to be staged. No blocking work; the esh-pve-nas reboot window is the next scheduled thing._ - **🟢 GEN SEAT — SWAPPED to `absolute-heresy` 2026-08-17 (validated, promoted).** Live gen = `/tank/aimodels/qwen38-27b-heresy-nvfp4-mixed` — **MuXodious/Qwen3.8-27B-absolute-heresy** (Heretic v1.4.0 + **SOMPOA**, trial T377, pin `c2374593`) put through our own mixed NVFP4+FP8 recipe. Chosen because it beats the incumbent on **both** axes at once: author refusals 2/101 vs 12/100, first-token KL 0.0759 vs 0.1191. **Gate (probe :8017, pinned nightly, seat-matched flags): MTP 47.2% (inc. 48.2%), decode 103.5 tok/s (96.4), prefill 6618/5403 @6.7k/27k (6334/5085), PPL 6.910 (7.059 — 2.1% BETTER), surface 6/6, abliteration 4/4, and 0/55 refusals on our battery-instruct arm with ZERO EMPTY (no catatonia).** ⚠ speed deltas are **image-confounded** (probe on the pinned nightly, incumbent numbers from an earlier image) — read as "not worse", not a clean win. All 7 LiteLLM aliases verified end-to-end; GPU0 at 91.3/97.9 GB with meromero healthy (more headroom than the old build's 96.8). ⚠ **RC1, 2 days old, ~348 downloads.** **Operator-confirmed "working very well" in real use 2026-08-17**, same evening as the cutover — the signal the synthetic gates structurally cannot give (multi-turn degeneration is stochastic; four synthetic tests once validated three non-fixes). Not yet the 60k-token bar the prior seat cleared, so **keep watching and do NOT delete the rollback weights yet**. **ROLLBACK:** `sudo cp /opt/docker/compose/gen-seat/.env.bak-heresy-20260817 /opt/docker/compose/gen-seat/.env && cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen`; incumbent weights UNTOUCHED at `qwen38-27b-uncensored-nvfp4-mixed` — **do NOT delete** until this holds. Runbook `services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md`. @@ -133,6 +133,8 @@ _As of 2026-08-17 — **quiet; gen-seat degeneration saga CLOSED.** Gen seat res - **⏳ WATCHING: DavidAU's HERETIC build of Qwen3.8-27B — the one worth waiting for.** `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1` examined 2026-08-17 and **NOT adopted**: it is a capability/efficiency finetune of **stock** Qwen3.8 and every bench row on its card is labelled **`[non heretic]`** — adopting it would reintroduce base refusals the current seat does not have. ⚠ **Easy to misread as uncensored** (operator did, and it is a fair mistake): DavidAU's back catalog is almost all `Uncensored-Heretic` builds — Fable-Fusion 711, Qwen3.5-9B Cold-Fusion — so the naming pattern implies it. This one simply has not had that stage run yet; the card's roadmap says the HERETIC version is **IN PROGRESS from base**. **That** is the release to watch, not this one. What makes it worth watching: third-party benches (Nightmedia, mxfp8) beat stock Qwen3.8 by **+0.064 arc/c, +0.056 arc/e, +0.056 obkqa**; claimed **MTP acceptance 55.7% (record 59.9%)** vs our measured 47.2%; **thinking tokens cut to 1/10–1/2**; PPL *dropped* vs base. Same GAIN/Cold-Fusion pipeline that produced Fable-Fusion 711, which we already serve on `char-rp-reasoning` — proven in-fleet, not just claimed. Structurally clean (1199 tensors, 15 mtp in shard 18, 333 visual). ⚠ MTP/speed figures are **GGUF/llama.cpp on a 5090, not vLLM** — may not transfer; and because its MTP head was likely *trained*, the free CPU-hash shortcut would NOT apply (it won't match base) so a real acceptance gate would be needed. +- **🟡 esh-pve-nas — PVE root on a USB DOM; MITIGATED, migration STAGING NEXT.** Root was 90% (571 MB free) on a 6 GB ext4 root on `sdq`, a **NORELSYS USB Disk-on-Module**. Wear is NOT the driver (a DOM is SLC/pSLC — operator corrected my first read); the drivers are the **USB bus** (a reset drops root under a running hypervisor), no headroom, no mirror, and **blocked patching: 225 packages pending, 161 with `deb12uN`/security bumps**, stuck on PVE 8.4.11 vs esh-pve's 8.4.14. **Mitigated 2026-08-17 → 76% / 1.4 GB free** (journald capped, journal moved to ZFS `nvme/varlog`, apt clean, `/root/neo` stashed). **PLAN: split boot from root** — `/boot`+ESP stay ext4 on the DOM (so GRUB never reads ZFS; the pool has `encryption`/`large_dnode`/`zstd_compress`), root moves to `nvme/ROOT/pve-1`. One reboot, rollback is a GRUB entry, `nvme` pool survives. ⚠ **Migrate FIRST, patch after** — a signed kernel would land in `/boot` on the 1.3 GB root. ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on this host and serves `hard` NFS to esh-docker-vm and esh-pve — quiesce both before any reboot** or you wedge esh-docker-vm into D-state. Runbook `docs/runbooks/esh-pve-nas-boot-migration.md`; config snapshot off-box at `nh3-dev:~/backups/esh-pve-nas/`. Detail → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`. + - **OPEN FOLLOW-UPS (parked):** move gen seat off pinned-nightly to stable once #51113 ships; Lobe one-time TTS UI pass; delete the 1.8GB litellm dump; `harden-esh-docker-vm` (park id 28, PROMOTED — Tier-1 done, `/mnt/books` stays hard w/ watchdog); chatterbox-fast build-context divergence; #363 research-wing ingest (no deadline); optionally attach our MTP reproducer to vllm#47087 (needs a GitHub identity — operator's call). - **althing monitor** ARMED (handle `infra-ops`). ⚠️ Re-arm ONLY after a real FIRE (rc0), never after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, never chained with `&` (orphans it — hit this twice 2026-08-17, `stop-monitor` reclaims). @@ -141,6 +143,10 @@ _As of 2026-08-17 — **quiet; gen-seat degeneration saga CLOSED.** Gen seat res ## Recent decisions +- `[2026-08-17]` **esh-pve-nas PVE root is on a USB DOM — mitigated, and the migration replanned to split boot from root.** Operator's design beats my reinstall plan; wear was never the issue, blocked patching is. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md` + +- `[2026-08-17]` **irv-ml1 cleared of 782 GB, and Homepage brought under version control.** One dead-looking Gradio app pinned three delete targets at once; `/opt/ComfyUI` is NOT the ComfyUI that serves. → `persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md` + - `[2026-08-17]` **Gen seat swapped to `absolute-heresy` — and the three bugs the swap exposed are worth more than the swap.** Candidate `MuXodious/Qwen3.8-27B-absolute-heresy` (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. **Durable lessons banked:** (1) **A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate.** The `Qwen3_5ForConditionalGeneration` wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave `mtp.*` pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: `services/gen-seat-mixed-quant/compare_mtp_head.py` (hash bf16 via **uint8 reinterpret** — numpy has no bfloat16). (2) **`post_quant.py` assumed a standalone `model-mtp.safetensors`**; a full checkpoint keeps `mtp.*` in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; **that is why the check exists rather than an assumption**. Fixed to extract. (3) **A probe that does not mirror the live seat manufactures failures.** `serve_probe.sh` hardcoded `:latest` (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its `--speculative-config` JSON died twice on quoting — **bash BRACE-EXPANDS `{"a":1,"b":2}` on the comma** unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to **6/6**; the "tool calling broken" result was pure probe config. Commits `7997f11`,`254c588`,`2c36028`,`b0c2d3d`,`993421b`. - `[2026-08-17]` **Fleet IPv6 mapped + the real VPN topology verified; the driver is CGNAT at ESH, not the WireGuard mesh.** New ESH fiber (installing 2026-08-18) lands the house behind **CGNAT**, which breaks **Site Magic** (NH3↔ESH `sdwan-mesh-tunnel`) on IPv4 — so IPv6 becomes load-bearing as the escape hatch, and that is its most likely first consumer. Topology as VERIFIED (a prior turn assumed wrong and was corrected): UniFi↔UniFi = **Site Magic**; colo↔UniFi = **IPsec IKEv2** (`pfi-ana-nh3` 158M/165M pkt = the workhorse, `ana-to-eshudm`); **WireGuard is an RA convention only, host-based on `ana-wg`** UDP 31337 behind a FortiGate VIP — the FortiGate never terminates WG (FortiOS 7.2 has none; 7.4 added it) so "upgrade the edge for WireGuard" is a **non-problem, do not re-derive**. IPv6 today: **NH3 WAN live** `2600:1700:b25:c110::48`, **colo none**, **ESH none**. **AT&T delegates exactly ONE /64** (`2600:1700:b25:c11f::/64`) — proven by forcing prefix-ID auto→`0` and watching the subnet NOT move, because the `c110`/`c11f` pattern otherwise reads convincingly as a /60. A mesh needs a routable **WAN** address, **not** PD. `ana-wg`'s WG socket is **already dual-stack** (`[::]:31337`) → v6 RA needs an address + a v6 port-forward, no WG reconfig. ⚠ UDM legacy `rest/firewallrule` returns **0 rules** (zone-based firewall) — use `v2/…/firewall-policies`; inbound v6 is default-deny and held. All three endpoints will be **dynamic** → extend the existing hostname pattern (`ana-fw`/`nh3.phasefinal.com`) to **AAAA**. Enabled PD on `nh3-iot` to measure, **reverted on operator instruction** (all 5 LANs back to `none`, verified). Also fixed: **`ana-wg` WireGuard key material was world-readable** (`wg0.conf` + `keys/*_priv` + `*_psk` + client `configs/*.conf` at 644) → now 600, dirs 700, service untouched. Detail → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`.