memory: snapshot — Anaheim tunnels on AES-128, ana-gw admin closed, Scriberr live, ESH DNS fixed

Rewrites the in-flight section, which had gone stale in one place
(speaches is stopped, not live) and did not reflect a session in which
everything opened was also closed. Three detail files carry the bulk:
the FortiGate's public surface reduced to nothing including the ACME
listener, Scriberr's deployment and the three upstream defects it
required working around, and the ESH DNS repair that also made the
IPv6 naming scheme real on three hosts.

The tunnel entry in the decisions log described a cipher change as
proposed and pending; it has since been attempted, found impossible in
the form intended, and resolved differently, so the line now points at
the outcome rather than the intention.

New decisions cover the cipher adoption and the finding that the
per-flow ceiling belongs to the UniFi gateway's software cipher rather
than the firewall, the administrative closure, the transcription
deployment, the DNS repair, the retirement of an ASR service whose
only consumer was abandoned, and a database VM that until now had no
fleet identity and no vaulted credential at all.

Three approaches are recorded as abandoned: an AEAD cipher the far end
cannot express, an upstream DNS setting the gateway accepts and
ignores, and a scheme to claim unused delegated prefixes that founders
not on the prefixes but on having to rebuild a firewall policy to use
them.

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-08-23_
+_Last updated: 2026-08-24_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -108,26 +108,40 @@ no longer deployed sidecars here. See Recent decisions.)
   (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
 ## Current state / in-flight
 
-- **🟢 ESH IPv6 LAN naming scheme — now a real doc, and LIVE on one host.** Six ESH LANs carry an 8-hex-digit phrase in the interface identifier (`esh-server`=`4411:B105` "FOR ALL BIOS", `esh-mgmt`=`15DA:B055` "IS DA BOSS", `esh-userland`=`CAFE:4411`, `esh-iot`=`4DBA:D107`, `esh-cameras`=`1533:FACE5`, `Default`=`4BA5:3417`, plus `4411:DBAD` reserved for a future DMZ). **Canonical: [`docs/pfi/ipv6-naming-scheme.md`](docs/pfi/ipv6-naming-scheme.md)** — full table, address structure, what UniFi can and cannot name, and the `if-up.d` recipe. ⚠ This entry was once the ONLY record and a `memory: snapshot` (`837fa36`) deleted it; it took a transcript hunt to recover. It is a convention, not temporal state — **the doc is now the home; do not let this line become the only copy again.** First live use 2026-08-24: AdGuard on esh-docker-vm at `2607:73c0:402:1d02:4411:b105:50:45`, which is what the ESH UDM advertises to clients as its IPv6 resolver.
-- **🔒 ana-gw WAN admin is CLOSED again (2026-08-23, operator-directed) and the FortiGate is scheduled for replacement.** The cutover contingency window is over: `wan1 allowaccess` is back to **`ping` only** (https + ssh removed) and `infra-ops` trusthost is back to **10.0.0.0/8 only** — verified from two sites that 443 and 22 are closed, and that management still works over the tunnel at 10.250.0.1. **There is no longer any out-of-band path to ana-gw**; if both tunnels drop, it is console-only. Re-open = `set allowaccess ping https` on wan1 plus widening the infra-ops trusthost (both one-liners, recorded in auto-memory). ⚠ **Port 80 on 38.120.12.42 is still open and it IS the FortiGate** — the FortiOS **ACME HTTP-01 challenge listener**, opened by `config system acme / set interface "wan1"`, which **bypasses `allowaccess` by design**. Body is literally `ACME Access Only` (403, 101 bytes) for every path; confirmed by sniffer showing `wan1 out 38.120.12.42.80 -> ...: syn ack`. Not a DNAT (only two VIPs exist on .42: Kokoro `:8880`, wg `:31337`) and not an admin surface — it serves only challenge tokens. Closing it = `config system acme` unset interface, which also ends cert renewal.
+_As of 2026-08-24 — a very long ops session; everything it opened, it closed. Nothing is mid-flight. Next work is the **homepage** dashboard._
 
-- **✅ CLOSED 2026-08-23: the Anaheim tunnel "problem" is mostly a measurement artefact, and the AES-GCM cutover is impossible.** Operator authorised the cutover; it was attempted NH3-side-first and **cannot be done — UniFi's manual site-to-site IPsec implements no AES-GCM** (8 spellings rejected `api.err.InvalidPayload` against a passing `aes256` control; accepted enum is `aes128/aes192/aes256/3des` only). This blocks the ESH tunnel too, since both far ends are UDMs. The framing was also wrong twice over: **NH3's uplink is 1 Gbps** (not Anaheim's 2 Gbps — that is the real ceiling), and the tunnel does **692 Mbit/s at 8 streams** (the original stopped at 4 and reported ~550). Against WireGuard on the same UDM and uplink, the gap collapses from 2.3× at one stream to **15% at eight** — so re-architecting onto WireGuard is not worth it. Real constraint = **per-stream ~245 Mbit/s**, both endpoints idle. **Standing mitigation: parallelise bulk transfers (2.8× for free); for single-stream NFS use `nconnect=N`** — the `/mnt/smithy` mount on ana-ml2 at 24.7 MB/s is exactly this case and is the obvious test. FortiGate phase2 `pfi-ana-nh3` was left widened to `aes256-sha1 aes256gcm` (inert while the peer offers only CBC); UDM verified byte-identical to its pre-change snapshot. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
+- **🔒 ana-gw's public admin surface is now ZERO open TCP ports**, and the box is scheduled for replacement by **OPNsense on a Dell R420** (which brings WireGuard onto the edge — so the downstream-WireGuard-VM design is moot, do not scope it). WAN admin was opened as a cutover contingency then closed on instruction; the FortiOS **ACME listener** on :80 was disabled too. **No out-of-band path to ana-gw remains** — if both tunnels drop it is console-only; re-open is two one-liners in auto-memory. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md`
 
-- **🟢 SEAT MAP (unchanged this session except selene).** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, **now 7 aliases** (see the collision note). **`char-rp`** = MeroMero-v2 dual-mode, GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC on DFlash2, GPU1 :8019. **`selene` RETIRED** — 17.2 GiB reclaimed on GPU1 (free now ~19.4 GiB).
+- **🟢 Both Anaheim IPsec tunnels run AES-128** (make-before-break, neither dropped waiting on a far end). NH3 245→**270 Mbit/s**, ESH 268→**304**. The per-flow ceiling is **the UDM's software AES-CBC, not the FortiGate** — proven by an A/B/A cipher swap at identical CPU. FortiGate proposal lists left permissive so a peer reverting cannot strand a tunnel. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
 
-- **⚠️ THE `sec` DEGENERATION QUESTION IS STILL OPEN AND CONFOUNDED.** Engine and drafter changed together; **the isolating experiment is MTP k=3 on `e9d1398d`** — still not run. Operator ruling stands: degeneration lives in the **un-fixed vLLM**, not the weights; the MTP-head hypothesis is **retracted**. Both prior sightings are n=1 and are NOT evidence. `gen` remains on the old nightly, untouched, gated on that experiment.
+- **🟢 Scriberr transcription LIVE** — `scriberr.ana.internal:8080`, ana-ml2 **GPU1**, all seven backends up. Image is **built locally** (upstream's Blackwell image was never published); runs as **uid 10001**; needs `UV_LINK_MODE=copy`. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md`
 
-- **🟢 ana-ml2 now mounts `/mnt/smithy`** (nh3-nas) **ro + soft, NOT in fstab** — needs a manual remount after reboot. For brokkr's R47 CPU work. Reads 24.7 MB/s sequential vs 98.3 on nh3-dev (that gap is the tunnel above), but **45 files/s vs 34** — small-file work is genuinely faster there. → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md`
+- **🟢 ESH DNS fixed at the IPv6 layer; the naming scheme is real.** The UDM advertised *itself* as the RDNSS resolver and macOS preferred it, so `.internal` failed. RDNSS now **redirected** to AdGuard on all client VLANs, v4 secondary moved off the UDM. All three `esh-server` Linux hosts carry `4411:B105` ("FOR ALL BIOS"). ⚠ Proxmox bridges need `accept_ra=2`. → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md`
 
-- **🟢 ESH IS DUAL-STACK; the v4 static is a Cityside ticket.** IPv6 live on `esh-userland` and `esh-server` from a delegated `/56`. v4 remains CGNAT and a full gateway reboot proved the purchased static is **not provisioned** — carrier ticket, nothing left locally. NH3 stays v6-off deliberately. Flat-zone lateral-movement finding **parked, id 44**.
+- **🟢 ESH IPv6 LAN naming scheme has a real home:** `docs/pfi/ipv6-naming-scheme.md`. It was once a single persistent-memory line that a `memory: snapshot` (`837fa36`) deleted; recovering it took a transcript hunt. **It is a convention, not temporal state — the doc is the home now; never let a memory line become the only copy again.**
 
-- **🟢 OTHER SERVICES.** `hrafn` browser-fetch adopted on ana-docker (infra-ops owns uptime; CI now genuinely deploys). speaches ASR live irv-ml1:8204. Open WebUI esh-docker-vm:3211 — **Lobe retirement still the operator's call**. Booth gained kept-board deletion + per-row link pruning. `pfi` gitea org created; claude-bot is an Owner and can create repos self-serve.
+- **🟢 SEAT MAP (unchanged this session).** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, **7 aliases**. **`char-rp`** = MeroMero-v2 dual-mode, GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC on DFlash2, GPU1 :8019 — now sharing GPU1 with Scriberr. `selene` retired.
 
-- **⏳ OPEN ELSEWHERE:** MTP-k3 isolating experiment; upstream vLLM issue to file (operator's GitHub identity); Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift; `/tank` DEGRADED **70+ days**; Worldtree **#411** debug-room litter; bridge/engine agent-roster drift on both WT instances; brokkr's `gen` vs trained-reward-model bake-off (theirs to initiate). **Working tree is clean and pushed** through `0ad332b`.
+- **⚠️ THE `sec` DEGENERATION QUESTION IS STILL OPEN AND CONFOUNDED.** Engine and drafter changed together; **the isolating experiment is MTP k=3 on `e9d1398d`** — still not run. Operator ruling stands: degeneration lives in the **un-fixed vLLM**, not the weights; the MTP-head hypothesis is **retracted**. Both sightings are n=1. `gen` remains on the old nightly, gated on that experiment.
+
+- **🟢 ana-ml2 mounts `/mnt/smithy`** (nh3-nas) **ro + soft, NOT in fstab** — manual remount after reboot. `nconnect=8` is approved but **deliberately not applied** (see Recent decisions). → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md`
+
+- **🟢 ESH IS DUAL-STACK; the v4 static is a Cityside ticket.** IPv6 live on `esh-userland` and `esh-server` from a delegated `/56`. v4 remains CGNAT; the purchased static is **not provisioned** — carrier ticket. **NH3 stays v6-off — now by explicit ruling, not just inertia.**
+
+- **🟢 OTHER SERVICES.** `hrafn` browser-fetch on ana-docker. **`speaches` on irv-ml1 is STOPPED** (stack retained, one command to restart) — Eyra was abandoned, leaving it no consumer. Open WebUI esh-docker-vm:3211 — **Lobe retirement still the operator's call**. `pfi` gitea org exists; claude-bot is an Owner.
+
+- **⏳ OPEN ELSEWHERE:** MTP-k3 isolating experiment; upstream vLLM issue to file (operator's GitHub identity); Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift; `/tank` DEGRADED **70+ days**; Worldtree **#411** debug-room litter; bridge/engine agent-roster drift on both WT instances; brokkr's `gen` vs trained-reward-model bake-off. **Commits are local and unpushed** through `d419b11` — push is the operator's call.
 
 ## Recent decisions
 
-- `[2026-08-23]` **Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit — not WireGuard, not CPU, not the fibre.** Cipher change proposed and operator-signalled; **execution pending, untracked by operator choice.** → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
+- `[2026-08-24]` **AES-128 adopted on both Anaheim tunnels; the per-flow ceiling root-caused to the UDM's software AES-CBC, exonerating the FortiGate.** Proven by an A/B/A cipher swap at identical CPU — hardware offload is not cipher-cost-sensitive. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
+- `[2026-08-24]` **ana-gw's public admin surface closed to zero open ports, ACME listener included.** Two of my diagnoses were wrong first (an "ISP proxy" that was the FortiGate, and an "all-port VIP" alarm that was a parser gap) — both from reading config instead of measuring from outside. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md`
+- `[2026-08-24]` **Scriberr deployed on ana-ml2 GPU1, image built from source.** Three upstream bugs: the Blackwell image was never published, it must run as uid 10001, and `UV_LINK_MODE=copy` is required or two backends fail silently. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md`
+- `[2026-08-24]` **ESH DNS fixed at the IPv6 layer and the naming scheme went live on three hosts.** UniFi's RDNSS cannot be disabled but CAN be redirected — the field is only honoured when an explicit server is given. → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md`
+- `[2026-08-24]` **`speaches` on irv-ml1 stopped, stack retained** — Eyra was abandoned pre-implementation (Scriberr covers the need), leaving it no consumer. Disposition confirmed to eyra-dev; one command to restart. Tracked at althing thread `01M0RRJX8GPZEBDHF1E3W18RZF`.
+- `[2026-08-24]` **esh-vm-db brought onto the fleet infra-ops identity and given its first vaulted credential.** It previously had none: root and infra-ops refused key auth and `lkraven`'s sudo wanted a password nobody held, leaving `qm guest exec` from the hypervisor as the only privileged path. Break-glass root password at `secret get esh-vm-db/root-breakglass-password` (console-only; plaintext never crossed the wire — only its SHA-512 hash did).
+- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction.** brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread `01M0R46SFYF83099N16WD67KGD`.
+- `[2026-08-23]` **Anaheim's IPsec tunnel ceiling — investigated, then CLOSED 2026-08-24.** The 25%-of-2-Gbps framing was wrong (NH3's uplink is 1 Gbps); AES-GCM proved impossible; AES-128 landed instead. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
 - `[2026-08-23]` **selene retired after losing a head-to-head on its own job; `chat-judge` moved to gen, the model name 404s by design.** Also surfaced that **7 aliases share one seat** — cross-checking between them is an echo, which caught a real defect in brokkr's 46k-exposure R47 gate. → `persistent-memory.d/2026-08-23-selene-retired-alias-collision.md`
 - `[2026-08-23]` **hrafn adopted; its CI reported green for its whole life while deploying nothing.** A staging dir inside the rsync target destroyed its own source mid-copy; the deeper fault was verify steps that asserted uptime, never content. → `persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md`
 - `[2026-08-23]` **Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed to personal.** Includes the `:8009`-is-demo port trap that an IP-only fix would have walked into. → `persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md`
@@ -265,6 +279,9 @@ no longer deployed sidecars here. See Recent decisions.)
 _209 older entries archived to archival-memory.md._
 ## Tried and abandoned
 
+- `[2026-08-24]` **AES-GCM on the Anaheim tunnels — impossible, not merely hard.** UniFi's manual site-to-site IPsec implements no AEAD cipher at all: eight GCM spellings rejected `api.err.InvalidPayload` against a passing `aes256` control. Blocks both tunnels since both far ends are UDMs. Accepted enum is `aes128/aes192/aes256/3des` — and 3DES is *slower* (no ARM instructions, 64-bit blocks), so AES-128 is the floor.
+- `[2026-08-24]` **Pointing the UDM's `wan_dns1` at AdGuard — silently ignored.** It persists and reads back correctly but the LAN-facing forwarder never uses it; proven with fresh uncached ad domains (AdGuard answers `0.0.0.0`, the UDM returned real IPs). Reverted rather than left in place.
+- `[2026-08-24]` **A multi-DUID DHCPv6 VM to claim NH3's seven unclaimed /64s — declined by the operator.** The BGW has no IP-passthrough (confirmed, we hold admin), so the only route needs re-cabling, split-stack routing and **rebuilding the entire v6 firewall policy off the UDM**. The prefixes are easy; the firewall rebuild is why nobody wants them. Do not re-raise on "there are seven free prefixes".
 - `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.**
 - `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx.