diff --git a/persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md b/persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md new file mode 100644 index 0000000..ee441cf --- /dev/null +++ b/persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md @@ -0,0 +1,50 @@ +# [2026-08-24] ana-gw public admin surface closed to zero, ACME listener included + +WAN admin was opened at the start of the session as a cutover contingency +("so I don't have to drive down there"), then closed again on operator +instruction once the AES-128 work landed. Net result: **the FortiGate's WAN +address now exposes no TCP port at all.** + +## Final state + +External scan of `38.120.12.42`, 55 ports: **nothing open**. Verified from two +sites. `wan1 allowaccess` = `ping`; `infra-ops` trusthost back to `10.0.0.0/8`. + +**Consequence to hold: there is no out-of-band path to ana-gw.** If both tunnels +drop it is console-only. Re-open is two one-liners (allowaccess + trusthost) — +both are recorded in auto-memory `reference_fortigate_ana_gw_access`. + +## Port 80 was the FortiOS ACME listener, and I got it wrong first + +`38.120.12.42:80` answered a bare 403 (`ACME Access Only`, 101 bytes) with +`allowaccess` set to ping only. First diagnosis — "an ISP transparent proxy" — +was **wrong**, and the reason is worth keeping: + +> The sniffer filter was `dst host 38.120.12.42 and tcp port 80`. **`dst host` +> matches inbound only**, so outbound SYN-ACKs were excluded *by construction*, +> and concluding "the box sends no SYN-ACK" from that capture was unsound. + +Re-run bidirectionally (`host … and tcp port 80`) it immediately showed +`wan1 out 38.120.12.42.80 -> : syn ack`. **Rule: to test whether a box +*answers*, the filter must be bidirectional.** + +The listener is opened by `config system acme / set interface "wan1"` and +**bypasses `allowaccess` by design** — FortiOS needs port 80 for HTTP-01. It +was disabled (`config system acme / unset interface`); the LE cert (`ana-fw.pfi`, +valid to 2026-10-27) is untouched and simply stops renewing, which is fine +because WAN admin is closed and the box is being replaced. + +## Retracted in the same pass: the "four all-port VIPs" alarm + +Claimed four VIPs were unrestricted all-port static NAT. **False.** A FortiOS +VIP is scoped **two** ways — `portforward`+`extport`, *or* a `service` binding +on the VIP object — and only the first was checked. All 14 VIPs are scoped; +`Rustdesk` is TCP 21115–21119, `ssh-mapped-2223` is TCP 2223 only. + +Ground-truth external scan of all six public IPs is recorded in +`reference_fortigate_ana_gw_access`. Configured-but-dead: `:8443` +(mattermost-calls), `:8444` (webdav-nas), `:8880` (Kokoro-In) — tidy-up +candidates for the OPNsense translation, not exposure. + +**Lesson, twice in one session: measure from outside instead of parsing config.** +Both wrong answers came from a filter that answered a different question. diff --git a/persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md b/persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md new file mode 100644 index 0000000..bc94e3c --- /dev/null +++ b/persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md @@ -0,0 +1,77 @@ +# [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live + +Reported as "`scriberr.ana.internal` doesn't resolve on my Mac, and nslookup +shows an IPv6 DNS server." Operator's diagnosis was right; the fix took three +wrong turns worth recording. + +## Root cause + +`esh-userland` has IPv6 PD with RA at `pref high`, and the UDM advertises +**itself** as the resolver via RDNSS. macOS honours RDNSS and prefers it over +the DHCPv4-supplied resolver, so queries went to the UDM — which does not know +`.internal` — and returned NXDOMAIN. AdGuard was never consulted. + +Two adjacent gaps found while there: `esh-userland`'s **secondary** v4 resolver +was `10.0.10.1` (the UDM itself), and `esh-server` had **DNS handout disabled +entirely**, so every host there got the UDM and could never resolve `.internal` +— esh-docker-vm was living proof. + +## The three wrong turns + +1. **`dhcpdv6_dns_auto=false` alone does nothing.** It is only honoured **when an + explicit server is supplied**. Setting it bare looks like a no-op and invites + the conclusion that the field is inert — which I drew, wrongly. Despite the + `dhcpdv6_` prefix it *does* drive the RA's RDNSS option on a SLAAC network. +2. **`wan_dns1` is NOT used by the UDM's LAN-facing forwarder.** Setting it to + AdGuard persists, reads back, and changes nothing. Proven with **fresh + uncached ad domains** — AdGuard blocklists answer `0.0.0.0`, the UDM returned + real IPs. Reverted. +3. **`force-provision` returns `rc:ok` and proves nothing** — consistent with the + known `cmd/devmgr` behaviour. + +Every failed attempt returned `rc: ok`. **Verify by observed effect.** RAs were +probed with a stdlib raw-socket Router Solicitation parsing option type 25 +(`rdisc6`/`tcpdump` were both absent; nothing was installed). + +## What landed + +RDNSS **redirected** rather than disabled — better than switching it off: + +| VLAN | v4 | v6 RDNSS | +|---|---|---| +| `esh-userland` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` | +| `esh-server` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` | + +The v4 secondary moved from the UDM to the **NH3 AdGuard** — reachable over +Site Magic and authoritative for the zone. ⚠ **A secondary only fails over on +SILENCE, not on wrong answers**: NXDOMAIN is a *successful* answer, the client +accepts it and never retries. A secondary that doesn't know your private zone is +a coin flip, not a spare tyre. `esh-cameras` deliberately untouched — routing +camera DNS through AdGuard's filtering risks their cloud features. + +## The naming scheme became real + +The resolver address is the scheme's first live use, replacing a MAC-derived +SLAAC address that would break on a NIC change. All three `esh-server` Linux +hosts now carry `4411:B105` ("FOR ALL BIOS"): + +``` +esh-docker-vm …:4411:b105:50:45 esh-pve-nas …:4411:b105:50:55 +esh-vm-db …:4411:b105:50:60 +``` + +Applied by an `if-up.d` hook that **derives the prefix at runtime** (self-heals +on re-delegation), backgrounds itself with a retry (SLAAC may not have landed; +a blocking hook would stall bring-up on a headless box), and adds nothing to +existing config. **Not** an `iface … inet6 static` stanza — on Debian that sets +`accept_ra=0` and would strand the host. + +⚠ **Proxmox bridges need `accept_ra=2`.** `esh-pve-nas` had link-local only +despite every sysctl looking right: `vmbr0.forwarding=1`, and the kernel ignores +RAs on a forwarding interface unless `accept_ra` is explicitly `2`. Fixed with +`accept_ra_defrtr=0` alongside, so it takes the prefix but **declines the default +route** — an IPv6 identity with no change to a hypervisor's routing. Expect this +on every Proxmox node when its LAN gets v6. + +Canonical: `docs/pfi/ipv6-naming-scheme.md`. UniFi limits: +auto-memory `reference_unifi_dns_rdnss_limits`. diff --git a/persistent-memory.d/2026-08-24-scriberr-ana-ml2.md b/persistent-memory.d/2026-08-24-scriberr-ana-ml2.md new file mode 100644 index 0000000..153669e --- /dev/null +++ b/persistent-memory.d/2026-08-24-scriberr-ana-ml2.md @@ -0,0 +1,46 @@ +# [2026-08-24] Scriberr transcription deployed on ana-ml2, GPU1 + +Self-hosted audio/video transcription + diarization. Operator chose GPU +placement over ana-docker (8 cores shared with 50 containers, 37 GB disk) +against ana-ml2's 96 cores, `/tank`'s terabytes and GPU1's headroom. + +**Live:** `http://scriberr.ana.internal:8080` (DNS alias added), health `healthy`, +all seven backends up, zero failures: `whisperx pyannote sortformer parakeet +canary voxtral openai`. ~30 GB of weights on `/tank`. + +Stack: `stacks/scriberr/`. Full gotcha list in auto-memory +`reference_scriberr_ana_ml2`. + +## Three upstream bugs, none of them ours + +**1. The Blackwell image does not exist.** Upstream's README documents +`scriberr-cuda-blackwell`; GHCR has **no tags for it**. Published +`scriberr-cuda` covers sm_61–sm_89 only — on these sm_120 cards it fails or +silently drops to CPU. The real sm_120 path is `Dockerfile.cuda.12.9` +(CUDA 12.9.1, cu128 torch), **built from source**. Do not "simplify" the compose +back to the published image. + +**2. It must run as uid 10001, not 1000** — and the error lies: +`unable to open database file: out of memory (14)`. Error 14 is +`SQLITE_CANTOPEN`, not an OOM, on a box with 566 GB RAM. That Dockerfile creates +`appuser` at 10001 (Ubuntu 24.04 owns uid 1000 as `ubuntu`) and chowns `/app` to +it, while the entrypoint's PUID remap covers only the data dirs. +**Isolated by elimination**: SQLite writes fine to `/tank` as 1000 → not the +mount; fails on a plain named volume too → not the storage; the **published CPU +image works at PUID=1000** because there `appuser` *is* 1000. +Generalisable: *when a container "permission" bug appears, compare the uid the +image was BUILT for against the uid you are RUNNING as.* + +**3. `UV_LINK_MODE=copy` is required.** Scriberr builds each backend's Python env +with `uv` at start; uv's reflink mode fails on overlayfs+ZFS with +`Failed to clone … Resource temporarily unavailable (os error 11)`. **Partial +failure** — WhisperX and PyAnnote came up and the app looked fine while Parakeet +and Sortformer were silently absent. Occurrences 2 → 0 after the fix. + +## Related + +`speaches` on irv-ml1 **stopped** the same day (stack retained, one command to +restart): Eyra was abandoned pre-implementation because Scriberr covers the need, +leaving it with no consumer. Scriberr runs its **own** WhisperX in-container and +is **not** a speaches consumer. Idle footprint at stop was 274 MiB, not the +~5.9 GB quoted — that figure is the loaded-model working set. diff --git a/persistent-memory.md b/persistent-memory.md index 28a7a84..aa660e0 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-08-23_ +_Last updated: 2026-08-24_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -108,26 +108,40 @@ no longer deployed sidecars here. See Recent decisions.) (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`. ## Current state / in-flight -- **🟢 ESH IPv6 LAN naming scheme — now a real doc, and LIVE on one host.** Six ESH LANs carry an 8-hex-digit phrase in the interface identifier (`esh-server`=`4411:B105` "FOR ALL BIOS", `esh-mgmt`=`15DA:B055` "IS DA BOSS", `esh-userland`=`CAFE:4411`, `esh-iot`=`4DBA:D107`, `esh-cameras`=`1533:FACE5`, `Default`=`4BA5:3417`, plus `4411:DBAD` reserved for a future DMZ). **Canonical: [`docs/pfi/ipv6-naming-scheme.md`](docs/pfi/ipv6-naming-scheme.md)** — full table, address structure, what UniFi can and cannot name, and the `if-up.d` recipe. ⚠ This entry was once the ONLY record and a `memory: snapshot` (`837fa36`) deleted it; it took a transcript hunt to recover. It is a convention, not temporal state — **the doc is now the home; do not let this line become the only copy again.** First live use 2026-08-24: AdGuard on esh-docker-vm at `2607:73c0:402:1d02:4411:b105:50:45`, which is what the ESH UDM advertises to clients as its IPv6 resolver. -- **🔒 ana-gw WAN admin is CLOSED again (2026-08-23, operator-directed) and the FortiGate is scheduled for replacement.** The cutover contingency window is over: `wan1 allowaccess` is back to **`ping` only** (https + ssh removed) and `infra-ops` trusthost is back to **10.0.0.0/8 only** — verified from two sites that 443 and 22 are closed, and that management still works over the tunnel at 10.250.0.1. **There is no longer any out-of-band path to ana-gw**; if both tunnels drop, it is console-only. Re-open = `set allowaccess ping https` on wan1 plus widening the infra-ops trusthost (both one-liners, recorded in auto-memory). ⚠ **Port 80 on 38.120.12.42 is still open and it IS the FortiGate** — the FortiOS **ACME HTTP-01 challenge listener**, opened by `config system acme / set interface "wan1"`, which **bypasses `allowaccess` by design**. Body is literally `ACME Access Only` (403, 101 bytes) for every path; confirmed by sniffer showing `wan1 out 38.120.12.42.80 -> ...: syn ack`. Not a DNAT (only two VIPs exist on .42: Kokoro `:8880`, wg `:31337`) and not an admin surface — it serves only challenge tokens. Closing it = `config system acme` unset interface, which also ends cert renewal. +_As of 2026-08-24 — a very long ops session; everything it opened, it closed. Nothing is mid-flight. Next work is the **homepage** dashboard._ -- **✅ CLOSED 2026-08-23: the Anaheim tunnel "problem" is mostly a measurement artefact, and the AES-GCM cutover is impossible.** Operator authorised the cutover; it was attempted NH3-side-first and **cannot be done — UniFi's manual site-to-site IPsec implements no AES-GCM** (8 spellings rejected `api.err.InvalidPayload` against a passing `aes256` control; accepted enum is `aes128/aes192/aes256/3des` only). This blocks the ESH tunnel too, since both far ends are UDMs. The framing was also wrong twice over: **NH3's uplink is 1 Gbps** (not Anaheim's 2 Gbps — that is the real ceiling), and the tunnel does **692 Mbit/s at 8 streams** (the original stopped at 4 and reported ~550). Against WireGuard on the same UDM and uplink, the gap collapses from 2.3× at one stream to **15% at eight** — so re-architecting onto WireGuard is not worth it. Real constraint = **per-stream ~245 Mbit/s**, both endpoints idle. **Standing mitigation: parallelise bulk transfers (2.8× for free); for single-stream NFS use `nconnect=N`** — the `/mnt/smithy` mount on ana-ml2 at 24.7 MB/s is exactly this case and is the obvious test. FortiGate phase2 `pfi-ana-nh3` was left widened to `aes256-sha1 aes256gcm` (inert while the peer offers only CBC); UDM verified byte-identical to its pre-change snapshot. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` +- **🔒 ana-gw's public admin surface is now ZERO open TCP ports**, and the box is scheduled for replacement by **OPNsense on a Dell R420** (which brings WireGuard onto the edge — so the downstream-WireGuard-VM design is moot, do not scope it). WAN admin was opened as a cutover contingency then closed on instruction; the FortiOS **ACME listener** on :80 was disabled too. **No out-of-band path to ana-gw remains** — if both tunnels drop it is console-only; re-open is two one-liners in auto-memory. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md` -- **🟢 SEAT MAP (unchanged this session except selene).** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, **now 7 aliases** (see the collision note). **`char-rp`** = MeroMero-v2 dual-mode, GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC on DFlash2, GPU1 :8019. **`selene` RETIRED** — 17.2 GiB reclaimed on GPU1 (free now ~19.4 GiB). +- **🟢 Both Anaheim IPsec tunnels run AES-128** (make-before-break, neither dropped waiting on a far end). NH3 245→**270 Mbit/s**, ESH 268→**304**. The per-flow ceiling is **the UDM's software AES-CBC, not the FortiGate** — proven by an A/B/A cipher swap at identical CPU. FortiGate proposal lists left permissive so a peer reverting cannot strand a tunnel. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` -- **⚠️ THE `sec` DEGENERATION QUESTION IS STILL OPEN AND CONFOUNDED.** Engine and drafter changed together; **the isolating experiment is MTP k=3 on `e9d1398d`** — still not run. Operator ruling stands: degeneration lives in the **un-fixed vLLM**, not the weights; the MTP-head hypothesis is **retracted**. Both prior sightings are n=1 and are NOT evidence. `gen` remains on the old nightly, untouched, gated on that experiment. +- **🟢 Scriberr transcription LIVE** — `scriberr.ana.internal:8080`, ana-ml2 **GPU1**, all seven backends up. Image is **built locally** (upstream's Blackwell image was never published); runs as **uid 10001**; needs `UV_LINK_MODE=copy`. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md` -- **🟢 ana-ml2 now mounts `/mnt/smithy`** (nh3-nas) **ro + soft, NOT in fstab** — needs a manual remount after reboot. For brokkr's R47 CPU work. Reads 24.7 MB/s sequential vs 98.3 on nh3-dev (that gap is the tunnel above), but **45 files/s vs 34** — small-file work is genuinely faster there. → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md` +- **🟢 ESH DNS fixed at the IPv6 layer; the naming scheme is real.** The UDM advertised *itself* as the RDNSS resolver and macOS preferred it, so `.internal` failed. RDNSS now **redirected** to AdGuard on all client VLANs, v4 secondary moved off the UDM. All three `esh-server` Linux hosts carry `4411:B105` ("FOR ALL BIOS"). ⚠ Proxmox bridges need `accept_ra=2`. → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md` -- **🟢 ESH IS DUAL-STACK; the v4 static is a Cityside ticket.** IPv6 live on `esh-userland` and `esh-server` from a delegated `/56`. v4 remains CGNAT and a full gateway reboot proved the purchased static is **not provisioned** — carrier ticket, nothing left locally. NH3 stays v6-off deliberately. Flat-zone lateral-movement finding **parked, id 44**. +- **🟢 ESH IPv6 LAN naming scheme has a real home:** `docs/pfi/ipv6-naming-scheme.md`. It was once a single persistent-memory line that a `memory: snapshot` (`837fa36`) deleted; recovering it took a transcript hunt. **It is a convention, not temporal state — the doc is the home now; never let a memory line become the only copy again.** -- **🟢 OTHER SERVICES.** `hrafn` browser-fetch adopted on ana-docker (infra-ops owns uptime; CI now genuinely deploys). speaches ASR live irv-ml1:8204. Open WebUI esh-docker-vm:3211 — **Lobe retirement still the operator's call**. Booth gained kept-board deletion + per-row link pruning. `pfi` gitea org created; claude-bot is an Owner and can create repos self-serve. +- **🟢 SEAT MAP (unchanged this session).** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, **7 aliases**. **`char-rp`** = MeroMero-v2 dual-mode, GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC on DFlash2, GPU1 :8019 — now sharing GPU1 with Scriberr. `selene` retired. -- **⏳ OPEN ELSEWHERE:** MTP-k3 isolating experiment; upstream vLLM issue to file (operator's GitHub identity); Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift; `/tank` DEGRADED **70+ days**; Worldtree **#411** debug-room litter; bridge/engine agent-roster drift on both WT instances; brokkr's `gen` vs trained-reward-model bake-off (theirs to initiate). **Working tree is clean and pushed** through `0ad332b`. +- **⚠️ THE `sec` DEGENERATION QUESTION IS STILL OPEN AND CONFOUNDED.** Engine and drafter changed together; **the isolating experiment is MTP k=3 on `e9d1398d`** — still not run. Operator ruling stands: degeneration lives in the **un-fixed vLLM**, not the weights; the MTP-head hypothesis is **retracted**. Both sightings are n=1. `gen` remains on the old nightly, gated on that experiment. + +- **🟢 ana-ml2 mounts `/mnt/smithy`** (nh3-nas) **ro + soft, NOT in fstab** — manual remount after reboot. `nconnect=8` is approved but **deliberately not applied** (see Recent decisions). → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md` + +- **🟢 ESH IS DUAL-STACK; the v4 static is a Cityside ticket.** IPv6 live on `esh-userland` and `esh-server` from a delegated `/56`. v4 remains CGNAT; the purchased static is **not provisioned** — carrier ticket. **NH3 stays v6-off — now by explicit ruling, not just inertia.** + +- **🟢 OTHER SERVICES.** `hrafn` browser-fetch on ana-docker. **`speaches` on irv-ml1 is STOPPED** (stack retained, one command to restart) — Eyra was abandoned, leaving it no consumer. Open WebUI esh-docker-vm:3211 — **Lobe retirement still the operator's call**. `pfi` gitea org exists; claude-bot is an Owner. + +- **⏳ OPEN ELSEWHERE:** MTP-k3 isolating experiment; upstream vLLM issue to file (operator's GitHub identity); Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift; `/tank` DEGRADED **70+ days**; Worldtree **#411** debug-room litter; bridge/engine agent-roster drift on both WT instances; brokkr's `gen` vs trained-reward-model bake-off. **Commits are local and unpushed** through `d419b11` — push is the operator's call. ## Recent decisions -- `[2026-08-23]` **Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit — not WireGuard, not CPU, not the fibre.** Cipher change proposed and operator-signalled; **execution pending, untracked by operator choice.** → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` +- `[2026-08-24]` **AES-128 adopted on both Anaheim tunnels; the per-flow ceiling root-caused to the UDM's software AES-CBC, exonerating the FortiGate.** Proven by an A/B/A cipher swap at identical CPU — hardware offload is not cipher-cost-sensitive. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` +- `[2026-08-24]` **ana-gw's public admin surface closed to zero open ports, ACME listener included.** Two of my diagnoses were wrong first (an "ISP proxy" that was the FortiGate, and an "all-port VIP" alarm that was a parser gap) — both from reading config instead of measuring from outside. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md` +- `[2026-08-24]` **Scriberr deployed on ana-ml2 GPU1, image built from source.** Three upstream bugs: the Blackwell image was never published, it must run as uid 10001, and `UV_LINK_MODE=copy` is required or two backends fail silently. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md` +- `[2026-08-24]` **ESH DNS fixed at the IPv6 layer and the naming scheme went live on three hosts.** UniFi's RDNSS cannot be disabled but CAN be redirected — the field is only honoured when an explicit server is given. → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md` +- `[2026-08-24]` **`speaches` on irv-ml1 stopped, stack retained** — Eyra was abandoned pre-implementation (Scriberr covers the need), leaving it no consumer. Disposition confirmed to eyra-dev; one command to restart. Tracked at althing thread `01M0RRJX8GPZEBDHF1E3W18RZF`. +- `[2026-08-24]` **esh-vm-db brought onto the fleet infra-ops identity and given its first vaulted credential.** It previously had none: root and infra-ops refused key auth and `lkraven`'s sudo wanted a password nobody held, leaving `qm guest exec` from the hypervisor as the only privileged path. Break-glass root password at `secret get esh-vm-db/root-breakglass-password` (console-only; plaintext never crossed the wire — only its SHA-512 hash did). +- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction.** brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread `01M0R46SFYF83099N16WD67KGD`. +- `[2026-08-23]` **Anaheim's IPsec tunnel ceiling — investigated, then CLOSED 2026-08-24.** The 25%-of-2-Gbps framing was wrong (NH3's uplink is 1 Gbps); AES-GCM proved impossible; AES-128 landed instead. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` - `[2026-08-23]` **selene retired after losing a head-to-head on its own job; `chat-judge` moved to gen, the model name 404s by design.** Also surfaced that **7 aliases share one seat** — cross-checking between them is an echo, which caught a real defect in brokkr's 46k-exposure R47 gate. → `persistent-memory.d/2026-08-23-selene-retired-alias-collision.md` - `[2026-08-23]` **hrafn adopted; its CI reported green for its whole life while deploying nothing.** A staging dir inside the rsync target destroyed its own source mid-copy; the deeper fault was verify steps that asserted uptime, never content. → `persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md` - `[2026-08-23]` **Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed to personal.** Includes the `:8009`-is-demo port trap that an IP-only fix would have walked into. → `persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md` @@ -265,6 +279,9 @@ no longer deployed sidecars here. See Recent decisions.) _209 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-08-24]` **AES-GCM on the Anaheim tunnels — impossible, not merely hard.** UniFi's manual site-to-site IPsec implements no AEAD cipher at all: eight GCM spellings rejected `api.err.InvalidPayload` against a passing `aes256` control. Blocks both tunnels since both far ends are UDMs. Accepted enum is `aes128/aes192/aes256/3des` — and 3DES is *slower* (no ARM instructions, 64-bit blocks), so AES-128 is the floor. +- `[2026-08-24]` **Pointing the UDM's `wan_dns1` at AdGuard — silently ignored.** It persists and reads back correctly but the LAN-facing forwarder never uses it; proven with fresh uncached ad domains (AdGuard answers `0.0.0.0`, the UDM returned real IPs). Reverted rather than left in place. +- `[2026-08-24]` **A multi-DUID DHCPv6 VM to claim NH3's seven unclaimed /64s — declined by the operator.** The BGW has no IP-passthrough (confirmed, we hold admin), so the only route needs re-cabling, split-stack routing and **rebuilding the entire v6 firewall policy off the UDM**. The prefixes are easy; the firewall rebuild is why nobody wants them. Do not re-raise on "there are seven free prefixes". - `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.** - `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx.