memory: snapshot — Anaheim tunnels on AES-128, ana-gw admin closed, Scriberr live, ESH DNS fixed
Rewrites the in-flight section, which had gone stale in one place (speaches is stopped, not live) and did not reflect a session in which everything opened was also closed. Three detail files carry the bulk: the FortiGate's public surface reduced to nothing including the ACME listener, Scriberr's deployment and the three upstream defects it required working around, and the ESH DNS repair that also made the IPv6 naming scheme real on three hosts. The tunnel entry in the decisions log described a cipher change as proposed and pending; it has since been attempted, found impossible in the form intended, and resolved differently, so the line now points at the outcome rather than the intention. New decisions cover the cipher adoption and the finding that the per-flow ceiling belongs to the UniFi gateway's software cipher rather than the firewall, the administrative closure, the transcription deployment, the DNS repair, the retirement of an ASR service whose only consumer was abandoned, and a database VM that until now had no fleet identity and no vaulted credential at all. Three approaches are recorded as abandoned: an AEAD cipher the far end cannot express, an upstream DNS setting the gateway accepts and ignores, and a scheme to claim unused delegated prefixes that founders not on the prefixes but on having to rebuild a firewall policy to use them.
This commit is contained in:
@@ -0,0 +1,50 @@
|
||||
# [2026-08-24] ana-gw public admin surface closed to zero, ACME listener included
|
||||
|
||||
WAN admin was opened at the start of the session as a cutover contingency
|
||||
("so I don't have to drive down there"), then closed again on operator
|
||||
instruction once the AES-128 work landed. Net result: **the FortiGate's WAN
|
||||
address now exposes no TCP port at all.**
|
||||
|
||||
## Final state
|
||||
|
||||
External scan of `38.120.12.42`, 55 ports: **nothing open**. Verified from two
|
||||
sites. `wan1 allowaccess` = `ping`; `infra-ops` trusthost back to `10.0.0.0/8`.
|
||||
|
||||
**Consequence to hold: there is no out-of-band path to ana-gw.** If both tunnels
|
||||
drop it is console-only. Re-open is two one-liners (allowaccess + trusthost) —
|
||||
both are recorded in auto-memory `reference_fortigate_ana_gw_access`.
|
||||
|
||||
## Port 80 was the FortiOS ACME listener, and I got it wrong first
|
||||
|
||||
`38.120.12.42:80` answered a bare 403 (`ACME Access Only`, 101 bytes) with
|
||||
`allowaccess` set to ping only. First diagnosis — "an ISP transparent proxy" —
|
||||
was **wrong**, and the reason is worth keeping:
|
||||
|
||||
> The sniffer filter was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
|
||||
> matches inbound only**, so outbound SYN-ACKs were excluded *by construction*,
|
||||
> and concluding "the box sends no SYN-ACK" from that capture was unsound.
|
||||
|
||||
Re-run bidirectionally (`host … and tcp port 80`) it immediately showed
|
||||
`wan1 out 38.120.12.42.80 -> <scanner>: syn ack`. **Rule: to test whether a box
|
||||
*answers*, the filter must be bidirectional.**
|
||||
|
||||
The listener is opened by `config system acme / set interface "wan1"` and
|
||||
**bypasses `allowaccess` by design** — FortiOS needs port 80 for HTTP-01. It
|
||||
was disabled (`config system acme / unset interface`); the LE cert (`ana-fw.pfi`,
|
||||
valid to 2026-10-27) is untouched and simply stops renewing, which is fine
|
||||
because WAN admin is closed and the box is being replaced.
|
||||
|
||||
## Retracted in the same pass: the "four all-port VIPs" alarm
|
||||
|
||||
Claimed four VIPs were unrestricted all-port static NAT. **False.** A FortiOS
|
||||
VIP is scoped **two** ways — `portforward`+`extport`, *or* a `service` binding
|
||||
on the VIP object — and only the first was checked. All 14 VIPs are scoped;
|
||||
`Rustdesk` is TCP 21115–21119, `ssh-mapped-2223` is TCP 2223 only.
|
||||
|
||||
Ground-truth external scan of all six public IPs is recorded in
|
||||
`reference_fortigate_ana_gw_access`. Configured-but-dead: `:8443`
|
||||
(mattermost-calls), `:8444` (webdav-nas), `:8880` (Kokoro-In) — tidy-up
|
||||
candidates for the OPNsense translation, not exposure.
|
||||
|
||||
**Lesson, twice in one session: measure from outside instead of parsing config.**
|
||||
Both wrong answers came from a filter that answered a different question.
|
||||
@@ -0,0 +1,77 @@
|
||||
# [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live
|
||||
|
||||
Reported as "`scriberr.ana.internal` doesn't resolve on my Mac, and nslookup
|
||||
shows an IPv6 DNS server." Operator's diagnosis was right; the fix took three
|
||||
wrong turns worth recording.
|
||||
|
||||
## Root cause
|
||||
|
||||
`esh-userland` has IPv6 PD with RA at `pref high`, and the UDM advertises
|
||||
**itself** as the resolver via RDNSS. macOS honours RDNSS and prefers it over
|
||||
the DHCPv4-supplied resolver, so queries went to the UDM — which does not know
|
||||
`.internal` — and returned NXDOMAIN. AdGuard was never consulted.
|
||||
|
||||
Two adjacent gaps found while there: `esh-userland`'s **secondary** v4 resolver
|
||||
was `10.0.10.1` (the UDM itself), and `esh-server` had **DNS handout disabled
|
||||
entirely**, so every host there got the UDM and could never resolve `.internal`
|
||||
— esh-docker-vm was living proof.
|
||||
|
||||
## The three wrong turns
|
||||
|
||||
1. **`dhcpdv6_dns_auto=false` alone does nothing.** It is only honoured **when an
|
||||
explicit server is supplied**. Setting it bare looks like a no-op and invites
|
||||
the conclusion that the field is inert — which I drew, wrongly. Despite the
|
||||
`dhcpdv6_` prefix it *does* drive the RA's RDNSS option on a SLAAC network.
|
||||
2. **`wan_dns1` is NOT used by the UDM's LAN-facing forwarder.** Setting it to
|
||||
AdGuard persists, reads back, and changes nothing. Proven with **fresh
|
||||
uncached ad domains** — AdGuard blocklists answer `0.0.0.0`, the UDM returned
|
||||
real IPs. Reverted.
|
||||
3. **`force-provision` returns `rc:ok` and proves nothing** — consistent with the
|
||||
known `cmd/devmgr` behaviour.
|
||||
|
||||
Every failed attempt returned `rc: ok`. **Verify by observed effect.** RAs were
|
||||
probed with a stdlib raw-socket Router Solicitation parsing option type 25
|
||||
(`rdisc6`/`tcpdump` were both absent; nothing was installed).
|
||||
|
||||
## What landed
|
||||
|
||||
RDNSS **redirected** rather than disabled — better than switching it off:
|
||||
|
||||
| VLAN | v4 | v6 RDNSS |
|
||||
|---|---|---|
|
||||
| `esh-userland` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
|
||||
| `esh-server` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
|
||||
|
||||
The v4 secondary moved from the UDM to the **NH3 AdGuard** — reachable over
|
||||
Site Magic and authoritative for the zone. ⚠ **A secondary only fails over on
|
||||
SILENCE, not on wrong answers**: NXDOMAIN is a *successful* answer, the client
|
||||
accepts it and never retries. A secondary that doesn't know your private zone is
|
||||
a coin flip, not a spare tyre. `esh-cameras` deliberately untouched — routing
|
||||
camera DNS through AdGuard's filtering risks their cloud features.
|
||||
|
||||
## The naming scheme became real
|
||||
|
||||
The resolver address is the scheme's first live use, replacing a MAC-derived
|
||||
SLAAC address that would break on a NIC change. All three `esh-server` Linux
|
||||
hosts now carry `4411:B105` ("FOR ALL BIOS"):
|
||||
|
||||
```
|
||||
esh-docker-vm …:4411:b105:50:45 esh-pve-nas …:4411:b105:50:55
|
||||
esh-vm-db …:4411:b105:50:60
|
||||
```
|
||||
|
||||
Applied by an `if-up.d` hook that **derives the prefix at runtime** (self-heals
|
||||
on re-delegation), backgrounds itself with a retry (SLAAC may not have landed;
|
||||
a blocking hook would stall bring-up on a headless box), and adds nothing to
|
||||
existing config. **Not** an `iface … inet6 static` stanza — on Debian that sets
|
||||
`accept_ra=0` and would strand the host.
|
||||
|
||||
⚠ **Proxmox bridges need `accept_ra=2`.** `esh-pve-nas` had link-local only
|
||||
despite every sysctl looking right: `vmbr0.forwarding=1`, and the kernel ignores
|
||||
RAs on a forwarding interface unless `accept_ra` is explicitly `2`. Fixed with
|
||||
`accept_ra_defrtr=0` alongside, so it takes the prefix but **declines the default
|
||||
route** — an IPv6 identity with no change to a hypervisor's routing. Expect this
|
||||
on every Proxmox node when its LAN gets v6.
|
||||
|
||||
Canonical: `docs/pfi/ipv6-naming-scheme.md`. UniFi limits:
|
||||
auto-memory `reference_unifi_dns_rdnss_limits`.
|
||||
@@ -0,0 +1,46 @@
|
||||
# [2026-08-24] Scriberr transcription deployed on ana-ml2, GPU1
|
||||
|
||||
Self-hosted audio/video transcription + diarization. Operator chose GPU
|
||||
placement over ana-docker (8 cores shared with 50 containers, 37 GB disk)
|
||||
against ana-ml2's 96 cores, `/tank`'s terabytes and GPU1's headroom.
|
||||
|
||||
**Live:** `http://scriberr.ana.internal:8080` (DNS alias added), health `healthy`,
|
||||
all seven backends up, zero failures: `whisperx pyannote sortformer parakeet
|
||||
canary voxtral openai`. ~30 GB of weights on `/tank`.
|
||||
|
||||
Stack: `stacks/scriberr/`. Full gotcha list in auto-memory
|
||||
`reference_scriberr_ana_ml2`.
|
||||
|
||||
## Three upstream bugs, none of them ours
|
||||
|
||||
**1. The Blackwell image does not exist.** Upstream's README documents
|
||||
`scriberr-cuda-blackwell`; GHCR has **no tags for it**. Published
|
||||
`scriberr-cuda` covers sm_61–sm_89 only — on these sm_120 cards it fails or
|
||||
silently drops to CPU. The real sm_120 path is `Dockerfile.cuda.12.9`
|
||||
(CUDA 12.9.1, cu128 torch), **built from source**. Do not "simplify" the compose
|
||||
back to the published image.
|
||||
|
||||
**2. It must run as uid 10001, not 1000** — and the error lies:
|
||||
`unable to open database file: out of memory (14)`. Error 14 is
|
||||
`SQLITE_CANTOPEN`, not an OOM, on a box with 566 GB RAM. That Dockerfile creates
|
||||
`appuser` at 10001 (Ubuntu 24.04 owns uid 1000 as `ubuntu`) and chowns `/app` to
|
||||
it, while the entrypoint's PUID remap covers only the data dirs.
|
||||
**Isolated by elimination**: SQLite writes fine to `/tank` as 1000 → not the
|
||||
mount; fails on a plain named volume too → not the storage; the **published CPU
|
||||
image works at PUID=1000** because there `appuser` *is* 1000.
|
||||
Generalisable: *when a container "permission" bug appears, compare the uid the
|
||||
image was BUILT for against the uid you are RUNNING as.*
|
||||
|
||||
**3. `UV_LINK_MODE=copy` is required.** Scriberr builds each backend's Python env
|
||||
with `uv` at start; uv's reflink mode fails on overlayfs+ZFS with
|
||||
`Failed to clone … Resource temporarily unavailable (os error 11)`. **Partial
|
||||
failure** — WhisperX and PyAnnote came up and the app looked fine while Parakeet
|
||||
and Sortformer were silently absent. Occurrences 2 → 0 after the fix.
|
||||
|
||||
## Related
|
||||
|
||||
`speaches` on irv-ml1 **stopped** the same day (stack retained, one command to
|
||||
restart): Eyra was abandoned pre-implementation because Scriberr covers the need,
|
||||
leaving it with no consumer. Scriberr runs its **own** WhisperX in-container and
|
||||
is **not** a speaches consumer. Idle footprint at stop was 274 MiB, not the
|
||||
~5.9 GB quoted — that figure is the loaded-model working set.
|
||||
+28
-11
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-08-23_
|
||||
_Last updated: 2026-08-24_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -108,26 +108,40 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
(no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
|
||||
## Current state / in-flight
|
||||
|
||||
- **🟢 ESH IPv6 LAN naming scheme — now a real doc, and LIVE on one host.** Six ESH LANs carry an 8-hex-digit phrase in the interface identifier (`esh-server`=`4411:B105` "FOR ALL BIOS", `esh-mgmt`=`15DA:B055` "IS DA BOSS", `esh-userland`=`CAFE:4411`, `esh-iot`=`4DBA:D107`, `esh-cameras`=`1533:FACE5`, `Default`=`4BA5:3417`, plus `4411:DBAD` reserved for a future DMZ). **Canonical: [`docs/pfi/ipv6-naming-scheme.md`](docs/pfi/ipv6-naming-scheme.md)** — full table, address structure, what UniFi can and cannot name, and the `if-up.d` recipe. ⚠ This entry was once the ONLY record and a `memory: snapshot` (`837fa36`) deleted it; it took a transcript hunt to recover. It is a convention, not temporal state — **the doc is now the home; do not let this line become the only copy again.** First live use 2026-08-24: AdGuard on esh-docker-vm at `2607:73c0:402:1d02:4411:b105:50:45`, which is what the ESH UDM advertises to clients as its IPv6 resolver.
|
||||
- **🔒 ana-gw WAN admin is CLOSED again (2026-08-23, operator-directed) and the FortiGate is scheduled for replacement.** The cutover contingency window is over: `wan1 allowaccess` is back to **`ping` only** (https + ssh removed) and `infra-ops` trusthost is back to **10.0.0.0/8 only** — verified from two sites that 443 and 22 are closed, and that management still works over the tunnel at 10.250.0.1. **There is no longer any out-of-band path to ana-gw**; if both tunnels drop, it is console-only. Re-open = `set allowaccess ping https` on wan1 plus widening the infra-ops trusthost (both one-liners, recorded in auto-memory). ⚠ **Port 80 on 38.120.12.42 is still open and it IS the FortiGate** — the FortiOS **ACME HTTP-01 challenge listener**, opened by `config system acme / set interface "wan1"`, which **bypasses `allowaccess` by design**. Body is literally `ACME Access Only` (403, 101 bytes) for every path; confirmed by sniffer showing `wan1 out 38.120.12.42.80 -> ...: syn ack`. Not a DNAT (only two VIPs exist on .42: Kokoro `:8880`, wg `:31337`) and not an admin surface — it serves only challenge tokens. Closing it = `config system acme` unset interface, which also ends cert renewal.
|
||||
_As of 2026-08-24 — a very long ops session; everything it opened, it closed. Nothing is mid-flight. Next work is the **homepage** dashboard._
|
||||
|
||||
- **✅ CLOSED 2026-08-23: the Anaheim tunnel "problem" is mostly a measurement artefact, and the AES-GCM cutover is impossible.** Operator authorised the cutover; it was attempted NH3-side-first and **cannot be done — UniFi's manual site-to-site IPsec implements no AES-GCM** (8 spellings rejected `api.err.InvalidPayload` against a passing `aes256` control; accepted enum is `aes128/aes192/aes256/3des` only). This blocks the ESH tunnel too, since both far ends are UDMs. The framing was also wrong twice over: **NH3's uplink is 1 Gbps** (not Anaheim's 2 Gbps — that is the real ceiling), and the tunnel does **692 Mbit/s at 8 streams** (the original stopped at 4 and reported ~550). Against WireGuard on the same UDM and uplink, the gap collapses from 2.3× at one stream to **15% at eight** — so re-architecting onto WireGuard is not worth it. Real constraint = **per-stream ~245 Mbit/s**, both endpoints idle. **Standing mitigation: parallelise bulk transfers (2.8× for free); for single-stream NFS use `nconnect=N`** — the `/mnt/smithy` mount on ana-ml2 at 24.7 MB/s is exactly this case and is the obvious test. FortiGate phase2 `pfi-ana-nh3` was left widened to `aes256-sha1 aes256gcm` (inert while the peer offers only CBC); UDM verified byte-identical to its pre-change snapshot. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
|
||||
- **🔒 ana-gw's public admin surface is now ZERO open TCP ports**, and the box is scheduled for replacement by **OPNsense on a Dell R420** (which brings WireGuard onto the edge — so the downstream-WireGuard-VM design is moot, do not scope it). WAN admin was opened as a cutover contingency then closed on instruction; the FortiOS **ACME listener** on :80 was disabled too. **No out-of-band path to ana-gw remains** — if both tunnels drop it is console-only; re-open is two one-liners in auto-memory. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md`
|
||||
|
||||
- **🟢 SEAT MAP (unchanged this session except selene).** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, **now 7 aliases** (see the collision note). **`char-rp`** = MeroMero-v2 dual-mode, GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC on DFlash2, GPU1 :8019. **`selene` RETIRED** — 17.2 GiB reclaimed on GPU1 (free now ~19.4 GiB).
|
||||
- **🟢 Both Anaheim IPsec tunnels run AES-128** (make-before-break, neither dropped waiting on a far end). NH3 245→**270 Mbit/s**, ESH 268→**304**. The per-flow ceiling is **the UDM's software AES-CBC, not the FortiGate** — proven by an A/B/A cipher swap at identical CPU. FortiGate proposal lists left permissive so a peer reverting cannot strand a tunnel. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
|
||||
|
||||
- **⚠️ THE `sec` DEGENERATION QUESTION IS STILL OPEN AND CONFOUNDED.** Engine and drafter changed together; **the isolating experiment is MTP k=3 on `e9d1398d`** — still not run. Operator ruling stands: degeneration lives in the **un-fixed vLLM**, not the weights; the MTP-head hypothesis is **retracted**. Both prior sightings are n=1 and are NOT evidence. `gen` remains on the old nightly, untouched, gated on that experiment.
|
||||
- **🟢 Scriberr transcription LIVE** — `scriberr.ana.internal:8080`, ana-ml2 **GPU1**, all seven backends up. Image is **built locally** (upstream's Blackwell image was never published); runs as **uid 10001**; needs `UV_LINK_MODE=copy`. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md`
|
||||
|
||||
- **🟢 ana-ml2 now mounts `/mnt/smithy`** (nh3-nas) **ro + soft, NOT in fstab** — needs a manual remount after reboot. For brokkr's R47 CPU work. Reads 24.7 MB/s sequential vs 98.3 on nh3-dev (that gap is the tunnel above), but **45 files/s vs 34** — small-file work is genuinely faster there. → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md`
|
||||
- **🟢 ESH DNS fixed at the IPv6 layer; the naming scheme is real.** The UDM advertised *itself* as the RDNSS resolver and macOS preferred it, so `.internal` failed. RDNSS now **redirected** to AdGuard on all client VLANs, v4 secondary moved off the UDM. All three `esh-server` Linux hosts carry `4411:B105` ("FOR ALL BIOS"). ⚠ Proxmox bridges need `accept_ra=2`. → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md`
|
||||
|
||||
- **🟢 ESH IS DUAL-STACK; the v4 static is a Cityside ticket.** IPv6 live on `esh-userland` and `esh-server` from a delegated `/56`. v4 remains CGNAT and a full gateway reboot proved the purchased static is **not provisioned** — carrier ticket, nothing left locally. NH3 stays v6-off deliberately. Flat-zone lateral-movement finding **parked, id 44**.
|
||||
- **🟢 ESH IPv6 LAN naming scheme has a real home:** `docs/pfi/ipv6-naming-scheme.md`. It was once a single persistent-memory line that a `memory: snapshot` (`837fa36`) deleted; recovering it took a transcript hunt. **It is a convention, not temporal state — the doc is the home now; never let a memory line become the only copy again.**
|
||||
|
||||
- **🟢 OTHER SERVICES.** `hrafn` browser-fetch adopted on ana-docker (infra-ops owns uptime; CI now genuinely deploys). speaches ASR live irv-ml1:8204. Open WebUI esh-docker-vm:3211 — **Lobe retirement still the operator's call**. Booth gained kept-board deletion + per-row link pruning. `pfi` gitea org created; claude-bot is an Owner and can create repos self-serve.
|
||||
- **🟢 SEAT MAP (unchanged this session).** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, **7 aliases**. **`char-rp`** = MeroMero-v2 dual-mode, GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC on DFlash2, GPU1 :8019 — now sharing GPU1 with Scriberr. `selene` retired.
|
||||
|
||||
- **⏳ OPEN ELSEWHERE:** MTP-k3 isolating experiment; upstream vLLM issue to file (operator's GitHub identity); Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift; `/tank` DEGRADED **70+ days**; Worldtree **#411** debug-room litter; bridge/engine agent-roster drift on both WT instances; brokkr's `gen` vs trained-reward-model bake-off (theirs to initiate). **Working tree is clean and pushed** through `0ad332b`.
|
||||
- **⚠️ THE `sec` DEGENERATION QUESTION IS STILL OPEN AND CONFOUNDED.** Engine and drafter changed together; **the isolating experiment is MTP k=3 on `e9d1398d`** — still not run. Operator ruling stands: degeneration lives in the **un-fixed vLLM**, not the weights; the MTP-head hypothesis is **retracted**. Both sightings are n=1. `gen` remains on the old nightly, gated on that experiment.
|
||||
|
||||
- **🟢 ana-ml2 mounts `/mnt/smithy`** (nh3-nas) **ro + soft, NOT in fstab** — manual remount after reboot. `nconnect=8` is approved but **deliberately not applied** (see Recent decisions). → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md`
|
||||
|
||||
- **🟢 ESH IS DUAL-STACK; the v4 static is a Cityside ticket.** IPv6 live on `esh-userland` and `esh-server` from a delegated `/56`. v4 remains CGNAT; the purchased static is **not provisioned** — carrier ticket. **NH3 stays v6-off — now by explicit ruling, not just inertia.**
|
||||
|
||||
- **🟢 OTHER SERVICES.** `hrafn` browser-fetch on ana-docker. **`speaches` on irv-ml1 is STOPPED** (stack retained, one command to restart) — Eyra was abandoned, leaving it no consumer. Open WebUI esh-docker-vm:3211 — **Lobe retirement still the operator's call**. `pfi` gitea org exists; claude-bot is an Owner.
|
||||
|
||||
- **⏳ OPEN ELSEWHERE:** MTP-k3 isolating experiment; upstream vLLM issue to file (operator's GitHub identity); Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift; `/tank` DEGRADED **70+ days**; Worldtree **#411** debug-room litter; bridge/engine agent-roster drift on both WT instances; brokkr's `gen` vs trained-reward-model bake-off. **Commits are local and unpushed** through `d419b11` — push is the operator's call.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-08-23]` **Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit — not WireGuard, not CPU, not the fibre.** Cipher change proposed and operator-signalled; **execution pending, untracked by operator choice.** → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
|
||||
- `[2026-08-24]` **AES-128 adopted on both Anaheim tunnels; the per-flow ceiling root-caused to the UDM's software AES-CBC, exonerating the FortiGate.** Proven by an A/B/A cipher swap at identical CPU — hardware offload is not cipher-cost-sensitive. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
|
||||
- `[2026-08-24]` **ana-gw's public admin surface closed to zero open ports, ACME listener included.** Two of my diagnoses were wrong first (an "ISP proxy" that was the FortiGate, and an "all-port VIP" alarm that was a parser gap) — both from reading config instead of measuring from outside. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md`
|
||||
- `[2026-08-24]` **Scriberr deployed on ana-ml2 GPU1, image built from source.** Three upstream bugs: the Blackwell image was never published, it must run as uid 10001, and `UV_LINK_MODE=copy` is required or two backends fail silently. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md`
|
||||
- `[2026-08-24]` **ESH DNS fixed at the IPv6 layer and the naming scheme went live on three hosts.** UniFi's RDNSS cannot be disabled but CAN be redirected — the field is only honoured when an explicit server is given. → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md`
|
||||
- `[2026-08-24]` **`speaches` on irv-ml1 stopped, stack retained** — Eyra was abandoned pre-implementation (Scriberr covers the need), leaving it no consumer. Disposition confirmed to eyra-dev; one command to restart. Tracked at althing thread `01M0RRJX8GPZEBDHF1E3W18RZF`.
|
||||
- `[2026-08-24]` **esh-vm-db brought onto the fleet infra-ops identity and given its first vaulted credential.** It previously had none: root and infra-ops refused key auth and `lkraven`'s sudo wanted a password nobody held, leaving `qm guest exec` from the hypervisor as the only privileged path. Break-glass root password at `secret get esh-vm-db/root-breakglass-password` (console-only; plaintext never crossed the wire — only its SHA-512 hash did).
|
||||
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction.** brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread `01M0R46SFYF83099N16WD67KGD`.
|
||||
- `[2026-08-23]` **Anaheim's IPsec tunnel ceiling — investigated, then CLOSED 2026-08-24.** The 25%-of-2-Gbps framing was wrong (NH3's uplink is 1 Gbps); AES-GCM proved impossible; AES-128 landed instead. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
|
||||
- `[2026-08-23]` **selene retired after losing a head-to-head on its own job; `chat-judge` moved to gen, the model name 404s by design.** Also surfaced that **7 aliases share one seat** — cross-checking between them is an echo, which caught a real defect in brokkr's 46k-exposure R47 gate. → `persistent-memory.d/2026-08-23-selene-retired-alias-collision.md`
|
||||
- `[2026-08-23]` **hrafn adopted; its CI reported green for its whole life while deploying nothing.** A staging dir inside the rsync target destroyed its own source mid-copy; the deeper fault was verify steps that asserted uptime, never content. → `persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md`
|
||||
- `[2026-08-23]` **Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed to personal.** Includes the `:8009`-is-demo port trap that an IP-only fix would have walked into. → `persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md`
|
||||
@@ -265,6 +279,9 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
_209 older entries archived to archival-memory.md._
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-08-24]` **AES-GCM on the Anaheim tunnels — impossible, not merely hard.** UniFi's manual site-to-site IPsec implements no AEAD cipher at all: eight GCM spellings rejected `api.err.InvalidPayload` against a passing `aes256` control. Blocks both tunnels since both far ends are UDMs. Accepted enum is `aes128/aes192/aes256/3des` — and 3DES is *slower* (no ARM instructions, 64-bit blocks), so AES-128 is the floor.
|
||||
- `[2026-08-24]` **Pointing the UDM's `wan_dns1` at AdGuard — silently ignored.** It persists and reads back correctly but the LAN-facing forwarder never uses it; proven with fresh uncached ad domains (AdGuard answers `0.0.0.0`, the UDM returned real IPs). Reverted rather than left in place.
|
||||
- `[2026-08-24]` **A multi-DUID DHCPv6 VM to claim NH3's seven unclaimed /64s — declined by the operator.** The BGW has no IP-passthrough (confirmed, we hold admin), so the only route needs re-cabling, split-stack routing and **rebuilding the entire v6 firewall policy off the UDM**. The prefixes are easy; the firewall rebuild is why nobody wants them. Do not re-raise on "there are seven free prefixes".
|
||||
- `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.**
|
||||
- `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user