From b6340519bc5bd702bc363c720925eaec7f977fb8 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 23 Aug 2026 13:35:48 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20Anaheim=20tunn?= =?UTF-8?q?el=20at=2025%=20of=20circuit;=20selene=20retired;=20hrafn=20CI?= =?UTF-8?q?=20fixed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Session captured for a context reset. Six new detail entries. THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~550 Mbit/s aggregate against a circuit measured at 2,153 Mbit/s. Not WireGuard (it is IPsec on ana-gw), not CPU (idle), not crypto exhaustion (NPU-offloaded), not the fibre. Both tunnels negotiate aes256-sha1; AES-GCM proposed. Operator signalled authorization; execution pending, untracked by operator choice. Also recorded: selene retired after losing a head-to-head on its own job with chat-judge moved to gen and the model name left to 404; the 7-alias collision on the gen seat that makes cross-alias corroboration an echo; hrafn adopted and its CI found to have been reporting green while deploying nothing for its whole life; all three Worldtree instances de-armed from a 69-day-stale :latest and the Matrix homeserver re-plumbed to personal; every secret-bearing .env on ana-docker tightened to 0600; the pfi org closing the repo-creation half of the credential-migration directive; booth kept-board deletion and link pruning. Two entries under Tried and abandoned: the CI checkout assertion that broke the pipeline twice and was removed, and my proposal to alias a retired model name at a different model, which the operator correctly overruled. Index 271 -> 282 lines, under the 300 cap, so no archival fired. --- .../2026-08-23-ana-docker-env-perms-sweep.md | 34 ++++++++ ...2026-08-23-anaheim-ipsec-tunnel-ceiling.md | 71 ++++++++++++++++ ...26-08-23-hrafn-adopted-ci-frozen-source.md | 63 +++++++++++++++ ...26-08-23-selene-retired-alias-collision.md | 81 +++++++++++++++++++ .../2026-08-23-smithy-mount-ana-ml2.md | 50 ++++++++++++ .../2026-08-23-worldtree-b187-pins-matrix.md | 70 ++++++++++++++++ persistent-memory.md | 29 ++++--- 7 files changed, 389 insertions(+), 9 deletions(-) create mode 100644 persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md create mode 100644 persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md create mode 100644 persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md create mode 100644 persistent-memory.d/2026-08-23-selene-retired-alias-collision.md create mode 100644 persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md create mode 100644 persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md diff --git a/persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md b/persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md new file mode 100644 index 0000000..49bae49 --- /dev/null +++ b/persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md @@ -0,0 +1,34 @@ +# [2026-08-23] Every secret-bearing `.env` on ana-docker tightened to 0600 + +Found while taking uptime ownership of hrafn: its `.env` was mode 0644 with a live +bearer token. Not a hrafn lapse — **0644 was the de facto pattern on the host**. +Eight stacks carried secret-shaped vars in world-readable `.env` files on a box with +four interactive accounts, verified as real exposure by reading one as `nobody`. + +Swept: **vaultwarden, traefik**, beszel, gitea-runner, miniflux, news-digest, +searxng, vor. (hrafn and nevermore were fixed separately the same day.) Six other +stacks already used 0600, so this converged on the existing house pattern rather +than inventing one. Post-sweep the host has **zero** secret-bearing `.env` readable +by `nobody`. + +Playbook: `playbooks/tighten-env-perms.yaml`, one run per stack, re-runnable. + +## The check that matters + +Every run asserts `docker compose config` still renders **as the deploy user** +(`lkraven`), not as root. Checking the mode proves the bits changed; only rendering +as the deploy user proves the next deploy can still resolve its variables. + +## Two gotchas recorded in the playbook + +- **vaultwarden looked like it bind-mounted its `.env`** — which would mean the + *container's* UID reads it and 0600 could break the password vault. It does not: + that `- .env` is under `env_file:`, not `volumes:`. My grep matched the YAML list + item without checking its parent key. The playbook now **refuses** any stack that + genuinely bind-mounts its `.env`, since that case is read by the container UID. +- **elway prompted for a sudo password.** The `ana-docker` ssh alias resolves to + `lkraven`, who needs one; **`infra-ops@10.250.50.70` has NOPASSWD**. `corviduo-dev` + was repointed to infra-ops at some point and `ana-docker` was not. Run elway against + the infra-ops target on this host. + +Commit `a896c0a`. diff --git a/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md new file mode 100644 index 0000000..a198a5d --- /dev/null +++ b/persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md @@ -0,0 +1,71 @@ +# [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit + +The operator noticed site-to-site transfers were slow for a datacenter fiber +handoff and asked whether WireGuard was the limit. It is not WireGuard, and the +circuit is fine. + +## Measured + +``` +ana-docker -> internet, 8 parallel 2,153 Mbit/s <- the 2 Gbps handoff, delivering +ANA <-> NH3 through the tunnel, 4 par. 460 Mbit/s +FortiGate's own recorded peak 554 Mbit/s +ANA <-> NH3, single stream 227 Mbit/s +ANA <-> ESH, single stream 249-265 Mbit/s +ESH <-> NH3 (never touches ana-gw) 545-557 Mbit/s on a SINGLE stream +``` + +Method: stdlib TCP probe (no ssh, no crypto, no compression) between site +endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner +Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling. + +## What it is not + +- **Not WireGuard.** Both Anaheim tunnels are IPsec on ana-gw + (`pfi-ana-nh3` -> 70.230.226.88, `ana-eshudm-dyn` -> the ESH UDM). WireGuard + on ana-wg is remote-access only and is not in this path. Traceroute confirms: + both slow paths have hop 1 = `10.250.50.1` (the FortiGate); the fast + ESH<->NH3 path rides a `192.168.x` Site Magic overlay and never touches it. +- **Not CPU or crypto exhaustion.** FortiGate CPU was **100% idle across all + 8 cores** during the tests, and both live tunnels report `npu_flag=03` with + `dec_npuid=1 enc_npuid=1` — encrypt *and* decrypt are hardware-offloaded. +- **Not a 250 Mbit/s cap.** That was the first number and it is misleading — + single-stream TCP. Four parallel streams doubled it. Quote the aggregate. +- **Not the interface.** wan1: `rxe=0 txe=0 rxd=0 txd=0`, no collisions. + +## Most likely cause + +Both tunnels negotiate **`aes256-sha1`** in phase 1 *and* phase 2 (dhgrp 14, +IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly +faster on **AES-GCM**, which combines encryption and authentication in one +pass. The datasheet IPsec headline for an 80F assumes GCM with large packets, +not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x +shortfall is consistent with that. + +## Not executed + +Changing the proposal is a **production-edge change requiring a matching +change at the far end** (NH3 UDM and the ESH UDM), and each tunnel drops while +it renegotiates. Left for the operator. See the index entry for authorization +state. + +## Immediate mitigation, no config change + +Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a +single stream through IPsec gets 227. **Anything moving bulk data across the +Anaheim link should parallelise** — that alone roughly doubles throughput +today. + +## Practical consequence already observed + +`/mnt/smithy` mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s +from nh3-dev (same file, same mount) — that gap *is* this tunnel, not NFS and +not the NAS. See [[2026-08-23-smithy-mount-ana-ml2]]. + +## Access note + +ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. `sshpass` is absent +on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is +vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list` +prints live ESP session keys** — never paste its output into althing, a +booth, or a commit. diff --git a/persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md b/persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md new file mode 100644 index 0000000..27c8e4f --- /dev/null +++ b/persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md @@ -0,0 +1,63 @@ +# [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing + +`hrafn` — genuine-Chromium browser-fetch behind a REST API, for bot-gated sites +(Reddit first). Built by nevermore-claude on ana-docker, handed to infra-ops for +uptime ownership. Internal-only on `traefik-net`, no host port; consumers reach +`http://hrafn:8080`. Canonical at `stacks/hrafn/`. + +## Intake found a live credential exposure + +`/opt/docker/compose/hrafn/.env` was mode **0644 with a live 57-char bearer token** +— verified as real exposure by reading it as `nobody` on a box with four +interactive accounts. Tightened to 0600. That triggered the wider sweep (see +[[2026-08-23-ana-docker-env-perms-sweep]]). + +## The CI defect — the one worth remembering + +I authored the deploy (elway playbook + gitea workflow) to replace a hand-rsync, +tagging the image with the commit SHA for provenance. nevermore-claude later found +v1.0.0 deploying "green" while the host still served 0.1.0. + +**Root cause was mine and nastier than either hypothesis.** The staging dir was +`$compose_dir/.stage` — **inside** the rsync target. So +`rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted `.stage` from the +destination (absent from the source listing) **during** the transfer, destroying +its own source mid-copy. Reproduced exactly: + +``` +before: app.py="OLD" leftover.txt .stage/app.py="NEW" +after: app.py="OLD" leftover.txt GONE, .stage GONE +``` + +Deletion succeeded, the copy silently did not, rsync exited 0. So the directory +*looked* converged while host source stayed frozen at the first manual rsync — +and because the build's `COPY` inputs never changed, Docker full-cache-hit and +every SHA tag aliased one image. **The provenance the tagging existed to provide +was false for the pipeline's entire life.** + +**The real failure is the verification.** The verify steps asserted the marker, +container health, and a 200 from `/readyz` — all of which pass against a +completely frozen host. None measured *content*. A deploy that reports success +without asserting the bytes changed is verifying an **uptime**, not a deploy. + +## Fixes + +- stage at `/tmp/hrafn-deploy-stage`, outside the target +- CI computes `context_sha256` over the shipped file list; the playbook recomputes + it **on the host after the converge** and fails on mismatch +- compare the running container's `src/**/*.py` against the host's, so a SHA tag + cannot name layers the image lacks +- **compare `*.py` only** — `pip install .` generates `src/*.egg-info/*` inside the + image and `__pycache__` appears at runtime, so a naive `find src -type f` compare + false-fails on every healthy deploy. Verified against a known-good container + before shipping (12 host files, 18 in container, 0 content differences). +- declined `--no-cache`: a cache hit is *correct* when the context is genuinely + unchanged; assert the property rather than brute-force it. + +## Access + +Operator granted claude-bot **write** on `vh/hrafn`, so infra-ops maintains the +pipeline it owns instead of routing patches through the repo holder. `vh/hrafn` is +canonical; `stacks/hrafn/ci/` is a verified mirror. + +Commits `b6924de`, `b001d0c`, `11b9d18`, `b38c369`, `9642952`. diff --git a/persistent-memory.d/2026-08-23-selene-retired-alias-collision.md b/persistent-memory.d/2026-08-23-selene-retired-alias-collision.md new file mode 100644 index 0000000..9e3b004 --- /dev/null +++ b/persistent-memory.d/2026-08-23-selene-retired-alias-collision.md @@ -0,0 +1,81 @@ +# [2026-08-23] selene seat retired after losing a head-to-head; 7 aliases share one seat + +## Why selene went + +Benchmarked against `gen` on selene's own job — 24 designed judge items with +checkable ground truth, pairwise + absolute modes, 3 repeats, run on **both** a +neutral JSON prompt and Selene's **native Atla template** (288 calls, free local). + +``` +neutral JSON selene 20/24 (83%) gen 23/24 (96%) +native Atla selene 21/24 (88%) gen 22/24 (92%) +``` + +gen won on both templates and **selene's best sat below gen's worst**. Selene was +given its own fine-tuned template as a fairness check before any recommendation; +it gained one point, not three. + +**Decisive defect: selene cannot emit "tie"** — 0/2 on both templates, forcing a +winner on every equivalent pair. For eval work that is the case that matters. + +brokkr-smithy-dev independently corroborated from the other end with a **null +control** (an excerpt compared against ITSELF, where tie is definitional): +`chat-judge`(selene) TIE **27/60 = 45%**, gen **60/60 = 100%**; ground-truth +recovery on real-corpus ranking selene **47% — chance** vs gen 94%. My 83-vs-96 +understated it: on a *ranking* task selene was a coin flip. Absolute scoring on +designed items is an easier task than ranking real text — the harness is a +**screen, not a verdict**, and its README says so. + +Reclaimed **17.2 GiB** on ana-ml2 GPU1 (free 1,818 -> 19,450 MiB). + +## The naming rule, restated the hard way + +I proposed repointing `selene-1-mini-8b` at gen and was **correctly overruled**: + +> never repoint a named model at a different model's endpoint — that is +> intentionally misleading + +`chat-judge` is a **role** alias (ADR-0012: consumers bind the capability) and +moved to gen with a deterministic judge profile copied from `image-judge`. +`selene-1-mini-8b` is a **model** name and was removed outright — it now returns +`HTTP 400 Invalid model name`, verified. The discriminator: *does the string +promise a capability, or an identity?* + +## The 7-way alias collision — the finding with the longest reach + +``` +chat-judge classifier gen image-judge +qwen-image-bench summarizer summarizer-large -> qwen3.8-27b-uncensored :8015 +``` + +Also colliding: `gen-frontier`/`gen-frontier-reasoning`/`glm-5.2`/`glm-5.2-reasoning`; +`ext-tts`/`gpt-4o-mini-tts`/`tts-1`/`tts-1-hd`; `reranker`/`reranker-a3-bge-v2-m3`. + +**Cross-checking a result against another alias measures nothing when they are the +same weights — agreement is an echo, not corroboration.** Documented at the head of +`model_list` in the live gateway config, because it belongs where people read it. + +This caught a real defect within hours: brokkr's R47 premium-corpus gate was about +to run ~46,000 record-exposures against `gen` with `summarizer` shortlisted as an +independent second opinion. They pinned the backing model in the preregistration +and dropped the second-alias idea instead. + +## Provenance seam (brokkr's pushback, adopted) + +The gateway returns the **alias** in the response `model` field, not the backing +model — so a per-call guard catches a swap *during* a run and is blind to one +*between* runs. **Role alias for routing, concrete model for provenance.** +`GET :4000/model/info` with the shared key already exposes backing model + +api_base; resolve at run start AND end and void on mismatch. + +## Artifacts + +- Harness kept at `tools/judge-bench/` (`--models` REQUIRED — a stale default + would silently benchmark a retired seat). +- `stacks/selene/` keeps compose + a README explaining the retirement. +- Technique worth stealing, from brokkr: **a control constructed so the correct + answer is DEFINITIONAL rather than judged cannot inherit the designer's error.** + Item vs itself; response vs its own truncation; text vs its own clauses + permuted. Add those before adding more judged items. + +Commits `ca3c984`, `b8a5355`. diff --git a/persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md b/persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md new file mode 100644 index 0000000..e20be6b --- /dev/null +++ b/persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md @@ -0,0 +1,50 @@ +# [2026-08-23] `/mnt/smithy` mounted on ana-ml2 — read-only and SOFT, deliberately not matching nh3-dev + +brokkr-smithy-dev asked for `10.100.50.50:/volume1/smithy` on ana-ml2 to run R47's +CPU-bound corpus pipeline on 96 idle EPYC cores instead of one nh3-dev vCPU. Granted, +with two deliberate deviations from what was requested. + +``` +sudo mount -t nfs4 -o ro,soft,timeo=30,retrans=3,proto=tcp,vers=4.1 \ + 10.100.50.50:/volume1/smithy /mnt/smithy +``` + +The export already permitted ana-ml2 — no DSM change needed. Write is genuinely +refused. + +## Why soft, not hard + +They asked to match nh3-dev's mount, which is `hard`. **nh3-dev is same-site as the +NAS; ana-ml2 is not** — this is cross-site NFS on the box running the fleet's +inference seats. A hard mount turns a link blip into unkillable D-state, and this +fleet has already lost a host that way (esh-docker-vm; only fix was a reboot). Soft +returns EIO, the batch job fails, you rerun it. The soft-mount corruption caveat is a +**write** hazard and this is read-only. Mirrors the existing ESH books mount. + +## Why not in fstab + +Manual only, matching irv-ml1's `/mnt/smithy` precedent. A cross-site NFS entry in +fstab can hang boot on a GPU host with 71 days uptime. **Needs remounting after a +reboot.** + +## The performance reality, measured on the same file through the same mount + +``` + ana-ml2 (cross-site) nh3-dev (same-site) +sequential read 24.7 MB/s 98.3 MB/s +small-file rate 45.3 files/s 34.6 files/s +``` + +Two different stories, and file layout decides which you get: + +- **Many small records -> ana-ml2 wins on BOTH axes.** That path is bound by per-file + round-trips and NAS overhead, not bandwidth, and ana-ml2 is an idle 96-core box + while nh3-dev is a loaded 16-vCPU VM. +- **Bulk sequential streaming -> the link eats the win.** 4x read penalty against a + 6x CPU gain. `datasets/raw` is 126 GB, `datasets/derived` is 1.7 GB — which one the + pipeline traverses changes the answer by two orders of magnitude. Staging a subset + to ana-ml2 local disk (195 GB free) beats pulling it over the wire repeatedly. + +The 24.7 MB/s is the Anaheim tunnel, not NFS and not the NAS — see +[[2026-08-23-anaheim-ipsec-tunnel-ceiling]]. No mount tuning will move it; +parallelism will. diff --git a/persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md b/persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md new file mode 100644 index 0000000..7c726e0 --- /dev/null +++ b/persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md @@ -0,0 +1,70 @@ +# [2026-08-23] Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed + +## b187 pre-stage (#405 phases 1+2) + +The matrix bridge stopped embedding the engine and became an HTTP client of the +Conversation API, so `WORLDTREE_API_URL` became **boot-blocking** — absent from the +container env, the bridge exits by design. Demo's compose never passed it; the next +recreate would have crash-looped. Pre-staged on demo and personal (additive, backed +up, verified with `docker compose config`, nothing restarted). + +**Key decision, and I got its scope wrong first.** I argued demo should stay keyless +(no homeserver -> no rooms -> no turns -> no 401s). Right about turns, **wrong about +scope**: the engine preflight authenticates at boot regardless of homeserver, so demo +booted permanently degraded. Corrected — key `341c1488` minted under worldtree-dev's +recorded authorization, vaulted, wired, three-hop hash-verified. + +## The 69-day-stale `:latest` landmine + +All three instances pinned `WORLDTREE_IMAGE=.../worldtree:latest` in `.env` while +running SHA-tagged images built that day. Local `:latest` = `b19afd71d7cc`, built +**2026-06-14**. So ANY `docker compose up` — anyone's, for any reason — silently +downgraded that service by 69 days. Same footgun as the 2026-06-15 outage. + +Re-pinned all three to their running SHAs (Worldtree #410), verified by rendering +compose config rather than reading `.env`, containers untouched. Playbook at +`playbooks/repin-worldtree-image.yaml`. + +**`worldtree-pinned` was the worst case:** the instance whose entire purpose is being +frozen was running a **dangling image with no repo tags**, kept alive only by the +running container. One `docker rm` from garbage collection. Tagged +`:446e5807bf43` first, then pinned. + +The guard I wrote had two bugs the pinned case exposed: it compared the container's +`.Config.Image` **string** (only the tag it was CREATED from — pinned was created +from `:latest` back when that meant 446e5807), and it reported CHANGED +unconditionally. Now compares **image IDs** and skips when already correct. + +## Matrix homeserver ownership + +Operator ruled: **personal owns the Matrix bridge.** The appservice tokens were never +missing — both sat at length 64 in the vaulted dev `env.sh` while both deployed +instances had them at length **zero**. Someone wired four of six Matrix vars and +stopped. Wired them into personal, three-hop verified. + +**The trap worth remembering:** Synapse's registration pointed at +`http://10.100.10.50:8009` — nh3-dev, a dead epoch, with transaction 2801 queued at +512s backoff. The natural fix (swap the IP) gives `10.250.50.152:8009` which is +**DEMO's** bridge, and Synapse can reach both — it would have connected, delivered, +and looked correct while routing the operator's live rooms to the demo instance. +**Personal's bridge is :8010.** `docker port` is ground truth. + +Corrected the URL, restarted Synapse (healthy in 32s after 3.5 months up), verified +`GET /_matrix/app/v1/ping -> 200` from inside the Synapse container. worldtree-dev's +smoke passed first try: room created, mimir accepted the invite, a real engine turn +ran, mimir replied in persona voice. #408 closed. + +## Open on worldtree-dev's side + +- **#411** — personal's bridge logs `Debug sink init failed: Permission denied: + /app/sessions/debug_rooms.json`. It creates two debug rooms but cannot persist + their IDs, so **every restart mints a fresh pair on the live homeserver**. Room + litter that compounds silently. Needs a which-container-writes-what check on the + sessions volume before anyone chowns it. +- Bridge/engine agent-roster drift: 6 of the bridge's 9 configured agents are not + listed by the engine on either instance. +- Historical Domari pairwise verdicts from the selene era are coin-flip-grade + (see [[2026-08-23-selene-retired-alias-collision]]); worldtree-dev banked that so + no future arc leans on them without re-judging. + +Commits `064181a`, `bb19a96`. diff --git a/persistent-memory.md b/persistent-memory.md index 568efb4..db7c382 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-08-22_ +_Last updated: 2026-08-23_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -108,24 +108,32 @@ no longer deployed sidecars here. See Recent decisions.) (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`. ## Current state / in-flight -_As of 2026-08-22 — three AI seats live on ana-ml2; `sec` is the one that moved this session._ +_As of 2026-08-23 — a long multi-party ops session; the one thing waiting on the operator is the Anaheim tunnel cipher._ -- **SEAT MAP.** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, 7 aliases, **still on the OLD nightly `311b3513` with MTP k=3**. **`char-rp`** = MeroMero-v2 dual-mode (prose + streaming CoT, one weight set, two aliases), GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC, GPU1 :8019 — **rebuilt this session, see below**. +- **⏳ THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit.** Circuit measured at **2,153 Mbit/s**; the tunnel ceilings ~550 Mbit/s aggregate, ~230 single-stream. FortiGate CPU **idle**, IPsec **NPU-offloaded**, interface error-free — so it is not crypto exhaustion. Both tunnels negotiate **`aes256-sha1`**; **AES-GCM is the proposed change** and the operator has signalled he will authorize it. Not executed: production edge, needs a matching change at NH3 + ESH, each tunnel drops during renegotiation. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` -- **🟢 `sec` NOW RUNS DFLASH2 ON A NEWER vLLM — promoted to its compose stack after real-use testing.** `nightly-e9d1398d` (+259 commits over production, `behind_by=0`), `dflash` k=7 with the 3.85 GB drafter, **util 0.52 / max-model-len 420,000 / KV ~453k**, 2048² vision. `restart: unless-stopped`, survives reboot. Canonical in `stacks/mog-sec/` with a fully-commented `.env.example`. **ROLLBACK:** `.env.bak-pre-dflash2-20260822` on the host, or swap `MOG_SPEC_CONFIG` + `MOG_IMAGE`. ⚠ `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` was **deliberately dropped** — the validated container never had it. +- **🟢 SEAT MAP (unchanged this session except selene).** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, **now 7 aliases** (see the collision note). **`char-rp`** = MeroMero-v2 dual-mode, GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC on DFlash2, GPU1 :8019. **`selene` RETIRED** — 17.2 GiB reclaimed on GPU1 (free now ~19.4 GiB). -- **⚠️ THE `sec` DEGENERATION QUESTION IS OPEN AND CONFOUNDED.** It no longer degenerates, but **engine and drafter changed together**. **The isolating experiment is MTP k=3 on `e9d1398d`** — not yet run. Operator ruling: the degeneration lives in the **un-fixed vLLM**, not the weights; my MTP-head hypothesis is **retracted**. ⚠⚠ **Both the "degenerates at 2k" and "coherent to 10k" sightings are n=1 from uncontrolled sessions and are NOT evidence.** Production is **172 commits behind GDN spec-decode fix #53077**; `#51113` is present in both builds and is **necessary but insufficient**. +- **⚠️ THE `sec` DEGENERATION QUESTION IS STILL OPEN AND CONFOUNDED.** Engine and drafter changed together; **the isolating experiment is MTP k=3 on `e9d1398d`** — still not run. Operator ruling stands: degeneration lives in the **un-fixed vLLM**, not the weights; the MTP-head hypothesis is **retracted**. Both prior sightings are n=1 and are NOT evidence. `gen` remains on the old nightly, untouched, gated on that experiment. -- **⏳ `gen` IS UNTOUCHED and still on the old build.** If DFlash2 + the newer engine are the answer, gen is the obvious next beneficiary — but that decision is gated on the isolating experiment above, not on sec's n=1 result. +- **🟢 ana-ml2 now mounts `/mnt/smithy`** (nh3-nas) **ro + soft, NOT in fstab** — needs a manual remount after reboot. For brokkr's R47 CPU work. Reads 24.7 MB/s sequential vs 98.3 on nh3-dev (that gap is the tunnel above), but **45 files/s vs 34** — small-file work is genuinely faster there. → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md` -- **🟢 ESH IS DUAL-STACK; the v4 static is a Cityside ticket.** IPv6 live on `esh-userland` (SSID `PVC`) and `esh-server` from a delegated `2607:73c0:402:1d00::/56`; hosts egress over v6 as themselves, un-NATted. **v4 remains CGNAT (`100.104.3.250`) and a full gateway reboot proved the purchased static is NOT provisioned** — carrier ticket, nothing left to try locally. v6 firewall audited: default-deny inbound both versions, correct. NH3 stays v6-off deliberately (single /64 reserved for meshing). Flat-zone lateral-movement finding **parked, id 44**. +- **🟢 ESH IS DUAL-STACK; the v4 static is a Cityside ticket.** IPv6 live on `esh-userland` and `esh-server` from a delegated `/56`. v4 remains CGNAT and a full gateway reboot proved the purchased static is **not provisioned** — carrier ticket, nothing left locally. NH3 stays v6-off deliberately. Flat-zone lateral-movement finding **parked, id 44**. -- **🟢 OTHER SERVICES.** speaches ASR live irv-ml1:8204 (Eyra; loop closed). Open WebUI on esh-docker-vm:3211 (admin creds + admin-scoped API key vaulted; **Lobe retirement still the operator's call**). Waterland, Homepage/Skyfall, fleet `.internal` DNS all landed earlier and are stable. +- **🟢 OTHER SERVICES.** `hrafn` browser-fetch adopted on ana-docker (infra-ops owns uptime; CI now genuinely deploys). speaches ASR live irv-ml1:8204. Open WebUI esh-docker-vm:3211 — **Lobe retirement still the operator's call**. Booth gained kept-board deletion + per-row link pruning. `pfi` gitea org created; claude-bot is an Owner and can create repos self-serve. -- **⏳ OPEN:** the MTP-k3-on-new-build isolating experiment; file the drafted upstream vLLM issue (operator's GitHub identity); Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift (`:main` vs pinned v0.11.0); `/tank` DEGRADED **70+ days**; Brokkr duplicate `reranker-a3-bge-v2-m3` alias; **MANY commits unpushed** — push is the operator's call. +- **⏳ OPEN ELSEWHERE:** MTP-k3 isolating experiment; upstream vLLM issue to file (operator's GitHub identity); Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift; `/tank` DEGRADED **70+ days**; Worldtree **#411** debug-room litter; bridge/engine agent-roster drift on both WT instances; brokkr's `gen` vs trained-reward-model bake-off (theirs to initiate). **Working tree is clean and pushed** through `0ad332b`. ## Recent decisions +- `[2026-08-23]` **Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit — not WireGuard, not CPU, not the fibre.** Cipher change proposed and operator-signalled; **execution pending, untracked by operator choice.** → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` +- `[2026-08-23]` **selene retired after losing a head-to-head on its own job; `chat-judge` moved to gen, the model name 404s by design.** Also surfaced that **7 aliases share one seat** — cross-checking between them is an echo, which caught a real defect in brokkr's 46k-exposure R47 gate. → `persistent-memory.d/2026-08-23-selene-retired-alias-collision.md` +- `[2026-08-23]` **hrafn adopted; its CI reported green for its whole life while deploying nothing.** A staging dir inside the rsync target destroyed its own source mid-copy; the deeper fault was verify steps that asserted uptime, never content. → `persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md` +- `[2026-08-23]` **Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed to personal.** Includes the `:8009`-is-demo port trap that an IP-only fix would have walked into. → `persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md` +- `[2026-08-23]` **Every secret-bearing `.env` on ana-docker tightened to 0600** — eight stacks including vaultwarden and traefik, verified exposed by reading one as `nobody`. → `persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md` +- `[2026-08-23]` **`pfi` gitea org created; claude-bot is an Owner and creates repos self-serve.** Closes the repo-creation half of the credential-migration directive — `vh` is a USER namespace so no service account could ever create there. Repo creation needs `write:user` + `write:repository` + `write:organization`; `POST /users/{u}/tokens` is basic-auth only, so minting needs the account password. Default new repos to `pfi/`. (`vh/eitri-smithy` was its first tenant, then moved.) +- `[2026-08-23]` **Booth: kept boards are deletable and link rows are prunable.** `release` on a kept card drops the sentinel so the existing × applies; `booth links` / `booth unlink ` prune one row. Rows are addressed by **content id, never position** — the board is append-only and multi-writer. **Releasing a board RESETS its TTL clock** (unlink bumps the dir mtime), so unkeep-and-wait is a 24h delay, not a delete. (`4be880f`, `0ad332b`) + - `[2026-08-22]` **DFlash2 spec-decode measured on our own stack; `sec` promoted to it.** +18–21% accepted length and +15–18% throughput over MTP k=3, drafter proved model-agnostic across two finetunes to 0.06%, and the k=7 MTP *control* showed deeper MTP is a throughput trap. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` - `[2026-08-22]` **Quant pipeline shipped a crippled tokenizer for months — fixed at source.** `quant_mixed_nvfp4.py` baked its calibration truncation (`max_length 2048`) into every mixed-NVFP4 build; latent on old transformers, fatal on new. Both live quants corrected, pipeline now saves a source-pristine tokenizer and asserts it. Playbook §3.14. (`0755ba7`) - `[2026-08-22]` **`sec` retuned to util 0.52 / 420K after a runtime OOM at 0.55/480K** — `gpu-memory-utilization` is not a hard reservation; activation grows past the dummy-data profile and six vLLM containers share GPU1. Also measured: the KV pool varies ~6.6% between boots, so max-model-len must be sized against the *lower* observation. (`6e82899`) @@ -256,6 +264,9 @@ _As of 2026-08-22 — three AI seats live on ana-ml2; `sec` is the one that move _209 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.** +- `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx. + - `[2026-08-15]` **Grafted bf16 MTP loads UNINITIALIZED (0% accept) unless `re:^mtp.*` is in the quant-config `ignore`; and W4A16=Marlin (not native FP4) costs ~20% even on decode.** Cost a premature 79 GB delete of a good model (declared desync-dead off the 0%). Lessons: test MTP on bf16 FIRST, isolate before deleting; modelopt 0.43 is dependency-hell for qwen3_5 (list-vs-dict quant_cfg + transformers conflict) — use llm-compressor. Full → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md` - `[2026-08-03]` **ComfyUI `--enable-triton-backend` on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3.** adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to `COMFY_CMDLINE_EXTRA`, recreated) → `triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5")` in `comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8`, failing at **node 5 CLIPTextEncode**. Triton's fp8 dequant kernel targets `fp8e4nv` (Hopper/Ada e4m3); **sm_86 Ampere (A6000) lacks hardware e4m3** → the JIT compile dies. With triton on it grabs the **global** `--fp8_e4m3fn-text-enc` dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged `sha256:94afb8ca`, sage intact, prod restored). **The parked cu130 rebuild won't fix it** (e4m3 = hardware format, not CUDA version). **DEFERRED to the Ada refresh** (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). **Mechanics:** `--enable-triton-backend` is a compose `environment:` var, so toggling it needs `docker compose up -d` (**recreate**), NOT `docker restart` (reuses the baked env, no-ops silently). Full: auto-memory `parked_triton_backend_ampere_fp8`.