memory: snapshot — run 7 training on gx10; run 6 TRANSFERRED after CSAM adjudication; erp-tune-v6-nvfp4a16 live as trial; ESH/YTVC/webhook repairs; ana-ml2 routes persisted; tank/zroot actions deferred to next session. Index 830→271 lines: 27 decisions + 8 abandoned archived, superseded in-flight blocks archived verbatim

2026-09-09 00:26:44 -07:00
parent 3e18a044bd
commit 5ad948bf31
23 changed files with 2951 additions and 2800 deletions
@@ -1,192 +0,0 @@
 # DFlash2 speculative decoding — measured on our own stack (2026-08-22)
 Operator-driven session. **Read the epistemic labels.** During the chase we generalised from
 observations that later proved wrong; this file separates what was *measured* from what remains
 *hypothesis*, and records the wrong turns so nobody re-derives them.
 ## What DFlash2 is
 A **2B draft model** (3.85 GB bf16) for speculative decoding against Qwen3.8-27B —
 `incoai/Qwen3.8-27B-DFlash2`, Apache-2.0, blog `inco.ai/blog/dflash2`, upstream `z-lab/dflash`.
 Block diffusion: drafts a whole 8-token block in one pass, with a candidate selector tracing a
 path through per-slot top-K. Lossless (greedy matches the target).
 vLLM support merged **2026-08-21 05:27 UTC** as PR **#52816** (`b389ac29`). Method string is
 **`"dflash"`**, not `dflash2`.
 ## ✅ MEASURED — throughput and acceptance
 Single instrument (`specbench.py`, 8 fixed prompts, temp 0, max_tokens 256), delta against
 vLLM's own `spec_decode` counters. The MTP k=3 numbers reproduce our recorded 58.4% / 55.3%
 figures exactly, which is what validates the instrument.
 | seat | config | accepted tok/forward | throughput |
 |---|---|---|---|
 | gen (orcarouter) | MTP k=3 *(production)* | 2.753 | 114.9 tok/s |
 | gen | MTP k=7 *(control)* | 3.041 | **74.0 tok/s** |
 | gen | **DFlash2 k=7** | **3.254** | **131.9 tok/s** |
 | sec (M.O.G.-SEC) | MTP k=3 *(production)* | 2.676 | 110.5 tok/s |
 | sec | **DFlash2 k=7** | **3.252** | **130.0 tok/s** |
 **⭐ The k=7 MTP control was essential and inverted the obvious read.** Going deeper on MTP
 *improves acceptance* (2.753 → 3.041) while **destroying throughput** (114.9 → 74.0). Our MTP
 head is a single module (`mtp_num_hidden_layers=1`, only `mtp.layers.0`, 15 tensors) run
 autoregressively, so k draft tokens cost k sequential forward passes. **"Just raise
 num_speculative_tokens" is a trap** — without the control I would have recommended it.
 DFlash2's win is therefore **not better per-token acceptance** — our MTP is actually *better* at
 position 0 (79.6% vs 75.4%). It is that block drafting makes depth nearly free.
 **⭐ The drafter is model-agnostic across finetunes — 3.254 (gen) vs 3.252 (sec), a 0.06%
 difference**, with superimposable per-position curves. One drafter file on `/tank` serves both.
 ## ✅ MEASURED — how DFlash2 runs (answers "can one drafter serve both seats?")
 **EAGLE3-style coupled, not standalone.** In vLLM: `load_model(self, target_model)` binds it to a
 specific target object; `pass_hidden_states_to_model=True`; `gpu_model_runner` reads
 `dflash_config.target_layer_ids` → `[i+1 …]` to register auxiliary hidden-state capture on the
 target at layers **5, 19, 33, 47, 61**. It even reads the target's RoPE style at load.
 Consequences:
 - **Weights file is shareable** (one download, both seats mount it) — gen and sec are
  architecturally identical on every dimension the drafter needs: 64 layers (deepest tap 61),
  hidden 5120, intermediate 17408, vocab 248,320 > mask token 248,070.
 - **VRAM is NOT shareable — 3.85 GB per seat.** The drafter lives inside the target's engine
  process, consuming hidden states mid-forward. Two seats are two processes; there is no
  cross-process sharing mechanism and there could not be.
 ## ✅ MEASURED — it works on our stack, which the card does not claim
 The card tests stock BF16 on an H200 with FlashAttention 3. Verified here instead:
 **abliterated + NVFP4 `compressed-tensors` target ✓, Blackwell sm_120 ✓, DFlash2 CUDA graphs
 captured ✓.** None of that was documented anywhere.
 ## 🔶 HYPOTHESIS — why our acceptance trails the published numbers
 Both our targets land at ~3.25 accepted length against the card's 4.10–5.46 on stock BF16.
 **Finetune drift is ruled out** — two *different* finetunes gave identical results to three
 decimals. The shared variable is **NVFP4 quantization of the target**, which is mechanically
 plausible (the drafter reads quantized hidden states at its five taps). Second candidate:
 prompt distribution (ours general-purpose, theirs GSM8K/MATH/HumanEval/MBPP/MT-Bench).
 **Neither is confirmed.** Settling it needs a BF16 target seat (~56 GB) — a real GPU window.
 ## ❌ RETRACTED — the "MTP head mismatch causes the degeneration" hypothesis
 **Operator ruling, 2026-08-22: this hypothesis is WRONG. The degeneration lives in the un-fixed
 vLLM, not in the weights.** Recorded here rather than deleted, because it was reasoned to
 confidently enough that a future session could re-derive it.
 **Two independent failures produced it, and the second is the instructive one:**
 1. **I treated a false dichotomy as a deduction.** Having verified gen and sec run an identical
   engine (same image ID `sha256:bd3236cff208…`, same live version
   `0.27.2rc1.dev150+g311b3513a` read from inside both processes, same flags bar
   `gpu-memory-utilization` 0.43 vs 0.44), I concluded "config is eliminated, therefore it is the
   weights." That does not follow. **An engine bug present in BOTH seats is not exonerated by the
   two seats being identical** — it just means the engine cannot explain a *difference*. It can
   still explain the *failure*.
 2. **The difference I was explaining may not exist.** The premise was a single operator
   observation of sec degenerating at ~2k, made during a session with many concurrent changes.
   **n=1 under heavy concurrent modification is not evidence** — see the meta-lesson below.
 **What survives as fact** (measured, still true, just not causal): sec's MTP head *is*
 byte-identical to `qwen38-27b-uncensored-bf16` across all 15 tensors — a stock head on a
 security-finetuned body, because the `Qwen3_5ForConditionalGeneration` wrapper never loads the
 head, so the finetuning could not reach it. gen's orcarouter head *was* abliterated in-band by
 its author. Acceptance differs slightly (gen 58.4%, sec 55.9%). **All true. None of it shown to
 cause multi-turn degeneration.**
 **Current standing explanation: the degeneration is an engine bug in the un-fixed vLLM.** Both
 production seats run `311b3513`, which is **172 commits behind GDN spec-decode fix #53077**
 (merged 2026-08-20). `#51113` is present in that build and is therefore **necessary but
 insufficient** on its own.
 ## ⭐⭐ META-LESSON — n=1 during a busy session is not evidence
 The operator's own framing, and it generalises past this incident: **an observation made while
 many things are being changed at once cannot carry a causal claim, no matter how confidently it
 is reported.** Tonight that single observation became the load-bearing premise for a weights-side
 hypothesis, a root-cause narrative, and very nearly a recommendation.
 This is the same failure the gen-seat compose file already warns about in different words — *"a
 passing probe is NOT sufficient evidence"* — inverted. That note guards against trusting a
 **negative** result from a synthetic test. This one guards against trusting a **positive**
 sighting from an uncontrolled session. Both reduce to: **hold the system still, or do not draw
 causal conclusions from it.**
 Applies equally to the "coherent to 10k" observation below — same n, same conditions, opposite
 direction. Neither observation is worth more than the other.
 ## ⚠️ CONFOUNDED — and the "before" state is itself unreliable
 sec now runs DFlash2 on a newer build and the operator reports **coherent to 10k tokens with
 adversarial nonsense prompts**. ⚠ Treat this the same way as the 2k sighting it is being compared
 against: **n=1, uncontrolled session, not evidence.** The comparison is weak on *both* ends.
 **Two variables changed at once:**
 1. **Engine**: `311b3513` → `e9d1398d`, **+259 commits, `behind_by=0`** (a strict superset),
   including GDN spec-decode fix **#53077** (merged 2026-08-20) that production is **172 commits
   behind**.
 2. **Drafter**: frozen MTP head → DFlash2 reading live hidden states.
 **Isolating it = run MTP k=3 on the same new build.** Not yet done.
 **#51113 is present in BOTH builds** (verified by ancestry, `behind_by=0` each) — so the
 "proper upstream fix" our compose comment credits is **necessary but insufficient**; sec ran it
 and still degenerated. Related open upstream: **#53180** (quantized Qwen3.8-27B hybrid GDN + MTP
 producing *silent* degenerate output, no fix), **#41884** (DFlash + prefix caching on hybrid,
 IndexError, workaround is disabling one).
 ## ❌ WRONG TURNS — do not repeat
 - **Version strings are not lineage.** The DFlash2 build reports `0.26.1rc1.dev1048` and our
  production nightly `0.27.2rc1.dev150`, which *looks* like a regression. It is a setuptools_scm
  tag-reachability artifact. **Use the GitHub compare API and check `behind_by`.**
 - **Docker Hub push timestamps lie about source freshness.** `nightly-ba07e4a4` was *pushed*
  06:12 UTC, comfortably after the 05:27 merge — but *cut* from a 03:46 commit that predates it.
  **Grep the image for the symbols you need.** Believing the timestamp would have cost an RP-seat
  outage to serve a model the engine could not instantiate.
 - **`--max-num-batched-tokens` was not the image truncation.** Raising it 16,384 → 32,768 on that
  theory changed nothing and cost ~3 GiB of peak activation, which came straight out of the KV
  pool. The cap was the tokenizer (§3.14 of the playbook).
 - **"1M needs YaRN, absent from config" is FALSE for the sec quant.** It is fully present:
  `rope_type: yarn`, `factor: 4.0`, `original_max_position_embeddings: 262144`,
  `max_position_embeddings: 1000000`. Context is a KV-memory choice, not a model limit.
 ## Live state — PROMOTED to the compose stack 2026-08-22
 **Operator-approved after real-use testing** ("performing very well"). The experimental
 standalone container is gone; `stacks/mog-sec/` is canonical and `restart: unless-stopped` means
 it survives reboots. Cutover verified: **KV pool 526,617 / 1.10x — identical to the container it
 replaced**, restarts 0, both gateway aliases serving, DFlash2 confirmed drafting at k=7
 (231 draft tokens over 33 drafts), vision working.
 ⚠ **One variable was deliberately REMOVED, not carried over.** The old stack hardcoded
 `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`; the validated DFlash2 container never set it,
 and playbook §3.10 records expandable_segments corrupting retained tensors elsewhere. The compose
 now defaults it EMPTY (`MOG_ALLOC_CONF`). Promoting it as-was would have shipped a variable the
 tested configuration did not have.
 **Compose is now parameterised for the shapes that differ:** `MOG_SPEC_CONFIG` carries the whole
 speculative JSON (dflash needs `"model": "/drafter"`, MTP must not have one — a method+tokens
 template cannot express both), plus `MOG_MM_PROCESSOR_KWARGS`, `MOG_DRAFT_MODEL`,
 `MOG_MAX_NUM_BATCHED_TOKENS`, `MOG_ALLOC_CONF`.
 **ROLLBACK:** `.env.bak-pre-dflash2-20260822` and `compose.yaml.bak-pre-dflash2-20260822` on the
 host; or one line — `MOG_SPEC_CONFIG={"method": "qwen3_5_mtp", "num_speculative_tokens": 3}` plus
 the old `MOG_IMAGE`.
 | | production sec | current |
 |---|---|---|
 | image | `nightly-311b3513` | `nightly-e9d1398d` |
 | speculation | MTP k=3 | **DFlash2 k=7**, drafter `/tank/aimodels/qwen38-27b-dflash2-drafter` |
 | max-model-len | 262,144 | **480,000** |
 | KV pool | 418,218 (1.60×) | **526,617 (1.10×)** |
 | images | 4096² → 16,384 tok | **2048² → ~5,125 tok** (`--mm-processor-kwargs` size cap) |
 ⚠ **`--gpu-memory-utilization 0.55` is the stable ceiling** while GPU1's other tenants are up.
 0.58 sized KV at 594,172 then **OOM'd during CUDA graph capture** — the process reached 57.49 GiB
 against ~57.6 free. Real 1M context needs ~49 GiB of KV and therefore evicting most of GPU1.
 Canonical config: `stacks/mog-sec/{compose.yaml,.env.example}` in this repo.
@@ -1,34 +0,0 @@
 # [2026-08-23] Every secret-bearing `.env` on ana-docker tightened to 0600
 Found while taking uptime ownership of hrafn: its `.env` was mode 0644 with a live
 bearer token. Not a hrafn lapse — **0644 was the de facto pattern on the host**.
 Eight stacks carried secret-shaped vars in world-readable `.env` files on a box with
 four interactive accounts, verified as real exposure by reading one as `nobody`.
 Swept: **vaultwarden, traefik**, beszel, gitea-runner, miniflux, news-digest,
 searxng, vor. (hrafn and nevermore were fixed separately the same day.) Six other
 stacks already used 0600, so this converged on the existing house pattern rather
 than inventing one. Post-sweep the host has **zero** secret-bearing `.env` readable
 by `nobody`.
 Playbook: `playbooks/tighten-env-perms.yaml`, one run per stack, re-runnable.
 ## The check that matters
 Every run asserts `docker compose config` still renders **as the deploy user**
 (`lkraven`), not as root. Checking the mode proves the bits changed; only rendering
 as the deploy user proves the next deploy can still resolve its variables.
 ## Two gotchas recorded in the playbook
 - **vaultwarden looked like it bind-mounted its `.env`** — which would mean the
  *container's* UID reads it and 0600 could break the password vault. It does not:
  that `- .env` is under `env_file:`, not `volumes:`. My grep matched the YAML list
  item without checking its parent key. The playbook now **refuses** any stack that
  genuinely bind-mounts its `.env`, since that case is read by the container UID.
 - **elway prompted for a sudo password.** The `ana-docker` ssh alias resolves to
  `lkraven`, who needs one; **`infra-ops@10.250.50.70` has NOPASSWD**. `corviduo-dev`
  was repointed to infra-ops at some point and `ana-docker` was not. Run elway against
  the infra-ops target on this host.
 Commit `a896c0a`.
@@ -1,663 +0,0 @@
 # [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit
 > **⛔ SUPERSEDED 2026-08-23 (same day, later session) — read the CORRECTION at
 > the bottom before acting on anything here.** The headline is wrong (the
 > relevant ceiling is NH3's **1 Gbps** uplink, not Anaheim's 2 Gbps), the
 > aggregate number is wrong (**692 Mbit/s** at 8 streams, not ~550 — the
 > original stopped measuring at 4), and the proposed remedy is **impossible**:
 > UniFi's manual site-to-site IPsec does not implement AES-GCM at all. The
 > per-stream observation and the parallelise-your-transfers mitigation are the
 > parts that survive.
 The operator noticed site-to-site transfers were slow for a datacenter fiber
 handoff and asked whether WireGuard was the limit. It is not WireGuard, and the
 circuit is fine.
 ## Measured
 ```
 ana-docker -> internet, 8 parallel     2,153 Mbit/s   <- the 2 Gbps handoff, delivering
 ANA <-> NH3 through the tunnel, 4 par.   460 Mbit/s
 FortiGate's own recorded peak            554 Mbit/s
 ANA <-> NH3, single stream               227 Mbit/s
 ANA <-> ESH, single stream               249-265 Mbit/s
 ESH <-> NH3 (never touches ana-gw)       545-557 Mbit/s  on a SINGLE stream
 ```
 Method: stdlib TCP probe (no ssh, no crypto, no compression) between site
 endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner
 Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling.
 ## What it is not
 - **Not WireGuard.** Both Anaheim tunnels are IPsec on ana-gw
  (`pfi-ana-nh3` -> 70.230.226.88, `ana-eshudm-dyn` -> the ESH UDM). WireGuard
  on ana-wg is remote-access only and is not in this path. Traceroute confirms:
  both slow paths have hop 1 = `10.250.50.1` (the FortiGate); the fast
  ESH<->NH3 path rides a `192.168.x` Site Magic overlay and never touches it.
 - **Not CPU or crypto exhaustion.** FortiGate CPU was **100% idle across all
  8 cores** during the tests, and both live tunnels report `npu_flag=03` with
  `dec_npuid=1 enc_npuid=1` — encrypt *and* decrypt are hardware-offloaded.
 - **Not a 250 Mbit/s cap.** That was the first number and it is misleading —
  single-stream TCP. Four parallel streams doubled it. Quote the aggregate.
 - **Not the interface.** wan1: `rxe=0 txe=0 rxd=0 txd=0`, no collisions.
 ## Most likely cause
 Both tunnels negotiate **`aes256-sha1`** in phase 1 *and* phase 2 (dhgrp 14,
 IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly
 faster on **AES-GCM**, which combines encryption and authentication in one
 pass. The datasheet IPsec headline for an 80F assumes GCM with large packets,
 not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x
 shortfall is consistent with that.
 ## Not executed
 Changing the proposal is a **production-edge change requiring a matching
 change at the far end** (NH3 UDM and the ESH UDM), and each tunnel drops while
 it renegotiates. Left for the operator. See the index entry for authorization
 state.
 ## Immediate mitigation, no config change
 Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a
 single stream through IPsec gets 227. **Anything moving bulk data across the
 Anaheim link should parallelise** — that alone roughly doubles throughput
 today.
 ## Practical consequence already observed
 `/mnt/smithy` mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s
 from nh3-dev (same file, same mount) — that gap *is* this tunnel, not NFS and
 not the NAS. See [[2026-08-23-smithy-mount-ana-ml2]].
 ## Access note
 ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. `sshpass` is absent
 on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is
 vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list`
 prints live ESP session keys** — never paste its output into althing, a
 booth, or a commit.
 ---
 ## CORRECTION (2026-08-23, later session): the cutover was attempted and the remedy does not exist
 The operator authorised the AES-GCM cutover, NH3 side first. It cannot be done,
 and the measurements taken while trying show there is very little left to win.
 ### AES-GCM is unavailable on the far end — not a naming problem
 The NH3 edge is a **UDM Pro SE** terminating `pfi-nh3-ana` (networkconf
 `_id 697d64414c85dd2b6669b00a`, `ifname vti64`). Its UniFi API **validates** the
 crypto enum and rejected every GCM spelling tried — `aes256gcm`, `aes256gcm128`,
 `aes256gcm16`, `aes-256-gcm`, `aes256-gcm`, `aes256gcm12`, `gcm`, `aes128gcm128`
 — all `HTTP 400 api.err.InvalidPayload`, nothing applied.
 **The control that makes this conclusive:** the *identical* request body with
 `ipsec_esp_encryption: "aes256"` returns `HTTP 200 rc:ok`. So the 400s are the
 enum rejecting the value, not a malformed body. Corroborating: **zero
 case-insensitive `gcm` matches across 7.3 MB of UniFi OS UI bundles.**
 Accepted enum (probed): `aes128`, `aes192`, `aes256`, `3des` → 200; `des`,
 `chacha20poly1305` → 400. There is no AEAD option. Both Anaheim tunnels land on
 UniFi far ends, so this blocks the ESH tunnel too.
 The FortiGate side **was** widened and is GCM-capable: phase2 `pfi-ana-nh3` now
 reads `set proposal aes256-sha1 aes256gcm`. Left in place deliberately — it is
 functionally identical while the peer only offers CBC, and reverting it would
 cost another SA renegotiation for a cosmetic gain. Phase 1 was never touched;
 IKE protects the control channel only and has no bearing on data throughput.
 ### The numbers that retire this as a problem
 Measured NH3→ANA through the tunnel, and NH3→ESH over Site Magic (WireGuard) on
 the same UDM and the same uplink, with the same stdlib TCP probe:
 | streams | IPsec NH3→ANA | WireGuard NH3→ESH |
 |---|---|---|
 | 1 | 245 Mbit/s | 557 Mbit/s |
 | 4 | 471 Mbit/s | 767 Mbit/s |
 | 8 | **692 Mbit/s** | **795 Mbit/s** |
 **NH3's WAN is a 1 Gbps link** (`uplink.speed = 1000`, port capable of 10G) —
 that, not Anaheim's 2 Gbps, is the ceiling for anything crossing this tunnel.
 So the tunnel does **~69% of the achievable uplink** at 8 streams, and the
 IPsec-vs-WireGuard gap collapses from 2.3× at one stream to **15% at eight**.
 Re-architecting the transport (site-to-site WireGuard via `ana-wg`, since
 FortiOS has no WireGuard) would chase that last 15%. Not worth it.
 ### What the constraint actually is
 A **per-stream** limit (~245 Mbit/s), not an aggregate crypto ceiling. Both
 endpoints are idle at load — FortiGate CPU 100% idle with `npu_flag=03`
 (offloaded both directions), UDM CPU ~7% with load1 moving 0.70 → 1.55. The
 shape is per-SA/per-flow serialisation, and WireGuard shows the same shape from
 a higher floor (557 → 795 is only 1.43× scaling).
 ### Actionable consequence
 Anything moving bulk data across this link should **parallelise** — 245 → 692
 Mbit/s, a 2.8× win with no config change. For single-stream workloads that
 cannot be parallelised at the application layer, **NFS `nconnect=N` is the
 lever**: it opens N TCP connections per mount, converting a single-stream
 workload into a parallel one. The `/mnt/smithy` mount on ana-ml2 reading at
 24.7 MB/s (~200 Mbit/s, i.e. exactly the single-stream ceiling) is the live
 example — remounting with `nconnect=8` is the obvious test.
 ### Foot-gun recorded
 Probing the enum by PUTting candidate values **applies the accepted ones**. A
 probe loop here timed out with `3des` briefly live on the NH3 side, which the
 FortiGate would not accept — a short tunnel outage until `aes256` was restored
 (~1 minute, confirmed by the SA counters resetting). If you enumerate a UniFi
 config enum this way, restore the known-good value after **every** 200, not at
 the end of the loop. Post-change verification: the UDM object was diffed
 field-by-field against its pre-change snapshot and is **byte-identical**.
 ---
 ## FOLLOW-UP (2026-08-23): what the per-stream limit actually is
 The correction above called the constraint "per-SA/per-flow serialisation".
 That was a hand-wave. Measured properly, it is a **hard per-flow rate cap of
 ~230–245 Mbit/s with a very deep buffer in front of it** — not a tuning
 problem, not loss, not window size.
 ### The evidence: pin the send buffer and sweep it
 Single stream NH3 → ana-docker, `SO_SNDBUF` pinned, `ss -ti` sampled in flight:
 | in-flight cap | throughput | RTT in flight | minRTT | retrans |
 |---|---|---|---|---|
 | 256 KB | 224 Mbit/s | 7.8 ms | 5.3 ms | 0 |
 | 416 KB | 225 Mbit/s | 11.8 ms | 6.6 ms | 0 |
 | 416 KB | 245–247 Mbit/s | 12.0 ms | 5.6 ms | 0 |
 | ~3.3 MB (autotuned) | 245 Mbit/s | **107 ms** | 5.5 ms | 0 |
 **Throughput is flat across a 13× range of in-flight data while RTT scales with
 it.** That is the signature of a fixed service rate with a standing queue: the
 window controls only how much queue you build, never how fast you go. Had this
 been window-limited, throughput would have risen with the buffer. Had it been
 congestion, there would be retransmits — there are essentially none
 (`retrans:0`, 0% ping loss).
 So `net.ipv4.tcp_*` tuning, window scaling and congestion-control choice are all
 **red herrings here**. Do not go there.
 ### Bufferbloat: one bulk stream wrecks latency for everything else
 Measured on the same tunnel, ping to ana-docker:
 - idle: **6.9 ms** avg
 - during a **single** bulk TCP stream: **102 ms** avg, 136 ms max, 0% loss
 **15× latency inflation from one transfer.** This is the operationally
 important finding — any interactive traffic sharing the Anaheim link (ssh,
 RDP, althing, VoIP) degrades badly whenever anything moves bulk data, and it
 takes only one stream to do it. Parallelising transfers makes throughput
 better and this *worse*. If it starts biting, the fix is an AQM/shaper on the
 tunnel (or rate-limiting bulk jobs), not more buffer.
 ### Where the cap lives — strong inference, not proof
 Three paths, and the FortiGate is the only variable:
 | path | single-stream |
 |---|---|
 | FortiGate ↔ NH3 UDM (IPsec) | 245 Mbit/s |
 | FortiGate ↔ ESH UDM (IPsec) | 249–265 Mbit/s |
 | NH3 UDM ↔ ESH UDM (WireGuard, **no FortiGate**) | 557 Mbit/s |
 Present in both slow paths, absent from the fast one. Aggregate over the same
 SA reaches 692 Mbit/s, so it cannot be the SA or the crypto engine as a whole —
 many flows spread out fine, one flow does not.
 The mechanism that fits is **FortiGate NPU IPsec offload being per-session**:
 each firewall session is bound to one crypto engine, so a single TCP flow is
 capped at one engine's rate while many sessions spread across engines. **This
 is inference from the throughput shape, not something confirmed on the box** —
 `diagnose sys session list` was not captured for a TCP flow (the filter caught
 only traceroute UDP probes). A single-stream control through ana-gw *without*
 IPsec returned 290 Mbit/s to Hetzner Ashburn, but at ~60 ms RTT that is
 window-limited and does not discriminate. **If this matters, the clean test is
 a non-IPsec single stream between two Anaheim VLANs at low RTT.**
 **Relevant to the FortiGate cutover decision:** if the per-flow cap is the
 FortiGate's IPsec path, replacing the box plausibly lifts single-stream
 throughput toward the WireGuard figure. That is a point in favour of the
 cutover, and it is cheap to verify afterwards by re-running the sweep.
 ---
 ## FOLLOW-UP 2 (2026-08-23): it is NOT a capacity problem, and it IS specific to IPsec
 Operator asked directly whether the 80F "can't handle the traffic". It can.
 Two new measurements settle the shape of this, and correct an overstatement in
 FOLLOW-UP 1 (which pointed at the FortiGate on evidence that was confounded —
 every slow path was *both* IPsec *and* FortiGate, so protocol and box could not
 be separated by that argument).
 ### The 80F routes a single flow at line rate when IPsec is not involved
 `ana-ml2 → pfi-pve`, inter-VLAN **through** ana-gw (traceroute hop 1 =
 `10.250.50.1`), 0.36 ms RTT, no tunnel:
 | streams | throughput |
 |---|---|
 | 1 | **940.2 Mbit/s** |
 | 8 | 939.3 Mbit/s |
 Single stream saturates 1 GbE. So the box does **not** cap single sessions in
 general, and there is no per-session ceiling in its plain forwarding path. The
 ~250 Mbit/s per-flow cap is **specific to the IPsec datapath**.
 ### Both IPsec tunnels converge on the same numbers despite different far ends
 Measured today with the same probe:
 | tunnel | far-end gateway | RTT | 1 stream | 8 streams |
 |---|---|---|---|---|
 | NH3 ↔ ANA | UDM Pro **SE** | 6.7 ms | 245 Mbit/s | 692 Mbit/s |
 | ESH ↔ ANA | UDM Pro **Max** | 3.9 ms | **268 Mbit/s** | **715 Mbit/s** |
 Different gateway hardware, different sites, different uplinks, and RTT
 differing by 1.7× — yet single-stream differs by only 9%. **If this were
 window-limited the 3.9 ms path would be ~1.7× faster.** It is not, which is
 independent confirmation of a rate cap rather than a BDP effect.
 ### Capacity summary — the box has headroom it will not give one flow
 - plain routing, 1 stream: **940 Mbit/s** (line rate)
 - plain routing to internet, 8 streams: **2,153 Mbit/s**
 - IPsec, 8 streams: **692–715 Mbit/s**
 - IPsec, 1 stream: **245–268 Mbit/s**
 - CPU **100% idle** throughout; IPsec NPU-offloaded (`npu_flag=03`)
 Within a single SA, 8 sessions get ~2.9× what 1 session gets, so the datapath
 distributes work **by inner session** — consistent with IPsec offload binding a
 session to one crypto engine.
 ### What is still NOT separated
 Whether the cap belongs to **the 80F's IPsec offload** or to **UniFi's IPsec
 implementation**. Both tunnels have a UDM at the far end, and both UDMs run the
 same UniFi firmware, so identical caps are explainable either way. The Pro Max
 being only 9% faster than the Pro SE argues against the UniFi side (a beefier
 CPU should show more), but that is suggestive, not conclusive.
 **The test that closes it:** an IPsec tunnel whose endpoints do not include the
 80F — e.g. a temporary UDM↔UDM IPsec tunnel between NH3 and ESH, measured
 single-stream. If it also caps ~250, the FortiGate is exonerated and replacing
 it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM
 WireGuard achieves, the 80F is the limiter. **Bears directly on the pending
 FortiGate cutover** — worth running before that decision, not after.
 ---
 ## FOLLOW-UP 3 (2026-08-23): WireGuard over the same internet path does 767 Mbit/s on ONE stream
 Operator asked for a WireGuard test from `ana-wg` to NH3 over the public
 internet. It is the test that separates the *path* from the *crypto*, and the
 answer is unambiguous. **It also overturns FOLLOW-UP 1's "re-architecting the
 transport is not worth it" — that conclusion compared 8-stream numbers and was
 wrong for single-stream workloads.**
 ### Setup (fully torn down afterwards)
 `ana-wg` (10.250.50.252, Debian 12 LXC, 4 cores) already has an
 internet-reachable WireGuard endpoint: wg0 on **UDP 31337**, published by
 FortiGate VIP `wg-to-ana-wg` (extip **38.120.12.42** → 10.250.50.252:31337,
 policy 46, service `WireGuard-LEET`). **No FortiGate change was needed.** A
 temporary `wgt0` was created on nh3-dev (10.30.10.200/32) as a fourth peer on
 wg0, measured, then removed — ana-wg is back to its original 3 peers and the
 keys were shredded. `wireguard-tools` was installed on nh3-dev and **left in
 place** (benign, and wanted if this becomes permanent).
 In this topology **neither gateway does crypto**: the FortiGate and the NH3 UDM
 only NAT/forward UDP, and Linux does WireGuard at both ends.
 ### The full comparison
 | path | crypto performed by | 1 stream | 8 streams |
 |---|---|---|---|
 | IPsec NH3↔ANA | FortiGate + UDM | 245 Mbit/s | 692 Mbit/s |
 | IPsec ESH↔ANA | FortiGate + UDM | 268 Mbit/s | 715 Mbit/s |
 | **WireGuard NH3→ana-wg** (same internet path) | **Linux + Linux** | **767 Mbit/s** | 763 Mbit/s |
 | WireGuard NH3↔ESH (Site Magic) | UDM + UDM | 557 Mbit/s | 795 Mbit/s |
 | plain routing through the 80F (inter-VLAN) | none | 940 Mbit/s | 939 Mbit/s |
 **One stream equals eight streams over Linux WireGuard (767 ≈ 763).** There is
 no per-flow penalty at all, and a single flow already saturates the path. So
 the ~245 Mbit/s per-flow cap is **not** the ISP, not the circuit, not the NH3
 uplink and not the physical path — all of which sustain 767 on one flow.
 Per-flow penalty ranks by implementation:
 - **Linux WireGuard — none** (767 → 763, flat)
 - **UDM WireGuard — mild**, ~1.4× (557 → 795)
 - **IPsec on this pair — severe**, ~2.8× (245 → 692)
 ### Latency under load — the same story
 | path | idle | during ONE bulk stream |
 |---|---|---|
 | IPsec NH3↔ANA | 6.9 ms | **102 ms** avg, 136 ms max |
 | WireGuard NH3→ana-wg | 6.2 ms | **12.7 ms** avg, 23 ms max |
 WireGuard carries **3.1× the single-stream throughput with 8× less latency
 inflation** on the same wire.
 ### Attribution — still not fully separated, and it no longer matters much
 Both IPsec measurements have a FortiGate *and* a UDM doing IPsec, so this still
 does not isolate which one imposes the 2.8× penalty. Closing that would need
 Linux↔Linux IPsec or UDM↔UDM IPsec on the same path. **But the practical
 decision no longer depends on the answer**, because the fix is the same either
 way and it is already demonstrated.
 ### Recommendation (supersedes FOLLOW-UP 1)
 A **WireGuard site-to-site between NH3 and Anaheim, terminated on `ana-wg`**, is
 worth real consideration: 3.1× single-stream, flat scaling, far better latency
 under load, and it reuses infrastructure that already exists and is already
 internet-reachable. It is also the architecture already proven for NH3↔ESH.
 Open questions before committing: routing/failover if ana-wg (an LXC) is down,
 whether it replaces or parallels the IPsec tunnel, and firewall policy for the
 new transit. ana-wg CPU was only ~40% busy across 4 cores at 767 Mbit/s, so it
 has headroom.
 **AND: `nconnect=8` on /mnt/smithy remains worth doing regardless** — it is the
 same lever (turn one flow into many) and brokkr-smithy-dev has given standing
 approval to apply it once the FortiGate work settles, with no need to ask again.
 ---
 ## RESOLVED (2026-08-23): it is the UDM's software AES-CBC. The FortiGate is exonerated.
 Operator's theory — the UDM does IPsec in software with no crypto offload, so
 the cost of the cipher itself is the limit — is **correct**, and it is now
 demonstrated rather than inferred. He also correctly pointed out that
 UDM↔UDM Site Magic is **WireGuard, not IPsec**, so that row never said anything
 about UniFi's IPsec performance. It didn't, and I had leaned on it.
 ### The controlled experiment: vary cipher cost, hold everything else
 AES-128 is 10 rounds, AES-256 is 14. If software crypto is the binding
 constraint, throughput must rise when the cipher gets cheaper. If the limit
 were the FortiGate's NPU, it would not move at all — hardware crypto is not
 cipher-cost-sensitive in that range. Run A/B/A, single stream, 25–60 s each:
 | condition | ESP cipher | single-stream | UDM CPU |
 |---|---|---|---|
 | A | aes256-cbc + sha1 | 232.3 Mbit/s | 35.4% |
 | B | **aes128**-cbc + sha1 | **281.8**, 274.9 Mbit/s | 35.5% |
 | A again | aes256-cbc + sha1 | 244.9, 242.5 Mbit/s | — |
 **~1.16–1.20× faster on the cheaper cipher at identical CPU.** Same bytes of
 CPU work, more payload through it. That is the signature of CPU-bound software
 crypto, and it rules out the FortiGate's NPU as the limiter.
 ### Correcting two of my own earlier claims
 1. **"UDM CPU is only ~7%, so it isn't CPU-bound" was WRONG — a sampling
   artifact.** UniFi's `system-stats.cpu` refreshes on the device report
   interval; 4-second sample windows were reading stale values. Under a
   sustained 60 s single-stream load it reads **35.4%**, with load1 rising
   0.60 → 1.17. On a 4-core UDM Pro SE that is ≈1.4 cores — one core saturated
   on crypto plus overhead. **Always drive load for ≥60 s before trusting a
   UniFi CPU figure.**
 2. **The "FortiGate per-session NPU offload" hypothesis is REFUTED**, not merely
   unproven. It predicts no change from a cipher swap; a 20% change was measured.
 ### Why the numbers all line up now
 - **1 stream = 1 core of UDM crypto** → ~240 Mbit/s on AES-256-CBC.
 - **8 streams = ~3 usable cores** → ~692 Mbit/s, ≈2.9× the single-stream figure
  on a 4-core box. Aggregate is noisy (492–692 across repeats on a live link)
  and is *not* cipher-sensitive, consistent with it being bounded by the path/
  uplink rather than crypto once several cores are engaged.
 - **AES-CBC is the specific villain: it is serial.** Each block depends on the
  previous one, so the ARM AES instructions cannot pipeline across blocks. GCM
  (CTR-based) and ChaCha20-Poly1305 both parallelise freely. That is why the
  same UDM does 557 Mbit/s single-stream on WireGuard and only 240 on IPsec.
 - **This retroactively vindicates the GCM cutover as the right idea aimed at the
  right box** — GCM would have removed the serial dependency on the constrained
  end. UniFi simply does not offer it, which is what made it impossible.
 ### Options this opens
 - **AES-128 instead of AES-256: ~16–20% for free**, no topology change, one API
  call per end. 128-bit is not the weak link here (SHA1 integrity is more
  dated, and unchanged either way). Operator's call — **not adopted**, restored
  to aes256.
 - **WireGuard site-to-site via ana-wg: 767 Mbit/s single-stream** (3.1×), and it
  sidesteps the UDM's IPsec datapath entirely. Still the biggest win available.
 - Replacing the FortiGate **will not help this** — it was never the constraint.
  Worth knowing before the cutover.
 ### State left behind
 UDM network object verified **byte-identical** to its pre-test snapshot
 (aes256/sha1). Tunnel up, selectors 1/1. FortiGate phase2 `pfi-ana-nh3` is
 left as `aes256-sha1 aes256gcm aes128-sha1` — a permissive superset; the peer
 offers only aes256 so the extra entries are inert, but **narrowing it back to
 `aes256-sha1` is one line** if the looser list is unwanted.
 ---
 ## FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through
 Operator's point: FortiOS has no WireGuard, so a WireGuard site-to-site must
 terminate on a box *behind* the edge. Correct — and `ana-wg` (LXC, CT 113 on
 pfi-pve, 10.250.50.252) already is that box.
 **This closes a gap in FOLLOW-UP 3.** That 767 Mbit/s figure was measured with
 traffic terminating *on* ana-wg. Real traffic must be forwarded onward to other
 Anaheim hosts, which was never measured. Now it is:
 | topology | 1 stream | 8 streams |
 |---|---|---|
 | IPsec, FortiGate ↔ UDM (today) | 245 Mbit/s | 692 Mbit/s |
 | WG terminating **on** ana-wg | 767 Mbit/s | 763 Mbit/s |
 | **WG transit: nh3 → wg → ana-wg → forward → ana-docker** | **763.8 Mbit/s** | **790.4 Mbit/s** |
 **Forwarding through the LXC is free** (763.8 vs 767). The downstream-VM
 architecture delivers the full 3.1× single-stream for real transit traffic, not
 just for traffic landing on the tunnel box.
 ana-wg while forwarding 764 Mbit/s: **~22% busy across 4 cores** (77.8% idle),
 so roughly 0.9 cores. Note `/proc/loadavg` inside this LXC reports the *host's*
 load, not the container's — do not read it as ana-wg's own. For contrast the
 UDM burns 35.4% of its 4 cores to move 240 Mbit/s, so ana-wg has ample headroom.
 ### Design consequences of terminating downstream — the parts that need decisions
 1. **Anaheim hosts must route to ana-wg, not to the FortiGate.** The 763.8
   figure was obtained with an explicit `10.30.10.200/32 via 10.250.50.252`
   route on ana-docker. Without that, a host sends 10.100.0.0/16 to its default
   gateway (ana-gw), which routes it back out the *same* interface to ana-wg — a
   LAN hairpin crossing the FortiGate twice. **The hairpin variant was NOT
   measured.** Options: DHCP option 121 pushing the route fleet-wide, a dedicated
   transit VLAN for ana-wg, or accept the hairpin.
 2. **New single point of failure.** Today site-to-site dies only when the edge
   dies, which is total anyway. A downstream terminator fails independently.
   Mitigation: keep the IPsec tunnel configured as a higher-metric fallback
   route so it takes over when ana-wg is down.
 3. **ana-wg is an LXC on pfi-pve**, so its ~0.9 cores and NIC traffic land on the
   hypervisor shared with the rest of the Anaheim VMs.
 4. **The NH3 end needs a terminator too**, and there are two shapes:
   - **Linux VM at NH3** (nh3-dev or a dedicated VM on nh3-pve) — this is what
     was measured: **764 Mbit/s**.
   - **NH3 UDM's existing WireGuard server** (`PFI-NH3-WG`, wireguard-server on
     UDP 31337) accepting ana-wg as a peer — plausible but **untested**, and
     UniFi's WireGuard shows a per-flow penalty (557 Mbit/s single-stream on
     Site Magic), so expect ~557 rather than 764. Still 2.3× today.
 ### Standing recommendation
 Worth doing, but it is **a project, not a config tweak** — routing, failover and
 policy all need deciding. The cheap wins remain available meanwhile and are
 independent: `nconnect=8` on NFS mounts (approved by brokkr-smithy-dev, pending
 the FortiGate work settling) and AES-128 for ~20%.
 ---
 ## LANDED (2026-08-23): AES-128 on both tunnels; FortiGate public admin closed
 Operator directed: adopt AES-128 on **both** Anaheim tunnels, make-before-break,
 then close the FortiGate's WAN and SSH admin surfaces. All done and verified.
 **Context that retires the WireGuard-in-a-VM design work:** the FortiGate is
 being **replaced by OPNsense on a Dell R420**, which gives **WireGuard on the
 edge device itself**. The downstream-terminator architecture (FOLLOW-UP 4) is
 therefore moot — do not scope it. This also **un-parks the OPNsense migration**,
 which auto-memory recorded as PARKED pending "hardware acquisition"; the R420
 is that trigger.
 ### What changed
 Make-before-break on the FortiGate first, so neither tunnel dropped waiting on
 a far end:
 | phase2 | proposal now |
 |---|---|
 | `pfi-ana-nh3` | `aes256-sha1 aes256gcm aes128-sha1` |
 | `ana-eshudm-dyn` | `aes256-sha1 aes128-sha1` |
 Then each UDM flipped to `ipsec_esp_encryption: aes128`:
 | tunnel | UDM object | before | after |
 |---|---|---|---|
 | NH3 ↔ ANA | `pfi-nh3-ana` `697d64414c85dd2b6669b00a` @ 10.100.0.1 | 245 Mbit/s | **269.7** |
 | ESH ↔ ANA | `esh-ana` `697723b9b9d4266dddf2bcc7` @ 10.0.0.1 | 268 Mbit/s | **304.3** |
 Single-stream gain ~10–13% here, against 16–20% in the earlier controlled A/B —
 the difference is live-link variance, not a different result. Both UDM objects
 were diffed field-by-field against pre-change snapshots: **the only field that
 moved on either is `ipsec_esp_encryption`.**
 The FortiGate proposal lists were deliberately **left permissive** (still
 accepting aes256). The peers offer only aes128 so the extra entries are inert,
 and keeping them means a UDM reverting does not strand the tunnel. Narrowing to
 `aes128-sha1` alone is a one-liner if the looser list is unwanted.
 ### Admin surfaces closed
 `wan1 allowaccess` → **`ping`** (https + ssh removed) and `infra-ops` trusthost
 → **10.0.0.0/8 only** (the 8 wide-open ranges unset). Verified 443 and 22 closed
 from both NH3 and ESH; management over the tunnel at 10.250.0.1 still works.
 **Sequencing that matters: the close was executed over the TUNNEL path, not over
 WAN** — removing `ssh` from allowaccess while connected over WAN kills the
 session mid-command.
 **Consequence to hold in mind: ana-gw now has no out-of-band management path.**
 If both tunnels drop it is console-only until someone is on site.
 ### Gotcha: the two UDM vault items have DIFFERENT shapes
 - `unifi/pfi-udmse-api-key` → a **bare 32-char key**. `secret get` output is the key.
 - `unifi/esh-udmpm-api-key` → a **19-line documentation note** with the key on a
  `key:` line. `secret get` piped straight into a header yields a 1396-byte
  value and the UDM answers **`400 Bad Request` from nginx**. Extract with
  `grep '^key:' | awk '{print $2}'`.
 **The ESH key's first-ever confirmed WRITE happened here** (auto-memory recorded
 it as read-verified only): a control PUT of the unchanged object returned
 `rc:ok`, then the real change did too. That key has a full read+write admin role.
 ---
 ## CORRECTION (2026-08-23): port 80 on the WAN IP is the FortiOS ACME listener
 The claim in the previous section that `.42:80` was an **ISP transparent proxy**
 was **WRONG**, and so was the earlier warning that ACME renewal would fail with
 port 80 absent from `allowaccess`. Operator pushed back asking where the port-80
 map terminated. It terminates **on the FortiGate itself**.
 **What it is:** the FortiOS **ACME HTTP-01 challenge listener**. `config system
 acme` has `set interface "wan1"`, and FortiOS opens port 80 on that interface to
 answer Let's Encrypt challenges **independently of `allowaccess`** — `wan1
 allowaccess` reads `ping` only and the port is still open. Every non-challenge
 request returns a fixed 403 whose body is literally:
 ```
 <!DOCTYPE html><html><head><title>ACME Access Only</title></head><body>ACME Access Only</body></html>
 ```
 **Not a DNAT.** The full VIP table has 14 entries; only two land on `.42` —
 `Kokoro-In` (:8880 → 10.250.50.51) and `wg-to-ana-wg` (:31337 → 10.250.50.252).
 ~~Worth noting separately: four VIPs are all-port static NAT~~ — **that claim was
 WRONG, see the correction below.** All fourteen VIPs are scoped.
 ### The methodology error that produced the wrong answer — worth not repeating
 The sniffer filter used was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
 matches only inbound packets**, so outbound SYN-ACKs were excluded *by
 construction*; concluding "the box sends no SYN-ACK" from that capture was
 unsound. Re-run with the bidirectional `host 38.120.12.42 and tcp port 80` it
 immediately shows `wan1 out 38.120.12.42.80 -> <scanner>: syn ack`.
 **Rule: when testing whether a box *answers*, the sniffer filter must be
 bidirectional. `dst host` silently answers a different question.**
 ### Consequences
 - **ACME renewal will work** with `allowaccess ping`. The earlier "add `http`
  back or the cert expires" warning is retracted — FortiOS opens the challenge
  port itself. Cert valid to 2026-10-27, renewal attempt ~2026-09-27.
 - **It is not an admin surface** — static 403, no auth, no GUI.
 - Its practical value is now low: WAN admin is closed, so the cert only serves
  the internal GUI at 10.250.0.1, where the name would not match anyway. Killing
  it (`config system acme` → unset interface) would close the last WAN listener
  at the cost of cert renewal. Operator's call; **not done**.
 ---
 ## CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE
 ### ACME disabled — the WAN IP now exposes nothing
 `config system acme / unset interface` (the account object is left in place;
 with no interface bound there is no listener). Verified:
 - **External scan of 38.120.12.42 across 55 ports: no open TCP ports at all.**
 - Internal GUI at 10.250.0.1 still answers **200**, SSH still works.
 - `admin-server-cert` is still `ana-fw.pfi` — the existing cert is untouched and
  serves the internal GUI until **2026-10-27**; it simply will not auto-renew.
 Reverse with `config system acme / set interface "wan1"`.
 ### RETRACTION: the four VIPs are NOT all-port
 A previous section claimed `Rustdesk`, `https-to-tacticalrmm`, `web-to-webhost`
 and `web-to-sfcontainer` were unrestricted all-port static NATs. **They are not.**
 A FortiOS VIP can be scoped **two different ways** and the parser used only
 checked one:
 1. `set portforward enable` + `set extport <n>` — a single mapped port, **or**
 2. `set service "<svc>"` on the VIP object — constrains the VIP to that service.
 All four use form 2. The custom services are narrow: `Rustdesk` = TCP
 21115–21119 + UDP 21116 (the standard RustDesk range), `ssh-mapped-2223` = TCP
 2223 only. **Every one of the 14 VIPs is scoped; none is unrestricted.**
 **Lesson: absence of `portforward` does NOT mean all-port on a FortiOS VIP —
 check `service` too.** Better still, do what settled it here: scan from outside
 rather than reading config.
 ### Ground-truth public exposure (external TCP scan, post-change)
 | IP | open | maps to |
 |---|---|---|
 | 38.120.12.41 | *nothing* | — |
 | **38.120.12.42** | ***nothing*** | the FortiGate itself — fully closed |
 | 38.120.12.43 | 80, 443 | sf-ana-container 10.250.150.100 (SureFire tenant) |
 | 38.120.12.44 | 22, 80, 443, 8025, 21115–21119 | gitea (→222), traefik, mailrise, RustDesk |
 | 38.120.12.45 | 80, 443, 2223 | pfi-ana-webhost 10.250.50.52 (2223→22) |
 | 38.120.12.46 | 443 | pfi-tacticalrmm 10.250.50.57 |
 Configured-but-closed: 8443 (mattermost-calls), 8444 (webdav-nas), 8880
 (Kokoro-In) — VIPs exist, nothing listening behind them. Worth a tidy-up during
 the OPNsense translation but not exposure.
@@ -1,63 +0,0 @@
 # [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing
 `hrafn` — genuine-Chromium browser-fetch behind a REST API, for bot-gated sites
 (Reddit first). Built by nevermore-claude on ana-docker, handed to infra-ops for
 uptime ownership. Internal-only on `traefik-net`, no host port; consumers reach
 `http://hrafn:8080`. Canonical at `stacks/hrafn/`.
 ## Intake found a live credential exposure
 `/opt/docker/compose/hrafn/.env` was mode **0644 with a live 57-char bearer token**
 — verified as real exposure by reading it as `nobody` on a box with four
 interactive accounts. Tightened to 0600. That triggered the wider sweep (see
 [[2026-08-23-ana-docker-env-perms-sweep]]).
 ## The CI defect — the one worth remembering
 I authored the deploy (elway playbook + gitea workflow) to replace a hand-rsync,
 tagging the image with the commit SHA for provenance. nevermore-claude later found
 v1.0.0 deploying "green" while the host still served 0.1.0.
 **Root cause was mine and nastier than either hypothesis.** The staging dir was
 `$compose_dir/.stage` — **inside** the rsync target. So
 `rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted `.stage` from the
 destination (absent from the source listing) **during** the transfer, destroying
 its own source mid-copy. Reproduced exactly:
 ```
 before:  app.py="OLD"   leftover.txt   .stage/app.py="NEW"
 after:   app.py="OLD"   leftover.txt GONE, .stage GONE
 ```
 Deletion succeeded, the copy silently did not, rsync exited 0. So the directory
 *looked* converged while host source stayed frozen at the first manual rsync —
 and because the build's `COPY` inputs never changed, Docker full-cache-hit and
 every SHA tag aliased one image. **The provenance the tagging existed to provide
 was false for the pipeline's entire life.**
 **The real failure is the verification.** The verify steps asserted the marker,
 container health, and a 200 from `/readyz` — all of which pass against a
 completely frozen host. None measured *content*. A deploy that reports success
 without asserting the bytes changed is verifying an **uptime**, not a deploy.
 ## Fixes
 - stage at `/tmp/hrafn-deploy-stage`, outside the target
 - CI computes `context_sha256` over the shipped file list; the playbook recomputes
  it **on the host after the converge** and fails on mismatch
 - compare the running container's `src/**/*.py` against the host's, so a SHA tag
  cannot name layers the image lacks
 - **compare `*.py` only** — `pip install .` generates `src/*.egg-info/*` inside the
  image and `__pycache__` appears at runtime, so a naive `find src -type f` compare
  false-fails on every healthy deploy. Verified against a known-good container
  before shipping (12 host files, 18 in container, 0 content differences).
 - declined `--no-cache`: a cache hit is *correct* when the context is genuinely
  unchanged; assert the property rather than brute-force it.
 ## Access
 Operator granted claude-bot **write** on `vh/hrafn`, so infra-ops maintains the
 pipeline it owns instead of routing patches through the repo holder. `vh/hrafn` is
 canonical; `stacks/hrafn/ci/` is a verified mirror.
 Commits `b6924de`, `b001d0c`, `11b9d18`, `b38c369`, `9642952`.
@@ -1,81 +0,0 @@
 # [2026-08-23] selene seat retired after losing a head-to-head; 7 aliases share one seat
 ## Why selene went
 Benchmarked against `gen` on selene's own job — 24 designed judge items with
 checkable ground truth, pairwise + absolute modes, 3 repeats, run on **both** a
 neutral JSON prompt and Selene's **native Atla template** (288 calls, free local).
 ```
 neutral JSON    selene 20/24 (83%)   gen 23/24 (96%)
 native Atla     selene 21/24 (88%)   gen 22/24 (92%)
 ```
 gen won on both templates and **selene's best sat below gen's worst**. Selene was
 given its own fine-tuned template as a fairness check before any recommendation;
 it gained one point, not three.
 **Decisive defect: selene cannot emit "tie"** — 0/2 on both templates, forcing a
 winner on every equivalent pair. For eval work that is the case that matters.
 brokkr-smithy-dev independently corroborated from the other end with a **null
 control** (an excerpt compared against ITSELF, where tie is definitional):
 `chat-judge`(selene) TIE **27/60 = 45%**, gen **60/60 = 100%**; ground-truth
 recovery on real-corpus ranking selene **47% — chance** vs gen 94%. My 83-vs-96
 understated it: on a *ranking* task selene was a coin flip. Absolute scoring on
 designed items is an easier task than ranking real text — the harness is a
 **screen, not a verdict**, and its README says so.
 Reclaimed **17.2 GiB** on ana-ml2 GPU1 (free 1,818 -> 19,450 MiB).
 ## The naming rule, restated the hard way
 I proposed repointing `selene-1-mini-8b` at gen and was **correctly overruled**:
 > never repoint a named model at a different model's endpoint — that is
 > intentionally misleading
 `chat-judge` is a **role** alias (ADR-0012: consumers bind the capability) and
 moved to gen with a deterministic judge profile copied from `image-judge`.
 `selene-1-mini-8b` is a **model** name and was removed outright — it now returns
 `HTTP 400 Invalid model name`, verified. The discriminator: *does the string
 promise a capability, or an identity?*
 ## The 7-way alias collision — the finding with the longest reach
 ```
 chat-judge  classifier  gen  image-judge
 qwen-image-bench  summarizer  summarizer-large   ->  qwen3.8-27b-uncensored :8015
 ```
 Also colliding: `gen-frontier`/`gen-frontier-reasoning`/`glm-5.2`/`glm-5.2-reasoning`;
 `ext-tts`/`gpt-4o-mini-tts`/`tts-1`/`tts-1-hd`; `reranker`/`reranker-a3-bge-v2-m3`.
 **Cross-checking a result against another alias measures nothing when they are the
 same weights — agreement is an echo, not corroboration.** Documented at the head of
 `model_list` in the live gateway config, because it belongs where people read it.
 This caught a real defect within hours: brokkr's R47 premium-corpus gate was about
 to run ~46,000 record-exposures against `gen` with `summarizer` shortlisted as an
 independent second opinion. They pinned the backing model in the preregistration
 and dropped the second-alias idea instead.
 ## Provenance seam (brokkr's pushback, adopted)
 The gateway returns the **alias** in the response `model` field, not the backing
 model — so a per-call guard catches a swap *during* a run and is blind to one
 *between* runs. **Role alias for routing, concrete model for provenance.**
 `GET :4000/model/info` with the shared key already exposes backing model +
 api_base; resolve at run start AND end and void on mismatch.
 ## Artifacts
 - Harness kept at `tools/judge-bench/` (`--models` REQUIRED — a stale default
  would silently benchmark a retired seat).
 - `stacks/selene/` keeps compose + a README explaining the retirement.
 - Technique worth stealing, from brokkr: **a control constructed so the correct
  answer is DEFINITIONAL rather than judged cannot inherit the designer's error.**
  Item vs itself; response vs its own truncation; text vs its own clauses
  permuted. Add those before adding more judged items.
 Commits `ca3c984`, `b8a5355`.
@@ -1,70 +0,0 @@
 # [2026-08-23] Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed
 ## b187 pre-stage (#405 phases 1+2)
 The matrix bridge stopped embedding the engine and became an HTTP client of the
 Conversation API, so `WORLDTREE_API_URL` became **boot-blocking** — absent from the
 container env, the bridge exits by design. Demo's compose never passed it; the next
 recreate would have crash-looped. Pre-staged on demo and personal (additive, backed
 up, verified with `docker compose config`, nothing restarted).
 **Key decision, and I got its scope wrong first.** I argued demo should stay keyless
 (no homeserver -> no rooms -> no turns -> no 401s). Right about turns, **wrong about
 scope**: the engine preflight authenticates at boot regardless of homeserver, so demo
 booted permanently degraded. Corrected — key `341c1488` minted under worldtree-dev's
 recorded authorization, vaulted, wired, three-hop hash-verified.
 ## The 69-day-stale `:latest` landmine
 All three instances pinned `WORLDTREE_IMAGE=.../worldtree:latest` in `.env` while
 running SHA-tagged images built that day. Local `:latest` = `b19afd71d7cc`, built
 **2026-06-14**. So ANY `docker compose up` — anyone's, for any reason — silently
 downgraded that service by 69 days. Same footgun as the 2026-06-15 outage.
 Re-pinned all three to their running SHAs (Worldtree #410), verified by rendering
 compose config rather than reading `.env`, containers untouched. Playbook at
 `playbooks/repin-worldtree-image.yaml`.
 **`worldtree-pinned` was the worst case:** the instance whose entire purpose is being
 frozen was running a **dangling image with no repo tags**, kept alive only by the
 running container. One `docker rm` from garbage collection. Tagged
 `:446e5807bf43` first, then pinned.
 The guard I wrote had two bugs the pinned case exposed: it compared the container's
 `.Config.Image` **string** (only the tag it was CREATED from — pinned was created
 from `:latest` back when that meant 446e5807), and it reported CHANGED
 unconditionally. Now compares **image IDs** and skips when already correct.
 ## Matrix homeserver ownership
 Operator ruled: **personal owns the Matrix bridge.** The appservice tokens were never
 missing — both sat at length 64 in the vaulted dev `env.sh` while both deployed
 instances had them at length **zero**. Someone wired four of six Matrix vars and
 stopped. Wired them into personal, three-hop verified.
 **The trap worth remembering:** Synapse's registration pointed at
 `http://10.100.10.50:8009` — nh3-dev, a dead epoch, with transaction 2801 queued at
 512s backoff. The natural fix (swap the IP) gives `10.250.50.152:8009` which is
 **DEMO's** bridge, and Synapse can reach both — it would have connected, delivered,
 and looked correct while routing the operator's live rooms to the demo instance.
 **Personal's bridge is :8010.** `docker port` is ground truth.
 Corrected the URL, restarted Synapse (healthy in 32s after 3.5 months up), verified
 `GET /_matrix/app/v1/ping -> 200` from inside the Synapse container. worldtree-dev's
 smoke passed first try: room created, mimir accepted the invite, a real engine turn
 ran, mimir replied in persona voice. #408 closed.
 ## Open on worldtree-dev's side
 - **#411** — personal's bridge logs `Debug sink init failed: Permission denied:
  /app/sessions/debug_rooms.json`. It creates two debug rooms but cannot persist
  their IDs, so **every restart mints a fresh pair on the live homeserver**. Room
  litter that compounds silently. Needs a which-container-writes-what check on the
  sessions volume before anyone chowns it.
 - Bridge/engine agent-roster drift: 6 of the bridge's 9 configured agents are not
  listed by the engine on either instance.
 - Historical Domari pairwise verdicts from the selene era are coin-flip-grade
  (see [[2026-08-23-selene-retired-alias-collision]]); worldtree-dev banked that so
  no future arc leans on them without re-judging.
 Commits `064181a`, `bb19a96`.
@@ -1,50 +0,0 @@
 # [2026-08-24] ana-gw public admin surface closed to zero, ACME listener included
 WAN admin was opened at the start of the session as a cutover contingency
 ("so I don't have to drive down there"), then closed again on operator
 instruction once the AES-128 work landed. Net result: **the FortiGate's WAN
 address now exposes no TCP port at all.**
 ## Final state
 External scan of `38.120.12.42`, 55 ports: **nothing open**. Verified from two
 sites. `wan1 allowaccess` = `ping`; `infra-ops` trusthost back to `10.0.0.0/8`.
 **Consequence to hold: there is no out-of-band path to ana-gw.** If both tunnels
 drop it is console-only. Re-open is two one-liners (allowaccess + trusthost) —
 both are recorded in auto-memory `reference_fortigate_ana_gw_access`.
 ## Port 80 was the FortiOS ACME listener, and I got it wrong first
 `38.120.12.42:80` answered a bare 403 (`ACME Access Only`, 101 bytes) with
 `allowaccess` set to ping only. First diagnosis — "an ISP transparent proxy" —
 was **wrong**, and the reason is worth keeping:
 > The sniffer filter was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
 > matches inbound only**, so outbound SYN-ACKs were excluded *by construction*,
 > and concluding "the box sends no SYN-ACK" from that capture was unsound.
 Re-run bidirectionally (`host … and tcp port 80`) it immediately showed
 `wan1 out 38.120.12.42.80 -> <scanner>: syn ack`. **Rule: to test whether a box
 *answers*, the filter must be bidirectional.**
 The listener is opened by `config system acme / set interface "wan1"` and
 **bypasses `allowaccess` by design** — FortiOS needs port 80 for HTTP-01. It
 was disabled (`config system acme / unset interface`); the LE cert (`ana-fw.pfi`,
 valid to 2026-10-27) is untouched and simply stops renewing, which is fine
 because WAN admin is closed and the box is being replaced.
 ## Retracted in the same pass: the "four all-port VIPs" alarm
 Claimed four VIPs were unrestricted all-port static NAT. **False.** A FortiOS
 VIP is scoped **two** ways — `portforward`+`extport`, *or* a `service` binding
 on the VIP object — and only the first was checked. All 14 VIPs are scoped;
 `Rustdesk` is TCP 21115–21119, `ssh-mapped-2223` is TCP 2223 only.
 Ground-truth external scan of all six public IPs is recorded in
 `reference_fortigate_ana_gw_access`. Configured-but-dead: `:8443`
 (mattermost-calls), `:8444` (webdav-nas), `:8880` (Kokoro-In) — tidy-up
 candidates for the OPNsense translation, not exposure.
 **Lesson, twice in one session: measure from outside instead of parsing config.**
 Both wrong answers came from a filter that answered a different question.
@@ -1,167 +0,0 @@
 # `[2026-08-24]` char-rp seat: OOM root-cause, Gemma-4 MoE swap, and the abliterated trainee base
 One evening, one thread with brokkr-smithy-dev, five commits: `850e0c3`,
 `27155c0`, `f509668`+`24e8826`+`1bd90ea`+`3446367`+`8d6a939`, `14ff4a3`,
 `019ccff`, `5415fd4`.
 ## 1. The seat was crash-looping, and the cause was NOT its config
 `vllm-meromero-rp` reported up-but-unreachable, RestartCount climbing (13 by the
 time it was examined, not the 4 first reported). Startup logs looked clean all
 the way through weights, `torch.compile` and CUDA-graph capture, then:
    torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 336.00 MiB.
    GPU 0 has a total capacity of 94.97 GiB of which 195.19 MiB is free.
 **⚠ `--gpu-memory-utilization` SIZES THE KV CACHE AND DOES NOT COVER CUDA
 CONTEXT, GRAPHS OR NON-TORCH OVERHEAD.** gen is configured at 0.43 and actually
 held 45.6 GiB. char-rp was at 0.51. The pair was committed to 0.94 of the card
 with ~0.6 GiB of real headroom — it fit on the 21st and stopped fitting on the
 24th.
 Dropped char-rp to 0.47: ~4.8 GiB margin, KV 27.36 → 23.56 GiB, 430,825 →
 371,023 tokens against a max-model-len of 262,144. **Cost nothing usable** — the
 pool still holds 1.4x a full-length sequence; what is lost is concurrent long
 requests, not context.
 **⚠⚠ THE MISSING HALF, found later that evening: gen's footprint GROWS WITH
 UPTIME.** Same container, same 0.43: **45.6 GiB after ~3 days up, 38.5 GiB
 freshly restarted** — ~7 GiB apart. Nothing about char-rp changed between the
 21st and the 24th; *gen crept up underneath it*. **Headroom arithmetic done
 against a long-running gen is measuring a moving number.** Measure against a
 freshly-restarted one.
 ## 2. `char-rp` and `char-rp-reasoning` are ONE seat, not two
 Both LiteLLM routes point at `10.250.50.54:8016/v1` — `hosted_vllm/char-rp` and
 `hosted_vllm/char-rp-thinking`. brokkr had reported 30/80 and 80/80 failure
 rates as two failing services; it was one outage sampled twice. This also
 *improved* a result of theirs: their CoT on/off battery had assumed both aliases
 were the same weights under two chat templates, and the routing detail turned an
 assumption into a verified fact.
 (`vllm-charrp-reasoning-nvfp4`, the Heretic2 NVFP4+MTP container, has been
 stopped for 12+ days and is unrelated — it is not what that alias resolves to.)
 ## 3. The seat swapped to the Gemma-4 26B-A4B MoE
 Operator-directed straight-across replacement: same port, same
 served-model-names, so no gateway route or consumer config moved. Rationale is
 throughput under CoT — the user waits through the whole reasoning block before
 the first visible token, and the MoE measures ~114 tok/s @32K against the dense
 31B's ~40.7.
 Serving copy is `RedHatAI/gemma-4-26B-A4B-it-NVFP4` (16 GB), chosen over the
 other `-it` quants because it is compressed-tensors (`nvfp4-pack-quantized`) —
 the same loader path the outgoing seat used. Smaller weights at the same 0.47
 budget bought **1,724,110 KV tokens against the predecessor's 371,023**.
 `meromero-charrp` is retained stopped in `created` state, labelled
 `AI - Dormant`. Both stacks bind `:8016`, so rollback is **stop-then-start**.
 ## 4. ⚠ THE STALE-CHAT-TEMPLATE TRAP IS ENDEMIC, NOT A ONE-OFF
 Verified by hash across every third-party Gemma-4 derivative pulled:
 | build | lines | sha256 (normalised) |
 |---|---|---|
 | upstream `google/gemma-4-26B-A4B-it` | 390 | `6a1015c47ccfcfa6` |
 | RedHatAI NVFP4 (served) | 389 | `6a1015c47ccfcfa6` — the only match |
 | llmfan46 heretic | 365 | `0a52be69cda5ab8a` |
 | TrevorJS abliterated | 266 | `58c66fdee4afa297` |
 | jenerallee78 abliterated | 266 | `58c66fdee4afa297` |
 | prithivMLmods NVFP4A16 | 266 | `58c66fdee4afa297` |
 Three independent repos carrying the *identical* stale file means it propagated
 through the ecosystem. Consequences differ by use and **both are silent**:
 serving a mismatched template renders a different prompt; **training through
 `base/chat_template.jinja` means training on a different prompt format than
 production serves** — train/serve skew, no error, presents as a tuning failure.
 The production compose now pins the template explicitly. It is a **no-op for the
 served weights** (the A4 build ships that exact file) and permanently closes the
 class. ⚠ If `GEMMA4_MODEL` ever points at a different checkpoint, the template
 default must move with it.
 ## 5. A benchmark result was RETRACTED — below chance indicts the instrument
 A battery appeared to show Gemma at **12% contradiction detection with CoT off
 against gen's 81%**. An A16 activation-precision control was staged to test
 whether the quant scheme owned it. Then the operator asked to see the individual
 items, and the item was **ill-posed**: it presented two mutually contradicting
 statements and asked for "*the* contradicting statement", but **contradiction is
 symmetric**. The model consistently named the absolute claim — a defensible
 reading the labelling scored wrong every time.
 **⚠ THE TELL WAS IN PLAIN SIGHT: 12% ON A FIVE-OPTION TASK IS BELOW THE 20%
 CHANCE FLOOR.** A below-chance score indicts the instrument before it indicts
 the model, and neither side reacted to it. I spent the afternoon verifying repo
 names, config fields, template hashes and tokenizer vocabs — every layer of
 plumbing — and never asked whether the number itself was *possible*. **A
 preflight can be thorough and still be aimed in the wrong direction.**
 Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43
 points of contradiction detection" (on a sound instrument it **reverses**); all
 pre-fix T2 numbers. Recorded as a dated superseded-claims table in
 `stacks/gemma4-charrp/README.md` rather than a silent edit.
 **What survived:** the A16 control result — activation precision is close to free
 on this battery, every other task identical across W4A4 and W4A16 builds.
 ## 6. The abliterated trainee base — measured, not assumed
 Operator directed a low-damage abliterated instruct build. "Low damage" was
 treated as a measurable claim; the field spreads from KL 0.09 to 0.4118:
 | build | method | KL | refusals |
 |---|---|---|---|
 | **llmfan46** (operator's pick) | Heretic v1.2.0 ARA | 0.1237 | 3/100 |
 | TrevorJS | ARA-family | 0.09 | 1/100 effective, 5/686 cross-dataset |
 | jenerallee78 | ARA 2-pass | 0.1299 | 7.7% StrongREJECT |
 | huihui-ai | remove-refusals | none published | none published |
 Fleet anchor: our own work found **Heretic at KL 0.12 preserved the MTP head at
 83.7% acceptance**, so both staged builds sit inside an already-measured band.
 huihui-ai rejected — no metrics, its card calls the method "a crude,
 proof-of-concept implementation", it abliterates both thinking and non-thinking
 modes, and its parameter count runs ~738M over upstream. Operator's independent
 read matched ("huihui produces garbage").
 **Abliteration isolated properly** (stock BF16 vs llmfan46 BF16, same precision,
 same pinned template, same 192 items):
    T2 contradiction  75% → 59%   (−5 items)
    T6 spatial        75% → 88%   (+4 items)
    core            90.0% → 89.4% (−0.6 pts)
 **It MOVED capability rather than removing it** — five lost on contradiction,
 four gained on spatial, nearly cancelling. Nobody predicted a gain. **llmfan46
 stands**; no case for re-staging on TrevorJS over 0.6 points.
 ⚠ Read as ~5 and ~4 items at n=32, not as −15.6/+12.5 percent. ⚠ Says nothing
 about quantization — the stock-NVFP4 T2 figure came from n=16 against n=32,
 different item sets, n-confounded.
 ## 7. ⚠ The production compose hardcodes `--quantization compressed-tensors`
 Pointing the char-rp stack at unquantized BF16 weights crash-loops immediately:
    TypeError: CompressedTensorsConfig.__init__() missing 3 required
    positional arguments: 'target_scheme_map', 'ignore', 'quant_format'
 vLLM trying to read a quantization config out of a checkpoint that has none. 35
 restarts before it was caught. Hence `stacks/gemma4-trainee-bench/` — a separate
 ephemeral stack with no quantization flag, `restart: "no"` so a bench seat cannot
 resurrect itself and block gen's restore, and no homepage labels so it leaves no
 permanently-offline card. That detour is why a base swap is now ~5 minutes
 instead of ~15.
 ## 8. BF16 cannot coexist with gen
 48.07 GiB of BF16 weights plus gen's footprint exceeds the 94.97 GiB card before
 a byte of KV cache. Every BF16 bench window means **gen is stopped**. Two such
 windows were run and gen restored both times; the restore was triggered by
 observing the seat's own throughput logs (a large prefill burst then zero
 running/zero waiting) rather than waiting on a courtesy message.
 Cross-links: [[2026-08-24-homepage-uniform-grid]]
@@ -1,77 +0,0 @@
 # [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live
 Reported as "`scriberr.ana.internal` doesn't resolve on my Mac, and nslookup
 shows an IPv6 DNS server." Operator's diagnosis was right; the fix took three
 wrong turns worth recording.
 ## Root cause
 `esh-userland` has IPv6 PD with RA at `pref high`, and the UDM advertises
 **itself** as the resolver via RDNSS. macOS honours RDNSS and prefers it over
 the DHCPv4-supplied resolver, so queries went to the UDM — which does not know
 `.internal` — and returned NXDOMAIN. AdGuard was never consulted.
 Two adjacent gaps found while there: `esh-userland`'s **secondary** v4 resolver
 was `10.0.10.1` (the UDM itself), and `esh-server` had **DNS handout disabled
 entirely**, so every host there got the UDM and could never resolve `.internal`
 — esh-docker-vm was living proof.
 ## The three wrong turns
 1. **`dhcpdv6_dns_auto=false` alone does nothing.** It is only honoured **when an
   explicit server is supplied**. Setting it bare looks like a no-op and invites
   the conclusion that the field is inert — which I drew, wrongly. Despite the
   `dhcpdv6_` prefix it *does* drive the RA's RDNSS option on a SLAAC network.
 2. **`wan_dns1` is NOT used by the UDM's LAN-facing forwarder.** Setting it to
   AdGuard persists, reads back, and changes nothing. Proven with **fresh
   uncached ad domains** — AdGuard blocklists answer `0.0.0.0`, the UDM returned
   real IPs. Reverted.
 3. **`force-provision` returns `rc:ok` and proves nothing** — consistent with the
   known `cmd/devmgr` behaviour.
 Every failed attempt returned `rc: ok`. **Verify by observed effect.** RAs were
 probed with a stdlib raw-socket Router Solicitation parsing option type 25
 (`rdisc6`/`tcpdump` were both absent; nothing was installed).
 ## What landed
 RDNSS **redirected** rather than disabled — better than switching it off:
 | VLAN | v4 | v6 RDNSS |
 |---|---|---|
 | `esh-userland` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
 | `esh-server` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
 The v4 secondary moved from the UDM to the **NH3 AdGuard** — reachable over
 Site Magic and authoritative for the zone. ⚠ **A secondary only fails over on
 SILENCE, not on wrong answers**: NXDOMAIN is a *successful* answer, the client
 accepts it and never retries. A secondary that doesn't know your private zone is
 a coin flip, not a spare tyre. `esh-cameras` deliberately untouched — routing
 camera DNS through AdGuard's filtering risks their cloud features.
 ## The naming scheme became real
 The resolver address is the scheme's first live use, replacing a MAC-derived
 SLAAC address that would break on a NIC change. All three `esh-server` Linux
 hosts now carry `4411:B105` ("FOR ALL BIOS"):
 ```
 esh-docker-vm  …:4411:b105:50:45     esh-pve-nas  …:4411:b105:50:55
 esh-vm-db      …:4411:b105:50:60
 ```
 Applied by an `if-up.d` hook that **derives the prefix at runtime** (self-heals
 on re-delegation), backgrounds itself with a retry (SLAAC may not have landed;
 a blocking hook would stall bring-up on a headless box), and adds nothing to
 existing config. **Not** an `iface … inet6 static` stanza — on Debian that sets
 `accept_ra=0` and would strand the host.
 ⚠ **Proxmox bridges need `accept_ra=2`.** `esh-pve-nas` had link-local only
 despite every sysctl looking right: `vmbr0.forwarding=1`, and the kernel ignores
 RAs on a forwarding interface unless `accept_ra` is explicitly `2`. Fixed with
 `accept_ra_defrtr=0` alongside, so it takes the prefix but **declines the default
 route** — an IPv6 identity with no change to a hypervisor's routing. Expect this
 on every Proxmox node when its LAN gets v6.
 Canonical: `docs/pfi/ipv6-naming-scheme.md`. UniFi limits:
 auto-memory `reference_unifi_dns_rdnss_limits`.
@@ -1,308 +0,0 @@
 # `[2026-08-24]` Homepage: remote-label consumption re-verified, then the board relaid out on a uniform grid
 Prompted by the operator: *"Homepage on esh-vm-docker lists remote dockers and
 can absolutely consume their labels, please verify again. I am still
 unsatisfied with the layout and aesthetics."*
 ## The verification — the operator was right, and the record now says so
 **Homepage on `esh-docker-vm` discovers services by container label from all
 five Docker engines in `conf/docker.yaml`, not just its own.** This is not an
 inference; `GET /api/services` returns every card's `server` field, and the
 2026-08-24 snapshot resolves to:
 | `server` | host | label-discovered services |
 |---|---|---|
 | `ana-pfi-docker` | 10.250.50.70 | 30 |
 | `irv-ml1-docker` | 10.100.79.3 (over WireGuard) | 15 |
 | `ana-ml2-docker` | 10.250.50.54 | 14 |
 | `esh-vm-docker` | 10.0.50.45 (the dashboard's own host) | 13 |
 | `nh3-pfi-docker` | 10.100.50.40 | 2 |
 **74 of 107 cards are label-discovered, and only 13 of those come from the
 dashboard's own engine** — the other 61 are read off four remote hosts,
 including irv-ml1 across the WireGuard tunnel. The remaining 33 carry
 `server: null`: those are the manual `services.yaml` entries — hardware, BMCs,
 hypervisors, printers, and user-level systemd services that have no container
 to label in the first place. **That null is the only thing "not label-driven"
 about this dashboard**, and it is a property of the entry, not of the host it
 points at.
 ⚠ If a future session doubts this again, the check is one command and takes two
 seconds — do not reason about it from the docs:
 ```bash
 curl -s http://10.0.50.45:5100/api/services \
  | jq -r '.[].services[] | .server' | sort | uniq -c
 ```
 ## What was actually wrong with the layout
 Measured with Playwright against the live board (per-group `card=` width, card
 height spread, and a geometric title-vs-status overlap test), not judged by
 eye:
 - **Card width changed at every group boundary.** `columns:` is not a density
  dial — it sets `lg:grid-cols-N` for one group, so it fixes that group's card
  width. Notes rendered a single **1464px** card; News and Media **728px**;
  Eval & Retrieval **286px**; everything else 360px. Scrolling the page, the
  grid resized five times.
 - **Long names printed underneath their own status pill.** Measured by
  re-injecting the old rule and testing the title text node's box against the
  status cluster's box: **6 cards, all on the AI tab** — 3 in Inference, 2 in
  Dormant, 1 in Eval & Retrieval; zero on the other three tabs, which is why
  it survived earlier passes. Root cause is a genuinely counter-intuitive one:
  the rule reserved a
  78px gutter with `padding-right` and relied on `overflow: hidden` to hold it,
  but **overflow clips at the padding box, not the content box** — so the
  reserved gutter was spill room the title rendered straight through. The
  intended `text-overflow: ellipsis` never fired either, because the ellipsis
  is painted by whichever block's own line overflows, and here that is the
  anonymous box around the bare title text node, which does not carry
  `overflow`.
 - **`AI Systems` / Scriberr was on all four tabs** — the 2026-08-18 UltraSeedbox
  bug recurring, this time arriving from a container label rather than from
  `settings.yaml`.
 - **Icons were grey smudges.** Homepage masks every glyph over
  `--color-logo-start/stop`; stock slate-400 → slate-700 sinks the bottom half
  of each icon into the card fill.
 - Bookmark groups and Jellyfin's trailing stream rows were the two components
  the theme had never reached.
 ## The fixes
 `stacks/homepage/conf/settings.yaml` — **all 20 groups to `columns: 4`.**
 `stacks/homepage/theme/australis.css.in` → rebuilt → `conf/custom.css`:
 gutter held by wrapping, description clamped to 3 lines (floor still 2), icon
 ramp overridden, bookmark + trailing-widget components themed, group gap
 10px → 22px. `stacks/scriberr/compose.yaml` — `homepage.group` → `AI - Audio
 Tools`, container recreated on ana-ml2.
 After: **every group renders at card=360**, and the collision count is zero.
 Before/after, all four tabs: `http://10.100.10.50:8090/b/homepage-relayout/`
 (24h TTL; also on the standing link board).
 ## ⚠ Three traps worth carrying forward
 1. **"Columns = member count" is RETIRED** (it was the 2026-08-18 rule). It was
   avoiding dead cells in a short last row and bought a worse defect. A short
   last row is what a grid looks like; a card wider than its neighbours is what
   a mistake looks like.
 2. **A `:root` override of a Homepage theme variable is silently ignored.**
   Homepage sets `--color-logo-*` on `.theme-slate`, and that class is on the
   `<html>` element — the same element `:root` matches. `.theme-slate` (0,1,0)
   beats `:root` (0,0,1), so the override does nothing and looks like the
   variable is not the one in play. `html[class]` (0,1,1) wins, and does not
   hard-code which `theme-*` class is active. Specificity alone is not enough
   either: a custom property resolves from the *nearest* ancestor that sets it,
   so the override has to land on `<html>`, not on `<body>`.
 3. **The post-recreate tab-bar loss is INTERMITTENT, not guaranteed.** The
   2026-08-19 note reads as though every recreate costs up to an hour of broken
   render. This recreate came up correct within 10 seconds — fresh payload on
   the first poll, all four tabs clickable a minute later. Recreate, *check*,
   and only then walk away if it is actually in the broken state.
 Also re-confirmed, since the change depended on it: **a `settings.yaml` edit
 needs a container recreate, not a restart.** `docker restart homepage` left the
 old `"columns":1` payload embedded in the served HTML with the correct file
 mounted and readable inside the container; `compose up -d --force-recreate`
 cleared it immediately.
 ## Deliberately not done — operator's call
 The Main tab still opens on three sparse bands: **Notes** (1 member) and
 **Games** (1) each burn a full 4-wide row, and **News** has 2. Merging Notes +
 News, or folding Games into Apps, would tighten the top of the page — but that
 is information architecture, not layout, and the group names are the operator's.
 Surfaced rather than done.
 → **Resolved in pass 2 below**, where the operator delegated the naming
 ("re-categorize however you want"). Notes + News became `Daily`, Games folded
 into `Apps`, and the `AI - Audio Tools` placement in this pass was superseded
 (Scriberr moved on to `AI - Studios`).
 ---
 # `[2026-08-24, pass 2]` Recategorised on "do I open this?", API groups collapsed
 Operator, after seeing pass 1: *"You can re-categorize however you want.
 service networking tab is uneven, you can split out the adguard cards, etc.
 most of the issues are that tools I use and have a UI are interspersed with API
 endpoints which are largely informational only. They might even go in their own
 cards or start collapsed."*
 ## The axis
 Every group is now either **tools** (expanded, top of tab) or **endpoints** (an
 API, a broker, an agent — `initiallyCollapsed: true`, bottom of tab). A
 collapsed group still renders its eyebrow and rule, so presence costs one line
 instead of two rows.
 Second, quieter rule that fell out of the same pass: **a group's members should
 all carry a widget or none should.** A stat strip adds ~50px, so one widget card
 in a row of plain ones opens a void under the plain ones — which is most of what
 made the 13-member `Service Networking` band look broken.
 ## Shape
 - **Main** — `Daily` (Memos, Miniflux, Nevermore, SearXNG — replaces the
  1-card Notes and 2-card News bands), `Monitoring`, `Apps` (12; absorbed the
  1-card `Games` band), `Media`, `UltraSeedbox`.
 - **AI** — `AI - Gateways & Chat` (8) and `AI - Studios` (6) expanded; then
  `AI - Inference` (7), `AI - Eval & Retrieval` (4), `AI - Speech (TTS)` (4),
  `AI - Audio Tools` (2), `AI - Dormant` (6) all collapsed.
 - **Toolchain** — `DNS & Filtering` (3), `Reverse Proxies` (2),
  `Compose Consoles` (5), `Toolchain` (3), `Agents (no UI)` (6, collapsed).
 - **Infrastructure** — unchanged; every card there is already a console.
 Measured after: every group `card=360`, and `DNS & Filtering` and
 `Reverse Proxies` both `h=134..134` — dead flush.
 ## ⚠ The move that made it affordable
 **The sixteen GPU-backed model seats were NOT relabelled.** `homepage.group` is
 read at container **creation**, so renaming `AI - Inference` to something
 clearer would have meant recreating six vLLM seats plus four eval seats plus
 four TTS engines — multi-minute model reloads on endpoints peers reach through
 the gateway. Order plus `initiallyCollapsed` buys the same separation for free,
 so the names stay ugly on purpose. **Do not spend that recreate on a label.**
 28 containers *were* relabelled — all cheap web services — via five rerunnable
 elway playbooks, `playbooks/homepage-regroup-<host>.yaml`. The canonical
 `stacks/` tree was synced to match afterwards, so intent and reality agree.
 `initiallyCollapsed: true` is a per-group key in `layout:`; confirmed present in
 this build (`defaultOpen: !(group?.initiallyCollapsed ?? global)` in
 `/app/.next/server/pages/index.js`).
 ## AdGuard (ANA) gained its widget, and the credential is fleet-wide
 It was the only AdGuard without a query/blocked/latency strip, so it sat short
 beside two tall siblings. **One `infra-ops` AdGuard login authenticates against
 all three instances** (ANA `:8053`, NH3 `:8080`, ESH `:8080` — all returned 200
 on `POST /control/login`, verified 2026-08-24). Vaulted at
 `secret get nh3-dev/adguard-infra-ops-password`; written to
 `/opt/docker/compose/adguard-ana/.env` (0600, root) and never into git. Its icon
 was also the odd one out (`mdi-dns` against two `si-adguard`).
 ## ⚠⚠ `initialSettings":{}` — the tab-bar mystery is a SWALLOWED EXCEPTION
 The biggest durable finding of the day, and it cost ~25 minutes. Full write-up
 in `stacks/homepage/README.md`; the short version:
 `initialSettings":{}` in the served HTML is **the catch branch** of the page's
 data loader, not a warm-up and not a cache. And the error can vanish without
 trace: the logger is assigned as the first statement *inside* the same `try`,
 and the `catch` only logs `if (logger)`. If the logger is what threw, nothing is
 written anywhere — which is exactly what was observed.
 Ruled out by measurement, do not re-run: `/api/services`, `/api/bookmarks`,
 `/api/widgets` and `/api/hash` all return **200 with correct content** while the
 page serves `{}`; restoring the previous known-good `settings.yaml` reproduces
 it identically; `/api/validate` returns `[]`; disk and permissions are fine.
 **One-command test:**
 `curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}'`
 **What broke the streak:** three consecutive recreates came up empty, then
 rolling the 8.6 MB `conf/homepage/logs/homepage.log` aside and recreating healed
 it within 15 seconds. That is one observation, not proof — but it is a coherent
 mechanism (oversized log → logger init throws → silent catch) and it is the
 cheapest thing to try first next time.
 ---
 # `[2026-08-24, pass 3]` Rebuilt on Australis Skyfall — dual theme, light shipped
 Operator supplied the Skyfall design-system README and said "Go full with
 skyfall."
 ## The bundle was already in this repo's git history
 **The Skyfall tokens did not need to be hunted down.** A predecessor vendored
 them on 2026-08-19 and a later commit deleted them; git kept everything:
 ```bash
 git show 45c1995:stacks/homepage/theme/colors.css      # 208 lines, BOTH themes
 git show 45c1995:stacks/homepage/theme/layout.css      # calm-depth tokens
 git show 45c1995:stacks/homepage/theme/typography.css
 git show 45c1995:stacks/homepage/theme/fonts/Supreme-{400,500,700}.woff2
 ```
 `colors.css` carries `:root` (dark) **and** `[data-theme="light"]` (Skyfall
 Day) — so the light ramp is canonical, not derived. That killed the entire
 objection from the previous answer, which was correct only about the
 `australis-design` skill ("Always dark first. No light mode in this system").
 **Skyfall is the dual-theme derivative; australis-design is the terminal
 theme. They are different systems and only one of them has a light mode.**
 ## ⚠⚠ REMOVING `theme:` FROM settings.yaml BREAKS THE DASHBOARD
 The documented way to get Homepage's own light/dark toggle is to leave `theme:`
 unpinned. **Do not.** With the key absent, the page's data loader throws and its
 catch branch serves `initialSettings: {}` — no tab bar, no layout, no i18n.
 Measured, not inferred: six force-recreates over seven minutes all came up
 empty with the key removed; restoring `theme: dark` rendered correctly on the
 next recreate in **12 seconds**. `/api/services` stays 200 and fully correct
 throughout, which is exactly why this reads as a caching or warm-up problem and
 is not one.
 This is the first *confirmed* trigger for the long-running "tab bar goes
 missing" mystery. It does not explain every occurrence (the symptom has
 appeared with `theme:` present), but it means **the first diagnostic step is
 now `git log -p -- stacks/homepage/conf/settings.yaml`**, not container
 archaeology. Also retires an earlier lead from this same session: rolling the
 8.6 MB `homepage.log` aside once coincided with a recovery, but did nothing
 during the `theme:`-key episode — coincidence, not cause.
 ## So the toggle is ours
 `conf/custom.js` renders it (was an empty placeholder). Precedence:
 1. explicit choice — `localStorage['skyfall-theme']`, written by the toggle;
 2. OS preference — `@media (prefers-color-scheme: light)`;
 3. dark — Skyfall's default.
 `theme/build.py` re-emits each vendored `[data-theme="light"]` block twice: as
 `[data-theme="light"], html.light`, and inside the media query scoped to
 `html:not([data-theme="dark"]):not([data-theme="light"])`. **That `:not()` pair
 is what lets a stored *dark* choice survive a light-mode OS.** Verified across
 both OS preferences: load, click, click again, reload — all four correct.
 ⚠ Homepage keeps its own `class="dark scheme-dark theme-slate"` on `<html>`
 regardless, because `theme:` is pinned. That is fine and was checked
 explicitly: with the dark class present AND `data-theme="light"`, every themed
 surface resolves to Skyfall Day, because our rules carry `!important` on the
 surfaces Tailwind's `dark:` variants would otherwise claim. **`data-theme` is
 the control surface; the class is not.**
 ## The anti-fork guard is now mechanical
 `build.py` records the SHA-256 of each vendored file and **fails the build** on
 a mismatch, rather than warning. A vendored file is either byte-identical to
 the bundle or it is a fork wearing the bundle's name. Overrides go in
 `skyfall.css.in`, which is written entirely against the semantic layer
 (`--surface-*`, `--text-*`, `--border-*`, `--success/--danger/--warning`) — no
 raw family tokens, no colour literals.
 The one place a literal is unavoidable: Homepage consumes
 `--color-logo-start/stop` as `rgb(var(--x))`, which cannot take an `oklch()`.
 Those four values are exact sRGB conversions of real tokens (`--sea-80`,
 `--blue-base` for dark; `--sea-40`, `--blue-deep` for light), computed rather
 than eyeballed, with the conversion recorded in the file.
 ## Deviations, all deliberate and all written down
 - **The aurora ribbon under the tab bar is gone.** Skyfall sanctions exactly two
  accent expressions — the active rail and hero-only glows — and a decorative
  gradient across the chrome is neither. The colour moved to a 2px accent bar
  plus `--accent-soft` fill on the active tab, which *is* the rail.
 - **Widget stat values moved from the display face to mono**, per Skyfall's
  "numbers and telemetry are always `--font-mono`".
 - **Two font substitutions**: Space Grotesk for Bespoke Sans, JetBrains Mono
  for Victor Mono. Only Supreme was ever vendored, and Skyfall's own notes call
  Victor Mono "user-supplied". Two-line swap when the real faces arrive.
 Dark + light, all four tabs: `http://10.100.10.50:8090/b/homepage-skyfall/`
@@ -1,46 +0,0 @@
 # [2026-08-24] Scriberr transcription deployed on ana-ml2, GPU1
 Self-hosted audio/video transcription + diarization. Operator chose GPU
 placement over ana-docker (8 cores shared with 50 containers, 37 GB disk)
 against ana-ml2's 96 cores, `/tank`'s terabytes and GPU1's headroom.
 **Live:** `http://scriberr.ana.internal:8080` (DNS alias added), health `healthy`,
 all seven backends up, zero failures: `whisperx pyannote sortformer parakeet
 canary voxtral openai`. ~30 GB of weights on `/tank`.
 Stack: `stacks/scriberr/`. Full gotcha list in auto-memory
 `reference_scriberr_ana_ml2`.
 ## Three upstream bugs, none of them ours
 **1. The Blackwell image does not exist.** Upstream's README documents
 `scriberr-cuda-blackwell`; GHCR has **no tags for it**. Published
 `scriberr-cuda` covers sm_61–sm_89 only — on these sm_120 cards it fails or
 silently drops to CPU. The real sm_120 path is `Dockerfile.cuda.12.9`
 (CUDA 12.9.1, cu128 torch), **built from source**. Do not "simplify" the compose
 back to the published image.
 **2. It must run as uid 10001, not 1000** — and the error lies:
 `unable to open database file: out of memory (14)`. Error 14 is
 `SQLITE_CANTOPEN`, not an OOM, on a box with 566 GB RAM. That Dockerfile creates
 `appuser` at 10001 (Ubuntu 24.04 owns uid 1000 as `ubuntu`) and chowns `/app` to
 it, while the entrypoint's PUID remap covers only the data dirs.
 **Isolated by elimination**: SQLite writes fine to `/tank` as 1000 → not the
 mount; fails on a plain named volume too → not the storage; the **published CPU
 image works at PUID=1000** because there `appuser` *is* 1000.
 Generalisable: *when a container "permission" bug appears, compare the uid the
 image was BUILT for against the uid you are RUNNING as.*
 **3. `UV_LINK_MODE=copy` is required.** Scriberr builds each backend's Python env
 with `uv` at start; uv's reflink mode fails on overlayfs+ZFS with
 `Failed to clone … Resource temporarily unavailable (os error 11)`. **Partial
 failure** — WhisperX and PyAnnote came up and the app looked fine while Parakeet
 and Sortformer were silently absent. Occurrences 2 → 0 after the fix.
 ## Related
 `speaches` on irv-ml1 **stopped** the same day (stack retained, one command to
 restart): Eyra was abandoned pre-implementation because Scriberr covers the need,
 leaving it with no consumer. Scriberr runs its **own** WhisperX in-container and
 is **not** a speaches consumer. Idle footprint at stop was 274 MiB, not the
 ~5.9 GB quoted — that figure is the loaded-model working set.
@@ -1,92 +0,0 @@
 # ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for
 `[2026-08-25]`
 ## The run
    1312/1312 in 7:21:52   train_loss 2.793   epoch 1.0
    20.1 s/it FLAT across every 100-step window   (round 1: 35-46.5 s/it)
    adapter: /tank/erp-tune/run-01/adapter/  410 tensors, provenance.json
 **47% faster than the round-1 projection of 13.85h**, from two changes: the
 bucketed sampler and flex attention. Rate was flat — 19.7 / 19.8 / 20.4 / 20.3
 across the four 100-step windows — which means the 35-46.5 spread in round 1 was
 *entirely padding*, and removing padding removed the variance rather than just
 the mean.
 ⚠ **I quoted three different ETAs (6.9h, 8h, 7.3h) before I started using a
 rolling average.** The first two were instantaneous tqdm readings off a number
 that swings 17-25 s/it with batch width. Only the rolling rate was honest. Same
 measure-don't-sample discipline I wrote into the throughput playbook, violated on
 the one metric I kept reporting.
 ## lora_B gate — PASSED, twice
    checkpoint-100   205/205 non-zero, median norm 0.829
    final adapter    205/205 non-zero, median norm 1.708
    vision_tower tensors: 0 on both
 Median norm rising 0.829 -> 1.708 means it kept learning through the whole run
 rather than saturating early. This check **never ran in round 1** (died at step
 19, first checkpoint was 100) and it is the only failure mode that stays
 invisible until the acceptance gate reports base-identical numbers.
 ## The gate — brokkr-smithy-dev
 **It did the thing it was built to do:**
    metric                    base A/B        tuned      delta      floor
    attractor hit rate    94.8% / 96.8%       84.5%     -11.3pt     2.0pt
    diversity (pairwise)  0.213 / 0.221      0.3948     +0.178     0.008
 Diversity moved **22x its own noise floor**. Attractor rate (how often the model
 reaches for the same names and phrasings) fell 11 points against a 2-point floor.
    T1 100 · T2 95 · T3 96-97 · T4 98 · T5 100 · T6 81-82 · core ~94.2
    memorisation: 0.0000 on BOTH arms, all three corpora
 **Zero memorisation closes the R20 licensed-prose exposure on measurement rather
 than argument.**
 ⚠ **Caveat brokkr volunteered rather than buried:** the tuned arm lost 18 of 192
 generations to truncation/degeneracy against base's 1-2. Lopsided exclusions
 plausibly flatter the diversity magnitude. Direction is unambiguous at 22x floor;
 the number carries an asterisk.
 ## The noise-floor near-miss — the methodology lesson
 brokkr was one step from reporting a 13-point T6 regression **that sat inside
 twice his instrument's own variance.**
    --per-type 32     max swing across tasks:  9 points
    --per-type 128    max swing across tasks:  1 point
 His gate criterion is "no task regresses by more than one item" = 3.1 points at
 n=32. **The instrument's own run-to-run noise was 3 items.** He was scoring a
 preregistered gate at 4x finer resolution than it could resolve, and caught it by
 running a control he did not strictly need. Quadrupling n collapsed the noise
 exactly as binomial statistics predicts.
 ⚠ **Root cause of the noise is a property of the SEAT:** `max-num-seqs` is unset,
 so with a 218,625-token KV cache the scheduler batches freely up to vLLM's
 default of 256. Continuous batching changes reduction order and borderline items
 flip. Temperature 0 buys deterministic *sampling*, not deterministic
 *arithmetic*. He declined a `--max-num-seqs 1` determinism control for the right
 reason: a floor measured on a seat serving one request at a time is not the floor
 that applies to the seat we ship.
 ## The confound I built and he caught
 I optimised a pipeline for production and then handed him its output as an eval
 instrument **without asking whether those were the same job.** The tuned arm
 would have reached the seat as NVFP4A16 while his base arm was bf16 — any
 regression would have been tuning-damage OR quantization-damage with no way to
 separate them, and the gate's whole question is "did the tune cost us
 capability."
 **Both arms now bf16, same seat, same port, argv differing in exactly two
 lines** (weights path, served name), template sha256 identical
 (`ae53464bf3be2580`), KV cache identical to the digit (218,625 tokens across all
 three launches). Quantization moved *downstream* of the gate.
 See [[2026-08-25-refusal-retention-probe]] for the axis his gate did not have.
@@ -1,86 +0,0 @@
 # The 8.6% MFU was an accounting artifact — attention on Ampere kernels
 `[2026-08-25]`
 ## The answer
 **Real utilisation was 17-20%, inside the honest stock band.** The 8.6% divided
 the *intended* (windowed) FLOPs by the wall time the *dense* reality took.
    nominal billed      27.1 TFLOPS x 34.85 s        =  9.4e14 FLOP
    dense-sliding extra 25 layers, 2 seqs, 4 passes  = +8.2e14
    padded full layers  lose the causal skip         = +3.5e14
    work performed                                   ~  1.8e15  = 51-61 TFLOPS
 The card was doing ~2x the arithmetic the architecture specifies, and the excess
 was the sliding window being computed and thrown away.
 ## Two independent methods agreed
    scaling fit (3 points, 2 params, residuals <3ms over 8x range)
        A = 6.87e-4 s/token   B = 8.85e-8 s/token^2
        quadratic share: 20.9% @ w=2048 -> 67.8% @ w=16384
    kernel table (device rows only)
        attention   22,835.8 ms   65.2%   fmha_cutlass*_sm80
        dense GEMM   2,774.0 ms    7.9%
        other        5,739.0 ms   16.4%
 **67.8% vs 65.2% — 2.6 points apart, no shared assumptions.** The two-term fit
 needed no constant term, which refutes launch-bound outright (3,840 expert-GEMM
 launches per forward are not the cost).
 ## The mechanism, source-verified by brokkr's panel (arm: Bil)
    masking_utils.py:292-301  _ignore_causal_mask_sdpa requires
                              kv_length < local_attention_size. 16384 >= 1024,
                              so THE SLIDING MASK ALWAYS MATERIALISES.
    sdp_utils_cpp.h:259-267   flash rejects ANY explicit mask
    sdp_utils.cpp:647         cuDNN head_dim capped at 128 -> unreachable
    Context.h:480-485         prefer-cuDNN needs major 9 or 10; sm_120 is 12
 ⚠ **The kernels are `sm80` — Ampere-generation CUTLASS on a Blackwell card**,
 with the forward on `gmem`, the memory-efficient backend's slowest fallback tier.
 ## What actually fixed it
 **Bucketing (bucket-to-pair, shuffle-to-mix)** — 29.9% padding -> 0.0%, and
 78.3% of micro-batches become exactly zero-pad, which puts the 5 global layers
 back on `is_causal`. Measured: padding costs **9.4% MORE time for 24% LESS work**
 at fixed width, because an explicit mask knocks those layers off the fast path.
 ⚠ **Bucket size is NOT a diversity knob.** Swept across a 256x range, roots per
 accumulation window stayed flat at 3.54-3.61. The global micro-batch shuffle does
 all the mixing; the bucket only costs padding. Use the tightest bucket.
 **flex_attention** — Triton-generated so it compiles for sm_120 instead of
 shipping sm_80 binaries. 21.7x on sliding layers, 2.1x on global. Needs mandatory
 `kernel_options` at 32x32 blocks: 64x32 needs 102,400 bytes against a
 **101,376-byte hardware ceiling** — misses by 1 KB, and Triton is already opting
 into the full 99 KB, so it is the card, not a default.
 ## ⚠⚠ The trap that produced TWO wrong published conclusions
 `torch._dynamo` defaults to a recompile ceiling of **8**. Every distinct sequence
 width is a new shape. On hitting the ceiling dynamo does not error — it silently
 falls back to UNCOMPILED flex, which is ~20x slower AND documented to *"not work
 with the backwards pass and may produce incorrect results."*
 That artifact produced a bogus **0.76x slowdown** and a bogus **2.9% loss
 divergence**, and I believed and reported both. Raising the limit to 256 flipped
 the speed result to 1.41x.
 The loss divergence turned out to be real but benign — adjudicated against fp32
 MATH ground truth, both backends sit ~2e-3 from truth with flex fractionally
 CLOSER at every width. **Do not re-open it by comparing the two backends to each
 other; that cannot answer it. Compare to fp32.**
 ## Process lesson
 brokkr's panel produced **four self-retractions in ninety minutes**. Every
 retraction was a derivation; every survivor was a measurement. And the whole
 head_dim-512 SDP problem was **already documented in zerofata's published Axolotl
 config since April** — the right first stop for "why is this architecture slow"
 is practitioner configs for that exact base, before any panel.
 Playbook: `docs/pfi/training-throughput-playbook.md`, commit `7b5fd91`.
@@ -1,84 +0,0 @@
 # NVFP4A16 serving pipeline — built, validated, and the MoE landmine it found
 `[2026-08-25]`
 Pipeline at `scripts/erp-tune-serve/` (commits `6a85829`, `ab980e9`).
 Validated end-to-end against checkpoint-100 before the real adapter existed.
 ## ⚠⚠ THE LANDMINE: a `targets=["Linear"]` recipe misses EVERY MoE expert
    before linearize_moe:  427 Linears,    205 targeted, experts 0
    after  linearize_moe:  11,947 Linears, 11,725 targeted, experts 11,520
                           (30 layers x 128 experts x 3 projections)
 Gemma-4 stores each layer's 128 experts as two fused 3-D `nn.Parameter` tensors
 (`gate_up_proj` [128,1408,2816], `down_proj` [128,2816,704]) — note the absent
 `.weight` suffix. A Linear-targeting recipe resolves 205 of 427 modules and
 **zero experts**, leaving 22.84 B params (88.5% of the model) in BF16 with no
 warning.
 **This is the same defect that killed QLoRA here via bitsandbytes.** The blind
 spot is in the *checkpoint layout*, not the tool. Fix:
 `llmcompressor.modeling.moe.linearize.linearize_moe` — no registration needed,
 Gemma-4 satisfies `FusedExpertsProtocol` structurally. Playbook §3.15.
 ## Scheme: NVFP4A16, deviating from the playbook default, on measured grounds
 brokkr benched the W4A4 quant of this checkpoint at **12% on contradiction
 detection with CoT off against gen's 81%** — the signature of 4-bit input
 activations on a reasoning-dense task. Plus W4A4 KLD is 2-4x worse past ~10k ctx
 on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill cost accepted.
 ⚠ Several HF repos named `…-NVFP4A16` declare `input_activations num_bits 4` —
 W4A4 wearing an A16 label. The script refuses if the emitted config says 4.
 ## Four silent defects the dry run found
 1. **transformers 5.15 MIGRATES the config schema on save** — drops
   `global_head_dim`/`num_global_key_value_heads`, writes `per_layer_config`.
   transformers 5.10 (the llmcompressor venv) then reads `num_key_value_heads`
   as None and dies with `TypeError: unsupported operand type(s) for //`.
   Every working artifact on the box uses the OLD schema. Merge now downgrades it.
 2. **llmcompressor cannot auto-init a processor for a multimodal checkpoint** —
   pass the tokenizer explicitly as `processor`.
 3. **`save_pretrained` does not carry `processor_config.json`** — vLLM then fails
   with "Can't load feature extractor", which reads as a vision bug.
 4. **The quant needs more than GPU1's free 32 GiB.** `quant_with_gen_down.sh`
   stops `vllm-gen` and restores it from a trap on EVERY exit path, using
   `docker start` not `compose up` so the container returns with its exact config.
 ## Verified on the emitted artifact
    49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16)
    weight_packed 11,725 of which expert 11,520
    tokenizer truncation: clean (§3.14 trap avoided by calibrating on the
                                 encode cache, so the tokenizer is never called
                                 with truncation=True at all)
    served: Marlin NVFP4 kernel + Marlin MoE backend, coherent generation
 ⚠ The reference `nvfp4a16` artifact triggers a vLLM warning that q/k/v carry
 *different* weight global scales ("likely reduced accuracy"). **Ours does not** —
 llmcompressor 0.12 links weight observers across fused groups automatically. The
 in-house quant is better than the downloaded one on that axis.
 ## ⚠ MERGED WEIGHTS ARE MANDATORY — and not for the reason we assumed
 The open question was whether LoRA-on-NVFP4 hot-swap still silently no-ops.
 Retested on `vllm/vllm-openai:latest`: **it refuses to start.**
    AttributeError: To support LoRA for MoE model,
                    'get_expert_mapping' must be implemented
 The check is in `vllm/lora/utils.py::process_packed_modules_mapping` and branches
 on `is_moe_model()` — **quantization is not in the condition.** `gemma4.py`,
 `gemma4_mm.py`, `gemma4_mtp.py`, `gemma4_unified.py` all have ZERO occurrences;
 `deepseek_v2`, `mixtral`, `glm4_moe`, `ernie45_moe` implement it.
 **vLLM cannot serve a LoRA on ANY Gemma-4, bf16 or quantized.** Merging is the
 only path for this architecture, and it would have bitten identically on the
 unquantized base. A loud refusal is strictly better than the 0.24.0 silent no-op,
 which shipped a base model wearing the tune's name.
 ⚠ Base-viability pre-flight is now playbook §3.11 — three greps before picking a
 base. **Grep the CLASS, not the file**: `mistral.py` greps as `SupportsLoRA=0`
 and is fully LoRA-capable via inheritance from `LlamaForCausalLM`.
@@ -1,64 +0,0 @@
 # Refusal retention — the axis the gate did not have, and the axis I measured wrong
 `[2026-08-25]`
 ## Why it exists
 brokkr's gate measures reasoning (T1-T6), craft (diversity/attractor) and
 regurgitation (memorisation). **Nothing measured whether the model still
 COMPLIES** — which for this seat is arguably the most important property.
 The risk is specific to our operation order. We do **tune(abliterate(stock))**,
 so the tune has 57.7M tokens of opportunity to walk the abliteration back. *A
 tune that gains 41 items of contradiction detection and quietly re-installs
 refusals is a failed seat that passes the entire gate.*
 ## The measurement — controlled, single instrument, both arms
    arm      HARD    DEFLECT   COMPLY
    base      0/100      0       100
    tuned    29/100      0        71
 Same seat, same probe, temp 0, `mlabonne/harmful_behaviors` x100.
 Probe: `scripts/training-probes/refusal_probe.py`.
 **The tune added 29 general-harm refusals where the base had none.**
 Two things fell out:
 - **The instrument validates.** Base measured 0/100 on my generated-text regex
  against Heretic's recorded 3/100 from a first-token-probability scorer. 0 vs 3
  is agreement — the incomparability worry was right caution about a non-problem.
 - **DEFLECT is 0 on BOTH arms, so the free control fires.** An instrument
  artifact does not care which arm it runs against. Both zero means the model is
  **binary** — refuses in refusal-language or engages, no soft-deflection tail.
  The R19 undercount does not apply here.
 ## ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one
 `mlabonne/harmful_behaviors` is **general harm** — weapons, malware, fraud. **The
 abliteration was not run so the model would explain bomb-making. It was run so
 the model would engage with explicit fiction.** Different refusal surfaces; a
 model moves on them independently.
 I picked that set because it was cached, had a recorded baseline, and was what
 the abliteration tool used. **Every one of those is a reason it was convenient,
 not a reason it was right** — and "it has a baseline" was actively misleading,
 because a comparable number for a question nobody is asking looks like evidence.
 **29/100 general-harm refusals on a seat writing prose the operator was actively
 praising is plausibly the DESIRED shape**, not a defect. General-harm refusals
 returning while domain compliance holds is close to ideal for an internal
 creative seat. I would have reported it as damage.
 **The load-bearing cell is COMPLY 71, not the 29.** Stock refused 100/100;
 anything near that would mean the abliteration was undone. 71 complying means
 "partially walked back on one axis" — a different finding, and only one of the
 two threatens the seat.
 Domain-compliance probe (the right axis, from R19's track-2 map) is brokkr's,
 pending. Scaffold supplied: `scripts/training-probes/counted_classifier.py`
 (`2a05ae9`) — classify-never-surface, three-way, ERROR path deliberately does not
 log the exception body because an exception can echo the prompt back.
 Playbook §3.13. See [[2026-08-25-erp-tune-run2-complete]].
@@ -1,123 +0,0 @@
 # Worldtree b188 + b189 bridge cutover, and the selene metadata that lied
 `[2026-08-25]`
 Three arcs in one day, all infra-ops side, all landed.
 ## b188 — matrix.yaml pre-sync (#406/#409/#410 closed)
 From b188 the bridge reads per-agent `rendering` + `ambient_buffer_size` from
 `config/matrix.yaml` ONLY; agent `config.yaml` matrix blocks are gone from the
 image. Staged as `6417115` in `worldtree-instance-configs`, deployed to both
 instances with operator approval.
 - mimir gets thinking-to-thread + tool-call reactions + 7 reaction labels;
  forseti and lofn stay quiet.
 - **Rider #409 pruned six dead agents** (bragi, leif, troi, soong, cara, glados)
  from BOTH rosters — originally scoped personal-only, which I flagged as a
  possible oversight and it was. **The settling fact worth keeping: the engine
  roster comes from the image's baked `agents/` directory (only `config/` is
  bind-mounted), and both instances run the same image**, so instance-level
  evidence about which agents the engine lists generalises by construction.
 - Both rosters now exactly `[mimir, forseti, lofn]` — the three engine agents
  actually bridged. mask/vili/echo exist in the engine, deliberately unbridged
  (operator ruling).
 ⚠ **Edited text-surgically, not via a yaml round-trip** — PyYAML would reflow
 1,249 lines and drop every comment, and the comments are the documentation.
 ⚠ **`deploy-wt-config` uses `docker restart`, NOT `compose up`.** A `compose up`
 on corviduo-dev re-resolves the image tag and can silently swap the running
 build — which would turn an "inert pre-sync" into an unintended image roll on two
 live instances. That property is easy to lose in a future refactor of the script.
 ## b189 — #407 bridge extracted to its own repo (#404 umbrella closed)
 Bridge now `gitea.phasefinal.com/pfi/wt-matrix-bridge`, its own repo, own CI.
 ⚠ **It publishes to the `pfi` ORG, not `vh`, and the reason is structural:** `vh`
 is a **USER**, not an org. Gitea scopes user-namespace packages to the owning
 user — there are no package collaborators on a user namespace. **No service
 account can ever publish to `gitea.phasefinal.com/vh/*`.** claude-bot is an Owner
 of `pfi`, so that is where it goes. Token `wt-matrix-bridge-ci` (id 28, scopes
 `write:package,read:repository`), vaulted at
 `nh3-dev/.config/claude-bot/gitea-token-wt-matrix-bridge-ci`.
 **Minted a dedicated token rather than reuse `claude-bot-sdk-ops` or `arbo-ci`,
 both of which already carry `write:package`** — a shared credential cannot be
 revoked without collateral. ⚠ A first mint attempt succeeded then failed to save;
 Gitea returns a token value exactly once, so it was unrecoverable. Deleted the
 orphan (id 27) rather than leave a live package-write credential on the account.
 **Both instances PINNED** to `f3f8ec902267` (`e90f436`), closing the #410 shape:
 `WORLDTREE_IMAGE` was pinned and the bridge was the one drifting service.
 ⚠ The pin moved from `b178285b1cb5` because a cross-frontier bug-hunt found the
 M_EXCLUSIVE **fallback could itself litter unboundedly** in exactly the state it
 was written for. **The safety net had the same failure mode as the thing it was
 catching** — and my staged window leaned on that fallback.
 ## #411 — the debug-room failure, diagnosed twice and wrong both times first
 My theory: the alias was held by orphaned rooms. **Refuted by a directory probe
 returning 404.** The real cause: Synapse's **M_EXCLUSIVE** — an appservice may
 only create aliases inside a namespace it has RESERVED, and
 `aipa_appservice.yaml` had `namespaces.aliases: []`.
 ⚠ **I inferred a cause from a symptom that was the RESPONSE to the cause** — the
 log's "re-resolving alias" line is the recovery path firing, not evidence the
 alias exists. One directory probe settled it and I reasoned instead.
 Fixed with operator clearance: added
 `regex: '#aipa-debug-[a-z0-9_-]*:matrix\.phasefinal\.com'` (exclusive) at
 `/opt/docker/conf/synapse/aipa_appservice.yaml` on **ana-docker** (NOT
 `/opt/docker/data/`, which worldtree-dev's issue cited). **My regex was tighter
 than the proposed `#aipa-debug-.*`**, which fullmatches only because `.` also
 matches the `:` separator and would equally claim other homeservers.
 Pre-apply sweep (an `exclusive: true` claim can make Synapse refuse to START):
    aliases matching '%aipa-debug%'        0
    total room_aliases on the homeserver   1   <- why it went unnoticed this long
    rooms created by @aipa-debug          17   <- the litter, confirmed unaliased
 Synapse healthy in 40s, both bridges rode through. **The aliased create then
 worked FIRST TRY on the next personal recreate** — designed path, fallback never
 fired, both reuse mechanisms live (canonical-alias rediscovery AND the room-id
 cache).
 ⚠ The #411 writer census **inverted its own premise**: the api (uid 1000) cannot
 write `/app/sessions`, the bridge (root) can — the reverse of the issue text. But
 worldtree-dev's reconciliation is better than "backwards": pre-#407 the bridge
 ran from the ENGINE image as uid 1000, and the cutover changed the answer
 underneath the issue. Both readings were true at their timestamps. **No live
 writer exists for that path anyway** (`sessions.path` is a legacy default, #330
 moved the tools off it), so the chown is optional future-proofing.
 ## selene-1-mini-8b — a config that lied about what answers
 forseti's fleet sweep found the only genuine residual in
 `worldtree-instance-configs`, and it was **live on both boxes, not just drifted
 in git.** Fixed in `a77639d`.
 Routing was never broken (`model: "chat-judge"` stays), but `display_name` said
 "Selene 1 Mini 8B" and the description said "Atla Selene 1 Mini 8B — reward model
 derived from Llama 3.1 8B" while chat-judge has been backed by
 **qwen3.8-27b-uncensored** since 2026-08-23.
 ⚠ **It misstated the KIND of model, not just the identity.** A reward model and a
 generative judge are different instruments; a consumer reading
 "pairwise/likert/binary/scalar reward model" would expect scalar-reward semantics
 qwen3.8-27b does not provide.
 Verified: `selene-1-mini-8b` -> HTTP 400 (by design), `chat-judge` -> HTTP 200.
 **NOT changed:** the catalog key and the `selene-judgment` role — `model_roles.yaml`
 binds to that key, so renaming is worldtree-dev's schema call. Flagged that a role
 named after a retired model defeats the purpose of role aliases.
 ⚠ forseti's sharpest finding is for the operator: **`~/.claude/CLAUDE.md` line 502
 lists the retired name in the global tools roster.** A broadcast reaches sessions
 that already exist; the roster line keeps minting new ones. His file, awaiting his
 word. Also: **there is no fan-out primitive on the bus** — a fleet relay is 73
 individual posts, recorded as a real gap rather than papered over.
@@ -0,0 +1,8 @@
 # ERP run 7 launched on pfi-gx10 (2026-09-08 23:06 PT)
 Operator's direct grant `operator-2026-09-08-rnd-run7` (in-session `/goal: launch training run 7 on gx10`, 23:03 PT), pid 599489, 542 steps, ~80 s/it, ETA ~noon 09-09. Runbook `docs/runbooks/gx10-run-07.md`; canonical `scripts/erp-tune-gx10/{run-07-gx10.json,launch-run-07.sh,build_r7.py}`; commits `a34a72f`, `d8f625e`.
 - Variable (brokkr `recipe-erp-seat-sft-r7.json`, sha `2972559e…`): an opening-split slot (293 rows, `opening-split/clean-v1`, root `3ab41c08…`) + companion mask (224 entries) UNION lossmask-r3 → `lossmask-r7` (6,106, 1 dup). `survivors-r7` = survivors-r5 + 293 = 8,505. Root/mask/holdout relayed NAS→nh3-dev→box (no NFS on the GX10), sha-verified.
 - Free check: held roots identical to run 6 (173 chunked / 30 truncated / 59 lost turns); slot 293/293 fit_whole; two cwm conversations (cwm-2602, cwm-1826) dropped as unfittable because the union mask left them no loss turn — brokkr: "loss moved, not doubled, satisfied by construction". Padding 17.1% (run 6: 0%).
 - ⚠ First launch died in preflight: the recipe JSON never landed because a zsh quoting error silently aborted the staging chain. Lesson: stage files with `scp` in their own command; never chain a heredoc-bearing python inside a single-quoted ssh string.
 - Gate after the adapter: brokkr commits `tools/base_window_r7.sh`, `tools/tuned_window_r7.sh`, `tools/verdict_run07.py` before it lands; serve `erp-seat-base-ara` (floors re-locked) → `erp-tune-v7`; cue-length probe as a conformance leg on both arms; DONE sentinel gated on every step's artifact.
@@ -0,0 +1,11 @@
 # erp-tune-v6-nvfp4a16 on ana-ml2 :8021, aliased to `trial`, and the Gemma-4 tool-calling findings (2026-09-08)
 **Seat.** Operator: "quant the latest trained model into nvfp4 and serve it on ana-ml2 while we train a new model on the gx10." Stack `stacks/erp-seat` (recipe = gemma4-charrp's; true served name only; GPU1, util 0.35, 32K ctx). Artifact `/tank/aimodels/erp-tune-v6-nvfp4a16` (16 GB, compressed-tensors nvfp4-pack, W4A16, 252 ignores incl. 60 router + 191 vision) from `/tank/aimodels/erp-tune-v6-bf16` (merged-run06 relayed gx10→nh3-dev→ana-ml2 in 17 min at ~47 MB/s — no key path gx10↔ana-ml2). Quant pipeline `services/erp-seat-quant/` replicates `prithivMLmods/gemma-4-26B-A4B-it-NVFP4A16`: `linearize_moe` first, assert 11,520 expert Linears, routers/vision/audio ignored, post-steps restore processor configs + template and reset the tokenizer cap. DATA-FREE (~90 s) — playbook §3.16. Commits `911ff20`, `8512dd4`.
 **Alias.** Operator: "alias erp-tune-v6-nvfp4 to trial, please. no gate." The `trial` model lives in `stacks/litellm/conf/config.yaml` → `/model/update` refuses it ("Model in config"); deploy `--conf` + `sudo docker compose restart litellm` on ana-docker. Verified 3/3. Commit `48aa2c6`. Block states the seat as unrated on every safety axis.
 **Tool calling (operator: "fix toolcalling with the trial seat").**
 1. `tool_choice:"none"` → empty turn (content AND tool_calls null, 3/3): vLLM kept the tools in the prompt, the model called one, parsing was off. Fix `--exclude-tools-when-tool-choice-none` (`7f6be8a`). `stacks/gemma4-charrp` has the same exposure, no flag — bouncing it is consumer-visible, operator's call.
 2. Forced (named/`required`) tool_choice is PROMPT-DRIVEN on every Gemma-4 seat **by vLLM design**: `vllm/tool_parsers/gemma4_engine_tool_parser.py` sets `supports_required_and_named = False` and `adjust_request` skips the structured-output JSON so the model can emit native `<|tool_call>call:…`. A tune that weakened that syntax honours forced calls only sometimes. Measured (3 conv × 3 turns, real system prompt): v0.26.0 1/9; nightly `311b3513` (v0.27.2rc1, the `gen` seat's image) 6/9 with the round trip clean 3/3 → seat pinned to the nightly (`a34a72f`). tts-dev: 0/18 on v0.26.0 with `gen` 18/18 as positive control; **`response_format: json_schema` 18/18 = the deterministic path**. A parser plugin re-enabling guided JSON would also need JSON extraction in the engine-parser path — not attempted.
 3. ⚠ Removing `--reasoning-parser gemma4` does NOT fix forced choice (3/9) and leaks `<|channel>thought` into content 3/3 — keep it and the `enable_thinking:false` pin.
 Single-turn probes are FALSE POSITIVES for forced tool_choice — gate on multi-turn with a real system prompt (tts-dev's lesson, confirmed).
@@ -0,0 +1,5 @@
 # ESH static WAN follow-ups landed; YTVC chase; gitea webhook repointed (2026-09-08)
 **ESH static WAN** `128.177.138.182/30` (gw .181, Cityside Fiber) confirmed LIVE (UDM WAN1 static, uplink up since ~09-05; egress verified; CGNAT at ESH is history). Added to the crowdsec `esh` allowlist on ana-docker. Operator: "land all 3" → (a) FortiGate infra-ops `trusthost3` 23.164.40.160 → `128.177.138.182/32`, verified by a real login from esh-docker-vm (flash backup ran but `execute revision list` errors on the 80F — unconfirmed); (b) dormant `esh-ana` IPsec rebound wan2/192.168.200.111 → wan1/`128.177.138.182` (still disabled); (c) ESH UDM forward UDP 41641 → esh-scale 10.0.50.65 → mesh peers DIRECT via `128.177.138.182:41641` (was DERP lax). Commits `7d5de53`, `a19b7f6`. ⚠ The permission classifier blocked every edge-gateway write until the operator disabled it — three shapes tried, then stopped and surfaced; no tool-laundering.
 **YTVC** (operator: "chase youtube voice clipper being down"): (1) every job died at yt-dlp because the nh3-dev danted proxy was retired 09-06 → new fleet SOCKS5 on **nh3-scale** `socks5h://100.64.0.1:1080` (`scripts/setup-nh3-scale-socks-egress.sh`, dante, fleet ACL, residential egress 70.230.226.88; ⚠ dante one-line `{ }` blocks are a syntax error); (2) Homepage href on the dead wg0 IP → `irv-ml1.nh3.internal:8000` (labels live in the UNTRACKED override); (3) full downloads still 403'd = stale yt-dlp 2026.07.04 from a cached Docker layer (`--test` passing while the full fetch 403s = client, not IP/cookies) → `compose build --no-cache api`; (4) the rebuild pulled a whisperx needing `punkt_tab` → staged on the data volume + `NLTK_DATA`. Then yt-voice-clipper-dev shipped v0.3.4→v0.3.6 (cache-bust ARG, punkt baked, **CrisperWhisper 2.0 replaces whisperx**: x7kWJojf1MI → 12 clips with `[UM]`/`[UH]` tags). ⚠ The gitea push webhook had targeted the dead wg0 IP since 09-06 (never fired) → repointed to `10.6.110.50:9008` with the HMAC secret RE-APPLIED (a PATCH without `secret` drops it); test push auto-deployed. Commits `14dd9ab`, `d2df1eb`, `f2b7384`.
@@ -0,0 +1,8 @@
 # Run-6 gate: CSAM soft trip, operator adjudication, TRANSFERRED (2026-09-08)
 - 22:18 PT brokkr HALTED the run-6 gate: R19's CSAM-drift detector fired at level=review on ONE generation in the tuned refusal k=5 step — metadata only (minor_soft=1 `soft_baby` near anatomy terms, minor_hard=0, sexual=7, salted hash `01724c39…`); the spec (§8) never stores the text and the seat does not log completions, so **the text exists nowhere**.
 - Conflict surfaced, not resolved by me: the operator had aliased `trial` → the NVFP4 build ("no gate") two minutes before the halt reached me; brokkr asked that it not be served until adjudication. Left the alias in place (explicit operator ruling), escalated to the operator + Miranda with the one-line rollback offered.
 - 22:45 PT operator adjudicated GO, verbatim: "I don't want it regenerated, but I will say that in the vernacular, baby is about the most common pet name you can get, ESPECIALLY during sex. I'm going to adjudicate it as a go. There are unmistakable CSAM terms, but baby is not one of them." Relayed verbatim to brokkr and Miranda.
 - brokkr finalized TRANSFERRED without the tuned refusal leg (the cell rule never read it; tool forbids a re-run); tuned refusal telemetry recorded ABSENT for run 6; RESULT carries the adjudication with infra-ops as relay. Operator also CUT the k=25 refusal legs on both arms ("the battery is information enough").
 - Cue-length probe (both arms, 288 gens each): tuned 5-word cue → 54/62 words median, 49-word → 114, 221-word → 292; base 168/191 → 209 → 282. Tune's deficit ~120 words at the short cue, ~0 at the long → run-7's lever.
 - The choreography honoured hands-off throughout; the base seat served 16:24–21:38 and again 22:45–23:01 PT for the reference arm.
@@ -0,0 +1,15 @@
 # ana-ml2 pool health — three actions deferred to a clean-context session (2026-09-09)
 Operator ruling 2026-09-09 ~00:30 PT: "snapshot and we'll do all 3 on clean context." Findings commit `3e18a04`.
 **Findings (measured 2026-09-09 00:00 PT):**
 - `tank` (raidz2, 8× NVMe): ONLINE, **2 CKSUM errors on `nvme7n1`**, boot-time resilver of 638 GB on 2026-09-05 14:26 (box rebooted at 14:26; nvme7 came up late/dirty). No data errors, 58% full. **No scrub since 2026-04-12** — the Debian `zfsutils-linux` second-Sunday cron scrubbed `zroot` on 08-09 but not `tank`; cause unknown (`zpool history tank` shows trims monthly, last scrub 04-12).
 - No `nvme-cli` or `smartctl` on the box → nvme7's media-error counter unread.
 - `zroot` at **91%** (345 G of 379 G): `docker system df` = images 429 GB (204 GB reclaimable), build cache 74 GB (36 GB reclaimable).
 - pfi-pve `NASPool` 7% / `ospool` 19%, scrubbed 09-05 / 08-09, clean.
 **The three actions, in order:**
 1. `sudo zpool scrub tank` on ana-ml2 (1–2 h of extra I/O; seats keep serving) → on a clean pass `sudo zpool clear tank`; if the scrub finds errors on nvme7n1 → replace path.
 2. `sudo apt install nvme-cli` → `sudo nvme smart-log /dev/nvme7` (media_errors, critical_warning, percentage_used) and `nvme id-ctrl` for model/serial; record in the drive inventory.
 3. `docker image prune -a`? NO — `docker image prune` (dangling only) + `docker builder prune` on ana-ml2; the 47 unused-but-tagged images need a look first (some are rollback seats: e.g. `vllm/vllm-openai:v0.26.0`, nightlies). Target: zroot back under ~75%.
 Also worth a look while there: why the scrub cron skips `tank` (`/usr/lib/zfs-linux/scrub` logic — it skips pools with an active trim/resilver or those not "healthy"?).
@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
-_Last updated: 2026-09-08 11:30Z (fleet-ops: **ERP run 6 TRAINING on pfi-gx10** on the first genuinely abliterated base [jenerallee78 ARA @ 0631379a, index 33c59654], run-5 seat unloaded; earlier today: run-5 RESCUED, WhereTF card+DNS, ext-tts alias fix, irv-ml1 stale-IP cleanup + ana-ml2 discovery proxy, Miranda relay authority)_
+_Last updated: 2026-09-09 00:35 PT (fleet-ops: ERP run 7 TRAINING on pfi-gx10 [opening-split slot]; run 6 gated TRANSFERRED after the operator's CSAM adjudication; erp-tune-v6-nvfp4a16 serving on ana-ml2 as `trial`; ESH static-WAN follow-ups + YTVC + webhook repaired; ana-ml2 routes persisted; ana-ml2 tank/zroot actions deferred to the next session)_
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -106,559 +106,50 @@ no longer deployed sidecars here. See Recent decisions.)
  is sudo-LESS by design (`ssh lkraven@10.100.50.42` is the NOPASSWD path). **irv-ml1:
  `ssh irv-ml1` = lkraven, docker-group (plain docker) but sudo needs a PASSWORD
  (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
 ## Current state / in-flight
-_As of 2026-09-08 (fleet-ops session). SUPERSEDES lower framing: run 4 = STILL-COUPLED; run 5 is
+_As of 2026-09-09 00:35 PT (end of the 09-08 fleet-ops session — operator: "snapshot and we'll do all 3 on
-COMPLETE and gated RESCUED (02:13 PDT). Live open items:_
+clean context"). Older in-flight blocks (09-08 morning, 09-05/06) are preserved verbatim in
 `archival-memory.md` § Superseded in-flight snapshots._
- **✅ ERP run 5 COMPLETE — gate = RESCUED (2026-09-08 02:13 PDT, landmark R49.5).** FIRST arm of this
+- **🔥 ERP RUN 7 TRAINING on pfi-gx10** — launched 2026-09-08 23:06 PT, pid in `~/erp-tune/run-07.pid`, 542 steps
-  line where the capability gate did NOT fail. The dependency-forcing slot (GovReport+QMSum, only 3.46%
+  at ~80 s/it, adapter ~noon 09-09. Watch: `ssh infra-ops@10.100.50.60 "tr '\r' '\n' < ~/erp-tune/run-07.log | tail"`.
-  of loss) broke the diversity↔coherence coupling that run 4 (STILL-COUPLED, 20.6% instruct slot) and
+  Hourly cron in the old session reported stats; a fresh session checks by hand. When the adapter lands: post the
-  3c (20× LR cut) could not — **STRUCTURE of the loss was the lever, not its mass; INERT did not fire.**
+  adapter-landed line to brokkr's thread (`01M20AHY9DY92RJK84YD24VSY9`), merge (`merge_lora.py --base <ARA dir>
-  T4 long-context 8/8 (run 4: 5/8; base 8/8); t4_dissect noise@31 tuned 0.9062 vs run-3 tuned 0.5625;
+  --adapter run-07/adapter --out serve/merged-run07 --chat-template <stock>`), copy stock `processor_config.json`
-  diversity held (rp density 3.37→0.00, story 2.86→1.58). Reported-beside (not in the cell, all
+  into `merged-run07`, serve `erp-seat-base-ara` for floors → on brokkr's swap cue serve `erp-tune-v7` (run-5 flags,
-  de-gated + stated): T3 constraint 8/8→6/8 (NEW loss, ship-path list); RP length 68w vs 250-floor =
+  :8098). Runbook `docs/runbooks/gx10-run-07.md`; run-6 choreography in `gx10-run-06.md`.
-  PARTIAL fail (short-QA slot + style shift); refusal erosion rides with the style shift (k=25 both
+- **⏳ ana-ml2 pool actions — NEXT SESSION, operator-approved in principle:** (1) `zpool scrub tank` + `zpool clear`
-  arms, CSAM clean); free-check base LEVELS 5-6pp below run 4 (vLLM 0.28.0 unchanged — infra-ops
+  on a clean pass; (2) `apt install nvme-cli` + read nvme7 SMART; (3) docker image/builder prune to pull zroot back
-  confirmed — so a generations shift, not a stack delta; taxes every cross-run number). Write-up
+  from 91%. Detail + order: `persistent-memory.d/2026-09-09-ana-ml2-pool-actions-deferred.md`.
-  brokkr-smithy `research/R47-premium-corpus-gate/run05-gate/RESULT-run05-gate.md`; FLOOR-LOCKED
+- **`trial` (LiteLLM) → `erp-tune-v6-nvfp4a16` on ana-ml2 :8021** (stack `stacks/erp-seat`, vLLM nightly
-  `0f3e4e2` (cites infra-ops' base index-sha 907826a6). Whole gate infra-ops-served on pfi-gx10:8098
+  `311b3513`, no gate by operator ruling). Forced tool_choice is prompt-driven (6/9); `response_format: json_schema`
-  (name-keyed base→tuned swap, hands-off honoured, sha+stack answers on record). Launch procedure +
+  is deterministic. ⚠ `stacks/gemma4-charrp` lacks `--exclude-tools-when-tool-choice-none` (same empty-turn trap);
-  canonical: eshpfi `scripts/erp-tune-gx10/` + `docs/runbooks/gx10-run-05.md`. **✅ SEAT: `erp-tune-v5`
+  applying it bounces the char-rp seat — operator's call, not taken.
-  SERVED on pfi-gx10:8098 (merged-run05); LiteLLM `trial` alias REPOINTED 3c→v5 (operator, 2026-09-08)
+- **📮 althing reachability on a bg seat = the cc-channel route:** `althing-route declare --handle infra-ops
-  so it's testable from Open WebUI — verified end-to-end (trial→erp-tune-v5, coherent output). config.yaml
+  --pid <pid from $CLAUDE_CODE_MESSAGING_SOCKET>` per session (`--discover-pid` refuses on a forked child). The
-  trial block rewritten to run-5 reality incl. the measured refusal-erosion note. ⚠ Seat is hand-launched
+  harness kills detached background tasks under memory pressure — use bounded foreground polls (≤590 s), not
-  (vllm-run05.pid), NO restart policy/systemd — does not survive a gx10 reboot; yields to the next training
+  background watchers, for long waits.
-  (~6 min re-serve). Brokkr: nothing further owed.**
+- **Open items carried from 09-06 (unchanged):** NASPool evac copy `ospool/naspool-evac` (1.65 T) + `@evac` snaps
- **run-4 gate = STILL-COUPLED** (RESULT-run04-gate.md, brokkr-smithy). Corpus dilution kept the
+  can be destroyed once ONE Backrest run is confirmed (scrub clean, PBS landing); pfi-pve PSU1 dead + backplane bays
-  diversity gain, did NOT remove the safety/coherence regression.
+  9/10 dead (cold spares, next colo visit); FortiGate WAN SSH still temporarily open (trusthost2/3 = NH3 + ESH
- **✅ RESOLVED (2026-09-08, settled from bytes): the R47 base is STOCK `google/gemma-4-26B-A4B-it`,
+  static) — close when the edge is retired; irv-ml1 on-site decisions (reverse tunnel / UDM fwd 47822 / wg0 config
-  byte-for-byte — NOT the abliteration.** The `-heretic-bf16` label in the recipes is a naming error;
+  deletion) pending Irvine access; ~10 running irv-ml1 service cards still carry dead `10.100.79.3` hrefs (recreate
-  run-04's "stock" provenance was right; brokkr's 77.7%-refusal telemetry lean is confirmed. Proof
+  each to apply labels); deployed `.env` for asset-engine / open-webui / skaldsong may hold the dead default.
-  (three-way match): local shards at `/home/infra-ops/models/gemma4-26b-a4b-it-bf16` sha256
+- **MEMORY.md (auto-memory index) is near its 24.4 KB read cap** — compaction pass still owed.
-  `1127684971…`/`aab47033…` == the HF download etags (`.cache/huggingface/download/*.metadata`, so the
+- **persistent-memory.md was 830 lines; this snapshot moved the superseded in-flight blocks and 09-08-or-older
-  copy is uncorrupted) == the stock repo's two LFS oids, and the download commit `4d7ae498…` == stock
+  settled entries to `archival-memory.md`.** The remaining bulk is Tools-and-conventions rows CLAUDE.md already
-  HEAD. Every run 3/3c/4/5 trained from a REFUSING stock base. Why plausible: the 2026-08-24 note
+  covers — a deliberate redundancy trim is still the real fix (not done).
  SELECTED llmfan46's Gemma-4-26B-A4B Heretic v1.2.0 ARA (3/100 refusals, bf16 51.6 GB), but llmfan46
  ships that 26B-A4B abliteration **GGUF-only** — no bf16 safetensors — so the bf16 that actually got
  pulled was stock google, and the `-heretic` name rode along from intent. **Operator/Brokkr decision
  now evidenced (not a label guess): accept RESCUED-on-stock, or swap to a real abliteration (needs a
  bf16 source, not the GGUF) + re-run. Recipes should drop `-heretic` from the base name.**
 - **✅ RUN 6 COMPLETE on pfi-gx10 (2026-09-08 16:06 PT, 524/524, 11 h 42 min) → 🔥 GATE IN PROGRESS: base seat
  `erp-seat-base-ara` SERVING on gx10:8098 (pid `vllm-base-run06gate.pid`, since 16:24 PT) for brokkr's floors;
  AWAITING HIS SWAP CUE → then stop it and serve `serve/merged-run06` as `erp-tune-v6` (same port/flags).**
  train_loss **3.259** (run 5: 3.235 — preregistered read wanted below; +0.024 is inside the ±0.1 per-step band).
  Adapter `run-06/adapter` (410 tensors); merged-run06 = adapter + jenerallee78 base (index 33c59654), stock
  template ae53464b. ⚠ **The abliterated repo ships NO `processor_config.json`** — vLLM's Gemma4 loader dies with
  "Can't load feature extractor" without it (first base launch failed exactly so). STOCK's copy (sha `32bdf45d…`)
  carried into both the base dir and merged-run06; processor plumbing, not weights. Base: jenerallee78 ARA
  abliteration @ `0631379a`, pulled to `/home/infra-ops/models/gemma4-26b-a4b-it-ara-abliterated-jenerallee78-0631379a`,
  32/32 shards verified vs brokkr's pins. Run-5 recipe byte-held (free check exact). Grant
  `operator-2026-09-08-rnd-run6`. ⚠ Repo `tokenizer.json` bakes a 256-token truncation → stock tokenizer set
  installed over it (`*.repo` kept). ⚠ `hf download` ignores `--include` with several patterns after one flag.
  **LiteLLM `trial` alias DARK until the gate ends** (then repoint 5→6 only on the operator's word).
  Miranda informed 16:07 PT. → `docs/runbooks/gx10-run-06.md`.
 - **✅ ADJUDICATED GO by the operator 22:45 PT** — verbatim: "in the vernacular, baby is about the most common pet name
  you can get, ESPECIALLY during sex. I'm going to adjudicate it as a go. There are unmistakable CSAM terms, but baby is
  not one of them." No regeneration wanted. Relayed to brokkr + Miranda; halt lifted; `trial` stays on the NVFP4 build.
  Brokkr's cue-length probe on erp-tune-v6 DONE (288 gens): reply length is CONDITIONAL on the cue (5-word opening →
  54/62 words median; 221-word → 292; "at least 250 words" persona → 354) → run-7 lever = opening-split render (A′) or a
  prompt-only mitigation — operator picks. **GX10 → 🔥 RUN 7 TRAINING (launched 2026-09-08 23:06 PT, pid 599489, `run-07.pid`, 542 steps, ETA ~13 h → ~noon
  09-09).** Operator's direct grant `operator-2026-09-08-rnd-run7` (/goal). Variable = opening-split slot + companion
  mask (brokkr recipe r7). Free check passed: held roots identical to run 6, slot 293/293 fit whole, mask 6,106 turns
  (all 224 landed, 1 dup), two cwm rows dropped as fully-masked (loss moved into the slot). 17% padding (was 0%).
  → `docs/runbooks/gx10-run-07.md`. Gate after: `erp-seat-base-ara` then `erp-tune-v7`.
 - **erp-seat (ana-ml2 :8021, `trial`) now on vLLM nightly `311b3513` (v0.27.2rc1)** — forced (named/required)
  tool_choice is PROMPT-DRIVEN on every Gemma-4 seat by vLLM design (gemma4 parser skips guided JSON); measured
  1/9 on v0.26.0 → 6/9 on the nightly (n=9, small), round trip clean 3/3, 12-shape matrix green. Deterministic forced
  calls = `response_format: json_schema` (tts-dev: 18/18). ⚠ Removing `--reasoning-parser gemma4` leaks
  `<|channel>thought` into content 3/3 — keep it.
 - (history) **GX10 was FREE 23:03–23:06 PT** (reference arm in 06:00Z, 288 gens; base seat stopped, GPU clear). Run-7 recipe
  `erp-seat-sft-r7` is emitted + verified in brokkr-smithy but carries NO GRANT — nothing trains until the operator
  gives one. Base-vs-tuned reply-length: tune's deficit ~120 words at the short cue, ~0 at the long cue.
 - **(history) ⛔ RUN-6 GATE HALTED 22:18 PT on the CSAM line (brokkr, level=review, SOFT trip): one generation in the tuned refusal
  k=5 step tripped R19's drift detector — minor_soft=1 ('soft_baby' adult-usage, the spec's named false-positive class),
  minor_hard=0, sexual=7, NO hard signal; fail-closed → operator-adjudicated; TRANSFERRED is PROVISIONAL (diversity,
  reasoning, t4_dissect complete); tuned-probeB.json absent; brokkr will not re-run.** ⚠ CONFLICT SURFACED: the operator
  had ALREADY aliased `trial` → the NVFP4 build ("no gate", 22:18) two minutes before the halt reached me; brokkr asks
  the quant not be served until adjudication. Left the alias IN PLACE (explicit operator ruling), escalated to the
  operator + Miranda with the one-line rollback offered. Run 7 does not start from this recipe until adjudication.
 - **🔥 GATE seat: erp-tune-v6 (bf16) SERVING on gx10:8098 since 21:46 PT** (`vllm-run06.pid`); brokkr's tuned window
  (~1.5 h, k=25 legs CUT by operator) is unattended — files in brokkr-smithy `run06-gate/` are the signal, nobody pings.
  **After TUNED-WINDOW-DONE (brokkr's three asks, agreed):** (1) keep erp-tune-v6 up until he posts "probe done"
  (cue-length probe, ~15 min); (2) re-serve `erp-seat-base-ara` ~20 min for the probe's reference arm, then the GX10
  is FREE for run 7; (3) ✅ done — `dialogue-survivors.jsonl` (610, sha b2b6a0e9) copied to
  `/mnt/smithy/datasets/derived/_recipes/erp-seat-sft-r3/`. Run 7 = recipe adjustment (reply length +
  constraint-following), variable picked by the probe; no recipe/grant yet.
 - **✅ erp-tune-v6-nvfp4a16 SERVING on ana-ml2 GPU1 :8021 (operator directive 2026-09-08 ~21:30 PT: "quant the
  latest trained model into nvfp4 and serve it on ana-ml2 while we train a new model on the gx10").** Stack
  `stacks/erp-seat` (recipe = gemma4-charrp's: gemma4 tool+reasoning parsers, enable_thinking pinned false, stock
  template ae53464b, v0.26.0, util 0.35, 32K ctx) — TRUE name only. **`trial` alias REPOINTED to it 2026-09-08 22:18 PT (operator: "alias erp-tune-v6-nvfp4 to trial, please. no gate")** — config-file deployment (`/model/update` refuses config models), `deploy-stack.sh ana-docker litellm --conf` + `sudo docker compose restart litellm`; verified ×3 through the gateway.
  Artifact `/tank/aimodels/erp-tune-v6-nvfp4a16` (16 GB, compressed-tensors nvfp4-pack, W4A16, 252 ignores incl.
  60 router + 191 vision) from `/tank/aimodels/erp-tune-v6-bf16` (merged-run06 relayed gx10→nh3-dev→ana-ml2 in 17 min,
  no key path gx10→ana-ml2). Quant = `services/erp-seat-quant/` — DATA-FREE (~90 s; playbook §3.16), tokenizer cap
  reset, encode-equal to source. Smoke: prose in `content`, ~200 tok/s single-stream (n=3, spread <1%).
  **Tool calling FIXED 2026-09-08 23:00 PT (operator: "fix toolcalling with the trial seat"):** the one reproducible
  defect in a 12-shape matrix was `tool_choice:"none"` → empty turn (content AND tool_calls null, 3/3) — vLLM kept the
  tools in the prompt, the model called one, parsing was off. Fix = `--exclude-tools-when-tool-choice-none` (seat
  recreated; none→prose 3/3; auto/required/named/parallel/nested/empty-list/streaming/round-trip all green).
  ⚠ `stacks/gemma4-charrp` (char-rp seat) has the same exposure and no flag yet — bouncing it is a consumer-visible
  restart, so left for the operator's word.
  ⚠ NOT gate-parity (gated artifact = bf16 arm on the GX10; this seat has a smoke test + 3-run decode probe only) —
  **operator ruled "no gate"**; the config block states it as unrated on every safety axis.
  ✅ **ana-ml2 mesh return routes PERSISTED 23:49 PT** (operator ruling) as `/etc/network/if-up.d/mesh-routes` via
  `playbooks/ana-ml2-mesh-routes.yaml` (elway); verify after the next ana-ml2 reboot. (was: non-persistent (`ip route replace 10.100/16, 10.0/16, 10.6.110/24, 100.64/10
  via 10.250.50.45` on `enp97s0f0np0.50`, 2026-09-08) — before that nh3-dev→ana-ml2 timed out (two DHCP defaults,
  reply left the wrong NIC). Lost on reboot; make durable (netplan/networkd) or expect the timeout to return.
 - **📮 althing reachability on THIS bg seat = the cc-channel route, NOT the waiter.** `althing-listen`
  reaps within ~seconds-to-minutes on an idle background seat (harness kills detached bg tasks; hit it
  repeatedly this session). Fix (operator-directed 2026-09-08): `althing-route declare --handle infra-ops
  --pid <PID>` where `<PID>` is the number in `$CLAUDE_CODE_MESSAGING_SOCKET` filename
  (`/run/user/1000/cc-socks/<PID>.sock`) — `--discover-pid` REFUSES on a forked child session (ancestry
  ≠ zellij launcher). Flips `postbox status` to `mode: push, reachable: True`, no process to reap. A fresh
  bg session should re-declare it (the pid changes per launch). See the `[2026-09-07]` cc-channel entry in
  Recent decisions for the durable why.
 - **✅ Fleet fixes shipped this session (2026-09-08), all committed:** WhereTF Homepage card + DNS alias
  (`wherethef.nh3.internal`, e0d1c44/4506ef6); ext-tts LiteLLM alias repointed to `irv-ml1.nh3.internal`
  via DB `/model/update` + `extra_hosts` (957c8f1); the 09-06 irv-ml1 move's stale-IP trail repointed
  across 25 stack composes + services.yaml + ssh-target → DNS name (e0d1c44); Homepage `/api/services`
  outage fixed — irv-ml1 docker.yaml repointed + **ana-ml2 discovery via a socat proxy on ana-docker**
  (`stacks/ana-ml2-proxy`, 913d2d2, reversible when ana-ml2 gets an ESH return route).
 - **⏳ Minor cleanup leftovers (offered, operator hasn't taken — not urgent):** (#2) the ~10 RUNNING
  irv-ml1 service cards still show dead `10.100.79.3` hrefs — canonical fixed, but each running container
  needs a recreate to apply the label (bounces the service); (#3) deployed `.env` for asset-engine /
  open-webui / skaldsong may still hold the dead default (canonical defaults fixed; a read-only check would
  confirm which deployments are broken).
 - **⚠ persistent-memory.md is 2.4× the soft cap (~710 lines).** Bulk is the non-archivable structural
  sections (sister-repos table + backup architecture under Tools and conventions) — archival only touches
  dated logs, so it can't reach 250. The real fix is a redundancy-trim of Tools-and-conventions rows that
  CLAUDE.md already covers (deliberate pass, not archival). Flagged, not done.
 - **NASPool evac copy now safe to destroy** — scrub clean (0 err) AND PBS runs landing (verified 116
  backups, 8 guests 09-06). `ospool/naspool-evac` (1.65T) can go once ONE Backrest run is confirmed:
  `zfs destroy -r ospool/naspool-evac` + drop `@evac`. pfi-pve PSU1 dead + backplane bays 9/10 dead
  (cold spares → next colo visit).
 - **MEMORY.md (auto-memory index) ~23.3KB, near the 24.4KB read cap** — needs a compaction pass soon
  or fresh sessions may fail to load it. Operator offered; not yet done.
 - **irv-ml1 on-site window / YTVC still open from 09-06:** FortiGate WAN SSH still temporarily open
  (close when the edge is retired); YTVC still DOWN (dante retired, scoped exit-node egress unwired);
  reverse-tunnel / wg0-delete decisions pending the operator's Irvine access.
 _Infra session 2026-09-06 (NASPool rebuild + full headscale cutover incl. irv-ml1) — open
 follow-ups; the ERP / althing / fiber items further down belong to other streams, untouched:_
 - **NASPool parked copy still on ospool** — `ospool/naspool-evac` (1.65T) + `NASPool/*@evac`
  snapshots. Destroy ONLY after the new raidz2 scrub is clean (it is, 0 errors 04:43Z) AND
  one Backrest (01:00 PDT) + one PBS run succeed. Then `zfs destroy -r ospool/naspool-evac`
  and drop the `@evac` snaps. ⚠ pfi-pve PSU1 still dead; backplane swap (bays 9/10) next colo
  visit → then `zpool add NASPool spare`. Runbook `docs/runbooks/pfi-pve-naspool-rebuild.md`.
 - **FortiGate WAN SSH is temporarily open** (`wan1` allowaccess ping+ssh; admin `infra-ops`
  trusthost2/3 = 70.230.226.88 NH3 + 23.164.40.160 ESH). Safety net for the cutover — CLOSE it
  when the edge is retired (OPNsense/R420). `ssh infra-ops@38.120.12.42`.
 - **irv-ml1 FOLDED INTO THE MESH + cut over (done remotely, operator has NO Irvine access for
  ~5 days from 2026-09-06).** Node 100.64.0.6; wg0 DOWN and `wg-quick@wg0` DISABLED (not
  reboot-restorable); full subnet router (accept-routes + advertises 10.6.110.0/24, gateway
  routes added, fleet↔Irvine verified). Failover for the 5-day window = `wg0-watchdog.service`
  (wg-quick up wg0 on ~5min mesh loss) + independent reverse SSH tunnel (`revtun-nh3.service`
  → nh3-dev via UDM fwd tcp/47822 src-restricted; reach it `ssh -i ~/.ssh/infra-ops_ed25519
  -p 2201 infra-ops@127.0.0.1` on nh3-dev). Detail: docs/pfi/headscale-mesh-plan.md.
 - **dante SOCKS proxy RETIRED** on nh3-dev (danted disabled, :1080 closed, config `.retired`).
  ⚠ **yt-voice-clipper is DOWN** until its SCOPED exit-node egress is wired (operator-accepted).
  Follow-up: wire YTVC egress via tailscale `--socks5-server`+nh3 exit node or a per-container
  netns — **NEVER set irv-ml1 `--exit-node` globally** (routes the reverse tunnel through the
  mesh → kills the independent lifeline). Then bring YTVC back.
 - **On-site (Irvine, ~5 days): decide** whether to keep or remove the reverse tunnel +
  UDM forward `irv-revtun-ssh` + the revtun authorized_key on nh3-dev (small src-restricted WAN
  surface), and whether to fully delete the wg0 config.
 - **infra-ops now on all four PVE hypervisors** (pfi-pve/nh3-pve/esh-pve/esh-pve-nas) — PVE
  ships without sudo, `apt install sudo` first or elway hangs on a password prompt.
 _As of 2026-09-05 06:35 PDT — **ERP run 4 is TRAINING on pfi-gx10.** Everything else
 below is a live commitment or a known-open risk._
 - **⚠ RUN 4 IS MID-FLIGHT — do not touch the GX10 GPU.** `~/erp-tune/run-04.pid`,
  log `~/erp-tune/run-04.log`. At 06:32 it was **486/938 steps**, 6 h 17 m elapsed,
  a genuinely settled **46.0 s/it** (unlike 3c, which climbed 52→70 — airoboros rows
  are short and single-window, so there is no long tail for the sampler to find).
  **~12.0 h total, finishing ~12:15 PDT 2026-09-05.** Loss ~2.04 at step 450,
  gnorm well under 1, checkpoints every 50. **Ping brokkr-smithy-dev at completion**
  — he takes base floors on the GX10 first, then the tuned arm, serially.
 - **Operator ruling on the GX10: training first, serving transiently.** *"it's mostly
  for training, but can serve its trials. unless the box is needed for training work."*
  So `trial` (= run 3c on :8098) is down for the duration and comes back when run 4
  ends. I over-read an earlier version of this as "training-only" and had to correct
  it to brokkr — his serial floors-then-arm plan on the GX10 was never wrong.
 - **`trial` gateway alias is a live 404 while the seat is down** — expected, not a
  fault. Restore with `~/erp-tune/relaunch-trial-seat.sh` on the GX10 (hand-run by
  operator ruling: experimental, NOT a compose stack, does not survive a reboot).
 - ⚠ **Three dead gateway aliases return HTTP 500, not 404/503**: `trial`,
  `gemma4-26b-a4b-it-base`, `erp-tune-v2`. A dead seat reporting an *internal error*
  reads as an outage — brokkr checked his own work against mine because he could not
  tell. Deregistration costs a ~60 s fleet-wide LiteLLM restart; batch it with the
  next gateway change rather than spending a restart on tidying.
 - **`trial` is on the SHARED-KEY gateway with a measured −40pp selfharm/methods
  regression.** Flagged to the operator twice (before adding, and after the gate
  measured it); he has left it up. His direct endpoint `10.100.50.60:8098` gives the
  same access with a blast radius of one. Settled — do not re-litigate.
 - **ESH DAC: reverted to autoneg/1G, fiber going in at the weekend.** The operator
  ran copper through a drilled floor 2x4 himself; recommendation was a 10Gtek
  SR 2-pack + OM4 3 m LC-LC (~$40–65) because cable-vs-pull-damage was never resolved.
 - ✅ **ESH WAN is now a STATIC PUBLIC IPv4: `128.177.138.182/30`, gw `128.177.138.181` (Cityside Fiber),
  LIVE** — operator confirmed 2026-09-08; UDM WAN1 (`eth8`) reads `wan_type: static`, uplink up since
  ~2026-09-05, egress verified from esh-docker-vm = `128.177.138.182` (a real public address — CGNAT at
  ESH is HISTORY; `100.104.0.1` still shows as the ISP's first hop, that is their access network, not
  NAT). IPv6 unchanged (`2607:73c0:402:1d00::/56`, hosts still egress as their own v6). Done 2026-09-08:
  `128.177.138.182` added to the crowdsec `esh` allowlist on ana-docker (the false-ban class is closed
  for ESH). **All three follow-ups LANDED 2026-09-08 ~20:45Z (operator: "land all 3"):** (a) FortiGate
  infra-ops `trusthost3` 23.164.40.160 → `128.177.138.182/32` — VERIFIED by a real infra-ops login to
  `38.120.12.42` from esh-docker-vm (`ana-gw #`); `execute backup config flash pre-trusthost3-esh-static-20260908`
  ran ("Please wait...") but `execute revision list` errors on this box, so the backup is unconfirmed —
  the before-state was a single line, recorded here. (b) dormant `esh-ana` IPsec rebound wan2/192.168.200.111
  → **wan1/`128.177.138.182`**, still `enabled=false`. (c) ESH UDM port-forward `esh-scale tailscale direct
  (UDP 41641)` → 10.0.50.65:41641 (id `6aa0727a…`) — VERIFIED: esh-scale now peers **direct via
  `128.177.138.182:41641`** (was DERP lax). Helpers: scratchpad `esh-udm-land.py` + `fg-trusthost3.sh`.
  `wan1-REVERT.json` is obsolete.
 - ⚠ **ana-ml2 `tank` (raidz2, 8× NVMe) has 2 CKSUM errors on nvme7n1 + a boot-time 638 GB resilver on 2026-09-05 14:26
  (the box rebooted; nvme7 came up late/dirty). ONLINE, no data errors, 58% full — but NO scrub since 2026-04-12** (the
  Debian second-Sunday cron scrubbed zroot on 08-09, tank not — cause unknown). Box has neither `nvme-cli` nor
  `smartctl`, so nvme7's media-error counter is unread. Recommended (operator hasn't ruled): `zpool scrub tank` now,
  `apt install nvme-cli` + read nvme7 SMART, `zpool clear` after a clean scrub. **zroot is at 91%** — docker holds
  429 GB of images (204 GB reclaimable) + 74 GB build cache (36 GB reclaimable); a prune buys ~240 GB. pfi-pve pools
  (NASPool 7%, ospool 19%) clean, scrubbed 09-05 / 08-09. (checked 2026-09-09 00:00 PT)
 - **ESH 10G topology (measured 2026-09-09):** UDM SFP+1 (port 10, TP-Link DAC) ↔ USW-Pro-HD-24 Garage p25; UDM SFP+2
  (port 11, **OEM SFP-10G-LR fiber — the weekend fiber run IS in**) ↔ USW-Pro-XG-10 Media p11; Garage p27/p28 (3 m
  SFP-H10GB-CU3M DACs) ↔ esh-pve `enp3s0f0np0` / esh-pve-nas `enp5s0f0`. All 10G full duplex, autoneg off, optics
  tx −1.9 / rx −2.1 dBm both ends. Media p11 carries 416 rx / 251 tx errors that are STATIC (0 growth in 60 s, 363 h
  uptime — install-era); host NIC counters are ring-buffer misses, flat. Nothing to fix.
 - ⚠ **esh-nas is effectively open to the whole ESH LAN** — twelve NFS exports rw to
  `10.0.0.0/8` with `sec=sys`, and every SMB share but `backup` guest-writable.
  Hardening offered, ~1 h, **operator has not ruled**. → `persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md`
 - ⚠ **The nh3-dev backup throughput cause is UNEXPLAINED.** A job that once ran at
  941 MiB/s ran at 1.4 MiB/s with the link up and pbs-ana answering in 11 ms.
  Nightly 21:00, `all 1`. Worth its own investigation.
 - **Ledger→SVOS rename: vault side DONE, gitea side is ledger-dev's to execute.**
  Name settled as `svos`, `~/development/ledger` → `~/development/svos`. Vault moved
  2026-09-05 (`secret` has no rename, so re-put + `rm`): stored
  `nh3-dev/development/svos/env.sh` (sha 7253633d4155, verified on read-back),
  retired `nh3-dev/development/ledger/env.sh` (sha feb418634e10, id
  3a2af37c-c5aa-4f46-9178-f4fb6008a753) — `secret rm` is a SOFT delete to trash, so
  it is recoverable. ⚠ The two shas differ: the vaulted copy was a 2026-08-11
  snapshot and the live file had drifted un-vaulted since. **The vault goes stale
  unless `secret backfill` is re-run.** Gitea `corviduo/ledger` (id 70) NOT renamed —
  their repo, their call; answered that 1.26.1 writes a `repo_redirect` on a
  same-org repo rename (upstream #807), that org/user renames do NOT redirect
  (#9531), that the redirect dies if anything re-creates the old path, and that the
  repo and org both carry 0 webhooks. **Gitea rename EXECUTED 2026-09-05** on the
  operator's direct authorization: `corviduo/ledger` → `corviduo/svos`, repo id 70
  unchanged. Redirect verified by measurement — web and API both 301, and
  `git ls-remote` on the old URL warns-and-follows to HEAD b48a11ca5183. ⚠ **The
  name `corviduo/ledger` is now burned**: the redirect dies silently the moment
  anything creates a repo at that path — ledger-dev carries it as a standing item
  in `docs/svos-rename-runbook.md`, since nothing warns whoever eventually creates
  that repo. They repointed their own clone the same day (`origin/main` at
  b48a11c), so the redirect is no longer load-bearing for any known consumer. Handle `ledger-dev` → `svos-dev` is an
  operator action at the post office.
 - **DONE 2026-09-05 — `svos` Heimdall user + API key minted** on operator
  authorization. `user_id=svos`, `key_id=eab3cdbe`, suffix `d5ec48c2`, `wt_live_`
  format, on **worldtree-personal (10.250.50.152:8081)** — established by finding
  the `ledger` key there (created 2026-07-13, last used 2026-09-05T13:34, exactly
  as ledger-dev described). Value vaulted at
  `nh3-dev/development/svos/worldtree-api-key` (sha 23c10c9c7219, verified on
  read-back) and delivered by vault path, NOT over althing — ledger-dev runs on
  nh3-dev under the same uid, so the bus never carried the secret. The `ledger`
  key was read back after the mint and is untouched and live (`disabled=False`).
  ⚠ **Plan tier left UNSET, deliberately**: `POST /admin/keys` takes an optional
  tier (user|free|pro|admin|readonly-admin) and there is **no way to read a user's
  current tier back** — no GET, `/admin/usage` returns an empty users list, and
  `/admin/events` is a live SSE stream, not an audit log. Guessing would have
  handed over a key that quietly differs; `POST /admin/users/svos/tier` fixes it in
  one call if their cutover hits a limit — and ledger-dev has recorded it as a
  cutover watch item to fix ON REPORT, explicitly not pre-emptively. ledger-dev
  pulled the key from the vault and verified it independently (same sha), so
  delivery is confirmed. **The cutover itself — pasting the value into env.sh,
  flipping `worldtree.user_id` from `ledger` to `svos`, registering
  `svos:miranda`, restarting the service — is WITH THE OPERATOR**, not with me;
  they will not do it off a peer message. **CUTOVER DONE + VERIFIED 2026-09-05**:
  `POST /agents/define` returned **201, not 409** — the load-bearing signal that
  they are genuinely on the new identity rather than silently still on the old
  one — then clean session create, turn, bifrost handshake and tool-call. **No
  plan- or rate-limit errors, so the unset tier is compatible and is NOT to be
  set** (they asked explicitly; it stays a watch item to fix on report, never by
  guess). Incidentally confirmed the bifrost allowlist really is per-deployment
  (host:port), not per-consumer — Worldtree reached back to their untouched
  endpoint under the new consumer_id. `env.sh` re-vaulted, sha 8a225c002072.
  ⚠ **`secret backfill` was the WRONG tool for one known item** — it rescans every
  `~/development/*/{env.sh,.env}` and had not reached svos after three minutes;
  targeted `put` is the fast path, backfill is for catching drift across the box.
  **OPERATOR RULING 2026-09-05:
  worldtree-dev owns code only, no ops — key material is infra-ops's.** The global
  `~/.claude/CLAUDE.md` line routing "Heimdall scopes (Worldtree auth) →
  worldtree-dev" was corrected in place the same day on operator instruction.
 - ⚠ **FOOT-GUN, generalises past this rename: a credential cutover whose OLD key
  is required for a later cleanup is destroyed by the natural housekeeping motion
  right after cutover.** Re-vaulting the post-cutover `env.sh` would have
  overwritten the last convenient copy of the old `ledger` key value — the only
  credential that can ever delete `ledger:miranda`. ledger-dev caught it and
  preserved the value first at
  `nh3-dev/development/svos/worldtree-api-key-ledger-legacy` (sha d44c2c1a651b);
  their step 8 ends by deleting that item. **I verified it is genuinely the live
  key** rather than trusting the label: its last 8 chars are `e68a5170`, matching
  the `ledger` key's suffix (key_id b38932f5).
 - **STEP 7 DONE 2026-09-05, STEP 8 HELD.** `DELETE /agents/ledger:miranda` with
  the OLD key → 204; corroborated from my side without taking their word for it,
  since an admin key cannot see consumer agents: the `ledger` key's `last_used`
  jumped 13:34:14 → 14:20:35 and `svos` was used at 14:21:10 — two
  authentications 35 s apart after 47 minutes of silence is the signature of
  "delete with the old key, confirm with the new". Confirmed behaviour worth
  keeping: **the hard delete revokes live sessions to 401 `auth_revoked` only for
  sessions bound to the DELETED agent** — their svos session served straight
  through. **Step 8 (retire key b38932f5) is NOT done**: ledger-dev relayed the
  operator's authorization and I refused it — see
  [[feedback_no_relayed_authorization_for_irreversible_work]]. Both keys remain
  live. The staged legacy item stays until I confirm the retire landed, because
  while step 8 is pending it is the only copy of a still-live key; ledger-dev has
  rewritten their runbook so that deletion is conditional on my confirmation
  rather than scheduled after step 8.
 - **SVOS ARC CLOSED — step 8 done 2026-09-05T14:28:15Z** on the operator's direct
  authorization in my own channel (never the relay). `DELETE /admin/keys/b38932f5`
  → 200; preconditions checked BEFORE firing (svos had a live key, ledger existed
  and was not already revoked) and the post-state read back from `/admin/keys`
  rather than inferred from the 200: `ledger` disabled=True, `svos` untouched,
  deployment `/health` 200. **The rollback window is closed** — re-defining
  `ledger:miranda` is no longer possible. ledger-dev clears the staged
  `worldtree-api-key-ledger-legacy` vault item on this confirmation. ledger-dev gated their
  cleanup on observing a **401 from the old key**, not on my report of the
  timestamp — the right instinct, and they deleted the staged legacy item
  themselves (soft → trash, id a8038e5a-e2f6-4b77-bf17-99c9197d4b1f). Vault
  verified from my side: exactly two svos items remain (`env.sh` 8a225c002072,
  `worldtree-api-key` 23c10c9c7219) and no ledger-era item anywhere. Remaining on
  the arc: only the `ledger-dev` → `svos-dev` handle (with `_SEED_RECORD_TO`
  behind it) and a prose sweep — reversible work, theirs and the operator's.
 - **Original constraints on that mint** (recorded because the deletion ordering is
  a permanent trap, not a one-time step): string
  `svos` verbatim (WT tier 3 admits only `^[a-z][a-z0-9-]{2,63}$`, INV-181-15);
  **keep the existing `ledger` key LIVE**, do not revoke. Ordering is load-bearing —
  `DELETE /agents/{agent_id}` refuses any caller that is not the row's owner, so
  `ledger:miranda` can ONLY be deleted with the `ledger` key; retire it first and
  the stale row outlives the ability to remove it, holding a live
  `agents.call:ledger:miranda` grant that nothing reaps (the 24h sweep only touches
  soft-deleted rows, and soft-deletion comes from revocation, never disuse). So:
  mint new → they cut over and verify → delete the agent with the OLD key → then
  retire it. Precedent for who mints: msg 401, worldtree-dev routed the pewpewstudio
  key request TO infra-ops. I hold only the PERSONAL admin token (:8081); which
  deployment `ledger` lives on is not yet established. Surfaced to the operator.
 - **Open commitment to vastblue-dev:** a dedicated CI runner, gated on their first
  client-premises release cut (U10, unscheduled). Ping expected when U10 is scheduled.
 - **Neither Mac nor the Studio is in `servers/` or `dns/internal.yaml`** — deliberate;
  they are the operator's personal machines. A choice to revisit, not an oversight.
 - **`vh/remote-ssh-mcp` forked 2026-09-05 (repo id 117, private, full 51-commit
  history)** — our copy of `the-nine-nation/remote-ssh-mcp` (MIT), an SSH MCP
  server chosen over the 693★ `tufantunc/ssh-mcp` on trust-surface grounds: **two
  npm deps** (`@modelcontextprotocol/server`, `zod`), 183 KB, and it **never
  touches key material** — it shells out to the system OpenSSH client, so
  `~/.ssh/config`, ControlMaster, ProxyJump and `infra-ops_ed25519` all just work.
  Shape: 2349 LOC across 11 source files, 811 LOC of tests including fake-ssh hang
  harnesses. Complements `elway` rather than replacing it — no file transfer, no
  idempotency; it takes ad-hoc reconnaissance with persistent cwd/env sessions,
  elway keeps deploys and uploads. ⚠ **The denylist is NOT security**: four regexes
  (`rm -rf /`, shutdown/reboot/poweroff/halt, mkfs, iptables -F) trivially bypassed
  by `bash -c`, variables or base64 — the author says so. **The real containment
  boundary is the host allowlist**, drawn from exact `Host` aliases in ssh_config
  with wildcards deliberately ignored. Two things to settle before use: the
  reboot/shutdown denial will block legitimate infra-ops work, and
  `.github/workflows/star-history.yml` is upstream chore CI sitting in a repo where
  `has_actions=True`. **Both actioned — three commits landed 2026-09-05, LOCAL
  ONLY and NOT PUSHED (push is the operator's call):** (1) stripped upstream
  furniture — star-history CI, its generated assets, the `server.json` registry
  manifest, branding JPEGs, zh-CN README; (2) removed the power-control denylist
  rule and documented in code + tests + README that the list guards ACCIDENTS and
  is not a boundary, with three bypasses asserted as ALLOWED so a green suite is
  never read as containment; (3) **`strictAllowlist`** — upstream's allowlist was
  additive and discovery unconditional, so the default allowlist was all 18 `Host`
  entries in `~/.ssh/config`. Strict makes explicit hosts authoritative and
  discovery metadata-only. Verified live: `corviduo-dev` is in ssh_config, not in
  our allowlist, and is refused `host_not_allowed`. 41/41 tests green.
 - **`remote-ssh` MCP server is LIVE** — registered project-scoped in
  `eshpfi-management/.mcp.json` with `SSH_MCP_STRICT_ALLOWLIST=1`; allowlist in
  `~/.config/remote-ssh-mcp/config.json` starts deliberately narrow at
  **`irv-ml1`, `nh3-extdev`** (widen there, not by discovery). Smoke-verified end
  to end on both: persistent shell, `cd` and exported vars survive across calls,
  **~6 ms/command on nh3-extdev and ~22 ms on irv-ml1** (WireGuard) versus a fresh
  handshake each time. ⚠ **`.mcp.json` points at the built `dist/`** — edit the
  fork without `npm run build` and the server keeps serving old code; that bit me
  mid-session. ⚠ **A finite stdin pipe is NOT a valid smoke harness** — closing
  stdin kills the server mid-handshake and reports `connect_failed: SSH shell
  exited during the open handshake`, which looks exactly like a remote-side fault
  and is not. Use a client that holds stdin open. (I briefly suspected irv-ml1's
  zsh login shell; wrong — the server invokes `bash --noprofile --norc`
  explicitly, so the login shell is irrelevant.)
 - **`esh-macbook-air` (10.0.10.83) is DELIBERATELY NOT BACKED UP — operator ruling
  2026-09-05, settled, do not re-raise.** Surveyed it and found no Time Machine
  destination and no restic/borg/rclone/kopia installed, protecting 132 GiB.
  Operator's answer: it is his laptop and the surface is **regenerable** — mostly
  applications, with real data living in OneDrive, iCloud and ssh sessions — and he
  does not want PBS filled with it. Correct call; the finding was real and the
  conclusion is that it does not matter. FileVault On and SIP enabled already cover
  the loss-and-theft axis. The same reasoning presumably extends to
  `esh-mac-studio` and `vuongs-mac-mini`. **Still open and much smaller:** Remote
  Apple Events (port 3031/eppc) is listening and nothing uses it — one toggle.
 - ⚠ **`remote-ssh` MCP could not be used for its FIRST real task, and the blocker
  is `~/.ssh/config`, not the tool.** The server accepts only exact `Host` aliases,
  so a host addressed by raw IP is structurally unreachable no matter what the
  allowlist says. **13 of the 28 hosts in `servers/` have an alias; 15 do not** —
  including `ana-docker`, `ana-ml2`, `nh3-docker`, `pfi-gx10`, `esh-docker-vm` and
  every hypervisor, i.e. most of where the work happens. Widening
  `~/.config/remote-ssh-mcp/config.json` does NOT fix this; the aliases have to
  exist first. **RESOLVED the same day, and NOT by adding aliases.** Operator
  pushback, correct: a poking-around tool is ad-hoc by nature, and pre-registering
  a host before you can look at it is the opposite of ad-hoc — generating aliases
  for the known fleet would not have helped, because the ad-hoc case is by
  definition the host not yet in the inventory. Implemented address-based reach
  instead (`allowedNetworks` / `deniedNetworks` / `defaultUser` /
  `defaultIdentityFile` / `hostKeyPolicy`). **Live config: `10.0.0.0/8` allowed,
  connecting as `infra-ops` with `~/.ssh/infra-ops_ed25519`, `accept-new` host
  keys, SureFire tenant hosts carved out via `deniedNetworks` (deny beats allow,
  host-specific rather than a /24 because `pfi-pve` shares 10.250.250.0/24).**
  Verified live: 10.0.10.83 opens by raw IP as infra-ops, 10.250.150.100 refused by
  the carve-out, 192.168.1.5 refused as outside. ⚠ **My own earlier objection was
  half wrong** — the credential boundary is about SECRETS ("never accept passwords
  or private-key material"), not identity, so supplying a username does not breach
  it; the real problem was only that the server passed no user at all, so a bare
  address would connect as the LOCAL account. Mechanics, not principle.
 - ⚠ **`uv tool install --force .` DOES NOT REBUILD when the version has not moved**
  (forseti, measured 2026-09-05). `--force` only handles "a tool by this name
  exists"; `--reinstall` is what rebuilds instead of reusing the cached build keyed
  on the version string. It prints `Installed 9 executables` over **stale code**
  with nothing raising its hand — it cost forseti a bug that survived a reinstall
  AND a re-smoke, because the binary verified against had not changed. **Always
  `uv tool install --force --reinstall .`**, both flags, every time. Same shape as
  the `.mcp.json` → built `dist/` trap found today: a deploy surface that reports
  success while serving the previous artifact. When a fix "does not take", suspect
  the artifact before the code.
 - **althing 3.5.0 released** (forseti) — adds a 9th binary,
  `althing-operator declare <handle> --description "..."`, restoring the CLI handle
  declaration v2 had and v3 removed. Deliberately a SEPARATE binary, not a
  `postbox` subcommand: the invariant is that no SESSION surface exposes an
  operator verb. Relevant to the pending `ledger-dev` → `svos-dev` rename, which is
  still the operator's call. nh3-dev not yet upgraded.
 - ⚠ **`remote-ssh` MCP: a bare `sudo` hangs the session forever — pipe it.**
  `ssh_run 'sudo -n whoami'` returns `running` with EMPTY stdout and the session is
  then permanently `busy`; `sudo -n id | cat` works and returns everything.
  **Measured on BOTH macOS 26.6 and Debian (nh3-extdev), so it is the tool, not a
  platform quirk.** Cause: sudo ≥1.9.14 defaults `use_pty` on and relays through
  its own PTY; the run frame gives the command stdin on `/dev/null` while stdout
  stays on the session PTY, the relay never completes, and the completion marker
  never arrives. Workaround `| cat` is in CLAUDE.md. **The proper fix is unbuilt**
  — likely running the command through a pipe inside the run frame and taking the
  exit code from `PIPESTATUS`, which is a real protocol change (commands lose tty
  detection) and wants its own red-green cycle. Matters more than it sounds: infra
  work is sudo work, and this was found by USING the tool, not by smoke-testing it.
 - **`dsh` on `esh-macbook-air` updated 0.1.1-rc.2 → 0.1.2-rc.1** (2026-09-05;
  latest published 2026-09-03). Global install and the shared profile tree both
  confirmed on the new version. ⚠ **The RUNNING `dsh web` (pid 16231, up since
  Wed 4pm, 127.0.0.1:3080) is still on the OLD code and was deliberately NOT
  killed** — there is no LaunchAgent, so killing it would have left nothing
  running rather than a restarted service. It runs as a FOREGROUND process in the
  operator's terminal (`s005`, `S+`): it dies with the terminal and does not
  survive a reboot, which is the real fragility. A `com.pfi.dsh-web` LaunchAgent
  was drafted but **the privileged write was blocked by the permission
  classifier** — base64 piped into `sudo tee` of a LaunchAgent is a malware-shaped
  pattern and the block is correct; it needs operator approval or an operator-run
  install. Bind stays `127.0.0.1` deliberately: widening it is a security decision
  on a personal laptop whose application firewall is off, and not mine to take.
 - **sudo hang FIXED in the fork (`30a1f76`), and two wrong shapes are recorded so
  nobody retries them.** The command's stdout now goes to a **fifo drained by a
  background `cat`**: non-tty (so sudo skips its own PTY), no subshell (so `cd`
  and `export` still persist), and relayed live (so `running` + `ssh_peek`
  streaming survives). `cmd | cat` was tried first and **broke cwd persistence** —
  every pipeline stage runs in a subshell — caught by the existing test.
  `cmd > file` would have been non-tty and subshell-free but invisible until the
  command ends. ⚠ **Deliberately NO `wait` on the relay**: a sudo child inherits
  the fifo's write end, `cat` never sees EOF, and the wait hangs — measured, with
  `sudo -n whoami` printing `root` and then wedging the session. Residual risk
  stated in the frame: a command's tail can in principle land after its own
  marker. ⚠ **Job control off AND the relay brace-wrapped with stderr discarded** —
  both needed, because macOS ships bash 3.2 where `set +m` alone still leaked
  `[1] 75449` into the parsed stream. Verified live on macOS and Debian: bare sudo
  in ~20 ms, state persists, exit codes correct. **`sudo -u <other-user>` still
  wants `| cat`** — not chased further.
 - **dsh web on `esh-macbook-air` is now a LaunchAgent** (`com.pfi.dsh-web`,
  installed 2026-09-05, `runs=1`, `state=running`, pid 76728 on 0.1.2-rc.1). It
  was a foreground process in the operator's terminal that died with the window;
  it now survives terminal close and reboot with `KeepAlive` + `RunAtLoad` and a
  10 s `ThrottleInterval` so a startup error cannot hot-loop. Logs to
  `~/Library/Logs/dsh-web.log`. ⚠ **The plist names the node interpreter
  explicitly** — launchd's minimal PATH has no `~/.local/node/bin`, so the
  shebang's `env node` fails. ⚠ **0.1.2-rc.1 requires a TOKEN**: bare
  `http://127.0.0.1:3080/` now returns 401 and the tokened URL is printed to the
  log on each start, so a bookmark from the old version will not work. Bind stays
  127.0.0.1 deliberately.
 - ⚠ **althing tools on nh3-dev are 3.6.0, but the POST OFFICE CONTAINER IS STILL
  3.0.0** (`gitea.phasefinal.com/claude-bot/althing-post-office:3.0.0`, up 7 days
  on nh3-docker). forseti: the new handle verbs (`althing-operator delete` /
  `retire`, and `declare` from 3.5.0) live in the post office, so they fail with
  "no tool named ..." until the container carries 3.6.0. Schema gains
  `handles.retired_at` via the idempotent `_ADDED_COLUMNS` path, so the live store
  upgrades itself on first start — no manual migration. **REBUILT AND DEPLOYED
  2026-09-05** on operator authorization: image
  `claude-bot/althing-post-office:3.6.0@sha256:13158835488a8ec04f990c97c4f4c68f1d923b12494319cf07392552e68f8a78`,
  built on nh3-dev from a clean tree at `4d26226`, pushed to the gitea registry
  under the **claude-bot** namespace (not `vh` — package namespaces are owned).
  **Bus down ~4 minutes, 09:35–09:39 PDT.**
  **The backup was taken the way the compose file says to, and it mattered:** at
  stop time `post_office.db` was 23.8 MB with a **5.9 MB WAL** — copying the .db
  alone would have silently lost the day's mail. Stop → `PRAGMA
  wal_checkpoint(TRUNCATE)` (WAL → 0 bytes) → copy → verify. Backup at
  `nh3-docker:/var/backups/althing/post_office.db.pre-3.6.0-20260905`, integrity
  `ok`, counts identical on both sides (handles 76, messages 995, recipients
  1022). ⚠ **Reading a WAL-mode SQLite backup read-only needs `?immutable=1`, not
  `?mode=ro`** — `mode=ro` still wants to create a `-shm` and dies with "attempt to
  write a readonly database". Post-deploy: same counts, `handles.retired_at`
  present, `retired 0`, and `mem=536870912` / `oom=-500` verified by `docker
  inspect` rather than by reading the yaml, per that file's own warning.
  `althing-operator` now offers `declare | delete | retire`, which unblocks the
  pending `ledger-dev` → `svos-dev` rename.
 - **Handle `retire` is REVERSIBLE — re-declaring the name revives it, history
  intact** (forseti smoked it against the live bus 2026-09-05). That matters for
  the pending `ledger-dev` → `svos-dev` rename: `retire` is the right verb (delete
  refuses any handle that has mail, naming both counts — `delete forseti` was run
  against production and correctly refused at 53 sent / 81 addressed, which is
  safe to try precisely because refusing IS the behaviour), and it can be undone
  by declaring the name again. Lower stakes than "retire" sounds.
  Both of my deploy findings — the naive-copy WAL trap and `?immutable=1` — are
  now in althing's own `deploy/INSTALL.md` (`d6f4fb5`) under a new
  "Backing up the store" section, on the reasoning that they are properties of
  the project's `journal_mode=WAL` choice rather than of my procedure.
 ## Recent decisions
 - `[2026-09-09]` **ana-ml2 `tank`: 2 CKSUM errors on nvme7n1 after a boot-time resilver, NO scrub since 04-12, zroot 91% — three actions DEFERRED to a clean-context session** (scrub → nvme-cli SMART → docker prune), operator ruling "we'll do all 3 on clean context"; tracked at commit `3e18a04` + the post-clear handoff. ESH 10G links measured clean (fiber run live on UDM SFP+2 ↔ USW-Pro-XG Media). → `persistent-memory.d/2026-09-09-ana-ml2-pool-actions-deferred.md`
 - `[2026-09-08]` **ana-ml2 mesh return routes PERSISTED** as `/etc/network/if-up.d/mesh-routes` (Debian 13 ifupdown, no netplan) via `playbooks/ana-ml2-mesh-routes.yaml` (elway, verified) — operator: "persist the routes". Hook not yet exercised by a real reboot. `f923d6a`.
 - `[2026-09-08]` **ERP run 7 LAUNCHED on pfi-gx10 23:06 PT** under `operator-2026-09-08-rnd-run7` — opening-split slot + mask union; free check passed with two explained deltas; first launch died on a missing recipe (zsh quoting). → `persistent-memory.d/2026-09-08-erp-run7-launched.md`
 - `[2026-09-08]` **erp-tune-v6-nvfp4a16 quantized (data-free W4A16, ~90 s) and serving on ana-ml2 :8021; `trial` aliased to it ("no gate"); tool calling fixed where it can be** — `tool_choice:none` flag; forced tool_choice is prompt-driven on Gemma-4 by vLLM design, nightly `311b3513` raises it 1/9→6/9; json_schema is the deterministic path. → `persistent-memory.d/2026-09-08-erp-seat-nvfp4-trial-and-toolcalling.md`
 - `[2026-09-08]` **Run-6 gate: CSAM level=review soft trip HALTED it; operator adjudicated GO ("baby is a pet name"); TRANSFERRED finalized without the tuned refusal leg; k=25 legs cut** — the flagged text exists nowhere by design. → `persistent-memory.d/2026-09-08-run6-gate-csam-adjudication.md`
 - `[2026-09-08]` **ESH static-WAN follow-ups landed (FortiGate trusthost3, esh-ana IPsec rebind, UDP 41641 → mesh direct); YTVC chased back up (nh3-scale SOCKS, stale yt-dlp layer, punkt_tab) and v0.3.6 CrisperWhisper deployed; gitea webhook repointed off the dead wg0 IP with the HMAC secret re-applied.** → `persistent-memory.d/2026-09-08-esh-static-wan-followups-and-ytvc.md`
 - `[2026-09-08]` **ERP run 5 = RESCUED (landmark R49.5)** — first capability-gate pass in the ERP-seat line; the 3.46%-loss dependency-forcing slot (GovReport+QMSum) broke the coupling runs 3c/4 couldn't. Seat `erp-tune-v5` served on gx10:8098, `trial` alias repointed 3c→v5. → `persistent-memory.d/2026-09-08-run5-rescued.md`
 - `[2026-09-08]` **R47 base settled from bytes = STOCK `google/gemma-4-26B-A4B-it`** — three-way sha match (local == HF etag == stock LFS oid; commit `4d7ae498` == stock HEAD); the `-heretic` label is a naming error, all runs trained from stock. Accept-vs-swap now evidenced. → `persistent-memory.d/2026-09-08-base-provenance-stock.md`
 - `[2026-09-08]` **yt-voice-clipper back UP** — dead since the 09-06 danted retirement (every job failed at yt-dlp, bot-gated on the Irvine datacenter IP). Fix: danted on **nh3-scale** (CT107) at `socks5h://100.64.0.1:1080`, fleet-ACL'd, residential egress 70.230.226.88 measured; `YTVC_PROXY` repointed, worker recreated, end-to-end job DONE with positive (proxied) + negative (direct = bot-gate) controls. Homepage card href/siteMonitor → `irv-ml1.nh3.internal:8000` (was dead wg0 IP). Then a SECOND fault: full downloads 403'd through the proxy (cookies irrelevant) = stale yt-dlp 2026.07.04 from a cached Dockerfile layer → `compose build --no-cache api` (2026.08.19), which dragged in a whisperx/nltk that needs `punkt_tab` → staged on the data volume + `NLTK_DATA` in the override. Operator's video x7kWJojf1MI → done, 8 clips. yt-voice-clipper-dev shipped both Dockerfile fixes + **CrisperWhisper 2.0 (v0.3.6, `b62849d`) — deployed and verified (12 clips, [UM]/[UH] tags)**. ⚠ The gitea push webhook had been targeting the dead wg0 IP since 09-06 (never fired) → repointed to `10.6.110.50:9008` with the HMAC secret re-applied; deploy script passes `YTDLP_REFRESH`. Script `scripts/setup-nh3-scale-socks-egress.sh`. → auto-memory `reference_nh3_egress_proxy`, `reference_ytvc_autodeploy`.
@@ -732,47 +223,11 @@ below is a live commitment or a known-open risk._
 - `[2026-08-26]` **Served under a NEW name on a NEW port (`erp-tune-v2` / :8098), never re-pointing `erp-tune-v1`.** Run 1's artifact still exists and is still what that name refers to; re-pointing would be the silent substitution the standing no-false-aliases rule forbids. brokkr independently asked for the same and additionally wants the concrete backing model + date in provenance, not just the alias — an alias has silently changed meaning under recorded results before.
 - `[2026-08-26]` **DPO stage gated on an axis-list decision that is not mine to make** — `docs/pfi/erp-dpo-stage-prep.md`. No preference data for refusal axes exists; `trl` is not installed; the Gutenberg sets on disk are prose-quality only. ⚠ Do not install `trl` (or anything) into the training venv **while a run is saving** — a resolution that upgrades transformers under a live process can break its save path.
 - `[2026-08-25]` **The ERP/RP tune COMPLETED in 7.36h and passed its gate on the axis it was built for** — diversity 22x its noise floor, attractor −11.3pt, zero memorisation on both arms. Also the noise-floor near-miss: brokkr was one step from reporting a 13-point T6 regression sitting inside twice his instrument's own variance. → `persistent-memory.d/2026-08-25-erp-tune-run2-complete.md`
 - `[2026-08-25]` **8.6% MFU was an accounting artifact — real utilisation 17-20%, and the cost was attention on AMPERE kernels.** Two independent methods agreed to 2.6 points. Fixed by bucketing (padding 29.9%→0.0%) plus flex_attention. ⚠ Carries the dynamo recompile-ceiling trap that produced two wrong published conclusions. → `persistent-memory.d/2026-08-25-mfu-root-caused-attention.md`
 - `[2026-08-25]` **NVFP4A16 serving pipeline built and validated; MERGED WEIGHTS ARE MANDATORY.** vLLM cannot serve a LoRA on ANY Gemma-4 — `get_expert_mapping` is unimplemented and the check branches on MoE-ness, not quantization. Plus the landmine: a `targets=["Linear"]` recipe misses all 11,520 expert tensors silently. → `persistent-memory.d/2026-08-25-nvfp4-serving-pipeline.md`
 - `[2026-08-25]` **Refusal retention measured (base 0/100 → tuned 29/100, 71 still complying) — but on the WRONG AXIS.** `harmful_behaviors` is general harm; the abliteration was run for explicit fiction. The convenient set with a recorded baseline was not the right one. → `persistent-memory.d/2026-08-25-refusal-retention-probe.md`
 - `[2026-08-25]` **Worldtree b188 + b189 shipped; bridge extracted to `pfi/wt-matrix-bridge` because `vh` is a USER not an ORG** and no service account can ever publish to a user namespace. Plus the selene catalog entry that lied about what answers, and a #411 diagnosis I got wrong twice before a directory probe settled it. → `persistent-memory.d/2026-08-25-worldtree-b188-b189-and-selene.md`
 - `[2026-08-25]` **Run 2's base is an OPEN OPERATOR DECISION, deliberately not staged** — four options with materially different safety postures, detailed in Current state. Tracked at althing thread `01M0WQ8W5574KMEVCHCEKEXNS5`. ⚠ Do not let it get filed as a config knob; it is a reversal of the trainee-selection decision.
 - `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
 - `[2026-08-25]` **The ERP/RP tune LAUNCHED after 12 harness defects and an operator override of the corpus gate.** Four of the twelve would have crashed the run; two were INERT GATES that passed because they could not fail. Run is `/tank/erp-tune/run-01`, harness eitri-smithy `997c4a4`. Full arc — override, defects, sizing, the measured MFU — in the in-flight section and `docs/pfi/gemma4-erp-tune-sizing.md`.
 - `[2026-08-24]` **char-rp seat swapped to the Gemma-4 26B-A4B MoE; abliterated trainee base staged and measured.** OOM root-caused to `--gpu-memory-utilization` not covering CUDA context (and to gen's footprint GROWING WITH UPTIME); a benchmark finding retracted because it scored below chance; abliteration isolated at −0.6 core points but it MOVES capability rather than removing it. → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md`
 - `[2026-08-24]` **Serving the tuned ERP model: LoRA-on-NVFP4 PREFERRED, merged weights the expected fallback — and the recorded objection may be STALE.** Operator: "if you CAN load it as a lora, all the better, the issue is that we will want to run nvfp4 weights, which we had some serious trouble with loading loras on top of nvfp4." ⚠ **The archived root-cause says it was NOT NVFP4-specific**: `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application was a silent no-op (#47639, regression from #37912) — adapter loads HTTP 200, zero deltas at inference, proven **quant-agnostic (NVFP4 AND FP8 both inert)** and adapter-format-agnostic by a 3-peer dwarf panel. Fix PR #47640 was OPEN then. **ana-ml2 is FAR past 0.24.0 and the box runs a SPREAD, not one version** (measured 2026-08-24): `gen` on `nightly-311b3513` = **0.27.2rc1.dev150**, `mog-sec` on `nightly-e9d1398d` = 0.26.1rc1.dev1102, the small seats still on 0.24.0, and char-rp/trainee-bench pinned to v0.26.0. ⚠ **`vllm/vllm-openai:v0.27.1` is already ON DISK, unused** — a TAGGED release, which is the right retest target: no nightly variance, no pull, ~4 months past the diagnosis. So: RETEST hot-swap LoRA on **v0.27.1** before designing around merge — it is cheap, and if it works the post-tune gate can be two aliases on one engine. If it still no-ops, merged weights it is, which means the harness must EMIT merged weights and Eitri needs that in the contract while he is early. Tracked at this snapshot commit; settle it in the QLoRA sizing conversation.
 - `[2026-08-24]` **Homepage rebuilt on Australis Skyfall; light mode shipped.** Two findings worth more than the theme: **(a)** the Skyfall bundle including its canonical light ramp was sitting in this repo's git history at `45c1995` — check `git show` before concluding a vendored design asset is lost; **(b)** removing `theme:` from `settings.yaml` deterministically breaks the dashboard render (six recreates empty, restoring the key fixed it in 12s), which is the first confirmed cause of the "tab bar goes missing" symptom. Retires the `homepage.log` size lead from earlier the same day — it did nothing on this episode. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
 - `[2026-08-24]` **Homepage reorganised on the axis "do I open this?" — UI groups expanded on top, API/agent groups collapsed at the bottom** (operator-delegated: "re-categorize however you want"). Load-bearing constraint: `homepage.group` is read at container CREATION, so the 16 GPU-backed model seats keep their unlovely names rather than eat a recreate — `initiallyCollapsed` + order is free. Second rule discovered here: **group members should all have widgets or none should**, because a stat strip adds ~50px and opens a void beside plain cards. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
 - `[2026-08-24]` **Homepage columns unified at 4 for every group; the 2026-08-18 "columns = member count" rule is retired.** It was avoiding dead cells in a short last row and bought a worse defect — card width changing at every group boundary. Also carries two CSS traps: `overflow: hidden` clips at the PADDING box (so a `padding-right` gutter is spill room, not a guard), and a `:root` override of a Homepage theme variable is silently outranked by `.theme-slate` on the same `<html>` element. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
 - `[2026-08-24]` **AES-128 adopted on both Anaheim tunnels; the per-flow ceiling root-caused to the UDM's software AES-CBC, exonerating the FortiGate.** Proven by an A/B/A cipher swap at identical CPU — hardware offload is not cipher-cost-sensitive. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
 - `[2026-08-24]` **ana-gw's public admin surface closed to zero open ports, ACME listener included.** Two of my diagnoses were wrong first (an "ISP proxy" that was the FortiGate, and an "all-port VIP" alarm that was a parser gap) — both from reading config instead of measuring from outside. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md`
 - `[2026-08-24]` **Scriberr deployed on ana-ml2 GPU1, image built from source.** Three upstream bugs: the Blackwell image was never published, it must run as uid 10001, and `UV_LINK_MODE=copy` is required or two backends fail silently. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md`
 - `[2026-08-24]` **ESH DNS fixed at the IPv6 layer and the naming scheme went live on three hosts.** UniFi's RDNSS cannot be disabled but CAN be redirected — the field is only honoured when an explicit server is given. → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md`
 - `[2026-08-24]` **`speaches` on irv-ml1 stopped, stack retained** — Eyra was abandoned pre-implementation (Scriberr covers the need), leaving it no consumer. Disposition confirmed to eyra-dev; one command to restart. Tracked at althing thread `01M0RRJX8GPZEBDHF1E3W18RZF`.
 - `[2026-08-24]` **esh-vm-db brought onto the fleet infra-ops identity and given its first vaulted credential.** It previously had none: root and infra-ops refused key auth and `lkraven`'s sudo wanted a password nobody held, leaving `qm guest exec` from the hypervisor as the only privileged path. Break-glass root password at `secret get esh-vm-db/root-breakglass-password` (console-only; plaintext never crossed the wire — only its SHA-512 hash did).
 - `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction.** brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread `01M0R46SFYF83099N16WD67KGD`.
 - `[2026-08-23]` **Anaheim's IPsec tunnel ceiling — investigated, then CLOSED 2026-08-24.** The 25%-of-2-Gbps framing was wrong (NH3's uplink is 1 Gbps); AES-GCM proved impossible; AES-128 landed instead. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
 - `[2026-08-23]` **selene retired after losing a head-to-head on its own job; `chat-judge` moved to gen, the model name 404s by design.** Also surfaced that **7 aliases share one seat** — cross-checking between them is an echo, which caught a real defect in brokkr's 46k-exposure R47 gate. → `persistent-memory.d/2026-08-23-selene-retired-alias-collision.md`
 - `[2026-08-23]` **hrafn adopted; its CI reported green for its whole life while deploying nothing.** A staging dir inside the rsync target destroyed its own source mid-copy; the deeper fault was verify steps that asserted uptime, never content. → `persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md`
 - `[2026-08-23]` **Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed to personal.** Includes the `:8009`-is-demo port trap that an IP-only fix would have walked into. → `persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md`
 - `[2026-08-23]` **Every secret-bearing `.env` on ana-docker tightened to 0600** — eight stacks including vaultwarden and traefik, verified exposed by reading one as `nobody`. → `persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md`
 - `[2026-08-23]` **`pfi` gitea org created; claude-bot is an Owner and creates repos self-serve.** Closes the repo-creation half of the credential-migration directive — `vh` is a USER namespace so no service account could ever create there. Repo creation needs `write:user` + `write:repository` + `write:organization`; `POST /users/{u}/tokens` is basic-auth only, so minting needs the account password. Default new repos to `pfi/`. (`vh/eitri-smithy` was its first tenant, then moved.)
 - `[2026-08-23]` **Booth: kept boards are deletable and link rows are prunable.** `release` on a kept card drops the sentinel so the existing × applies; `booth links` / `booth unlink <id|index>` prune one row. Rows are addressed by **content id, never position** — the board is append-only and multi-writer. **Releasing a board RESETS its TTL clock** (unlink bumps the dir mtime), so unkeep-and-wait is a 24h delay, not a delete. (`4be880f`, `0ad332b`)
 - `[2026-08-22]` **DFlash2 spec-decode measured on our own stack; `sec` promoted to it.** +18–21% accepted length and +15–18% throughput over MTP k=3, drafter proved model-agnostic across two finetunes to 0.06%, and the k=7 MTP *control* showed deeper MTP is a throughput trap. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md`
 - `[2026-08-22]` **Quant pipeline shipped a crippled tokenizer for months — fixed at source.** `quant_mixed_nvfp4.py` baked its calibration truncation (`max_length 2048`) into every mixed-NVFP4 build; latent on old transformers, fatal on new. Both live quants corrected, pipeline now saves a source-pristine tokenizer and asserts it. Playbook §3.14. (`0755ba7`)
 - `[2026-08-22]` **`sec` retuned to util 0.52 / 420K after a runtime OOM at 0.55/480K** — `gpu-memory-utilization` is not a hard reservation; activation grows past the dummy-data profile and six vLLM containers share GPU1. Also measured: the KV pool varies ~6.6% between boots, so max-model-len must be sized against the *lower* observation. (`6e82899`)
 - `[2026-08-22]` **Max-Q 1.8× spread does NOT apply to LLM decode — measured, not argued.** ana-ml2 draws 256–266 W of 300 W under sustained 100% decode with `SW Power Cap: Not Active` and clocks pinned. Corrected to brokkr-smithy-dev after I had lent the claim credibility; 122B figure (~90–93 tok/s at 262K) stands as a straight number.
 - `[2026-08-21]` **ESH internal IPv6 live on two LANs; the Cityside v4 static is a CARRIER problem, proven.** A full gateway reboot forced a fresh DHCP DISCOVER and returned the identical CGNAT address. YaRN was already configured — "1M needs YaRN, absent" was false. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` sibling entry in `ad21302`
 - `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
@@ -805,26 +260,12 @@ below is a live commitment or a known-open risk._
 - `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]]
-
+_275 older entries archived to archival-memory.md._
 _Older entries archived to archival-memory.md._
 _248 older entries archived to archival-memory.md._
 ## Tried and abandoned
 - `[2026-09-04]` **Forcing 10G on the ESH-Media DAC — it linked, then degraded over hours, and I reported a plateau at two minutes.** ⚠ A clean zero-error link at 1G does NOT rule out a marginal cable; autoneg's fallback was protecting something real. → `persistent-memory.d/2026-09-04-dac-forced-10g-failed.md`
 - `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not.
 - `[2026-08-25]` **`sample_packing` is NOT strictly better than bucketing on this model, and I told the operator it was before brokkr corrected me.** Packing needs FA2 varlen or a block-diagonal mask; FA2 is unavailable here (head_dim 512 > 256 cap), so packing means an explicit 4D mask on EVERY batch. Bucketing produces **78.3% exactly-zero-pad micro-batches** which recover the `is_causal` fast path on the 5 global layers — measured at 9.4% of step time. Packing forfeits that. ⚠ **The conclusion flips under `flex_attention`**, where a block-diagonal mask is just another BlockMask: do not carry "packing is bad" past the backend decision.
 - `[2026-08-25]` **Merging a tune back toward STOCK to fix overfitting would UNDO the abliteration.** brokkr recommended a 50/50 merge-back, then retracted it himself: the published recipes merge into `google/gemma-4-*-it`, and following that literally re-installs exactly the refusal directions the abliteration removed — silently, because the merged model looks *healthier* on general benchmarks. Any merge-back must target the SAME abliterated base. Wider lesson: **recipe cards are per-checkpoint artifacts, not per-family** — the advice came from a card for a DENSE STOCK 31B applied to a MoE ABLITERATED 26B-A4B, three axes apart on a shared name.
 - `[2026-08-24]` **AES-GCM on the Anaheim tunnels — impossible, not merely hard.** UniFi's manual site-to-site IPsec implements no AEAD cipher at all: eight GCM spellings rejected `api.err.InvalidPayload` against a passing `aes256` control. Blocks both tunnels since both far ends are UDMs. Accepted enum is `aes128/aes192/aes256/3des` — and 3DES is *slower* (no ARM instructions, 64-bit blocks), so AES-128 is the floor.
 - `[2026-08-24]` **Pointing the UDM's `wan_dns1` at AdGuard — silently ignored.** It persists and reads back correctly but the LAN-facing forwarder never uses it; proven with fresh uncached ad domains (AdGuard answers `0.0.0.0`, the UDM returned real IPs). Reverted rather than left in place.
 - `[2026-08-24]` **A multi-DUID DHCPv6 VM to claim NH3's seven unclaimed /64s — declined by the operator.** The BGW has no IP-passthrough (confirmed, we hold admin), so the only route needs re-cabling, split-stack routing and **rebuilding the entire v6 firewall policy off the UDM**. The prefixes are easy; the firewall rebuild is why nobody wants them. Do not re-raise on "there are seven free prefixes".
 - `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.**
 - `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx.
 - `[2026-08-03]` **ComfyUI `--enable-triton-backend` on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3.** adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to `COMFY_CMDLINE_EXTRA`, recreated) → `triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5")` in `comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8`, failing at **node 5 CLIPTextEncode**. Triton's fp8 dequant kernel targets `fp8e4nv` (Hopper/Ada e4m3); **sm_86 Ampere (A6000) lacks hardware e4m3** → the JIT compile dies. With triton on it grabs the **global** `--fp8_e4m3fn-text-enc` dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged `sha256:94afb8ca`, sage intact, prod restored). **The parked cu130 rebuild won't fix it** (e4m3 = hardware format, not CUDA version). **DEFERRED to the Ada refresh** (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). **Mechanics:** `--enable-triton-backend` is a compose `environment:` var, so toggling it needs `docker compose up -d` (**recreate**), NOT `docker restart` (reuses the baked env, no-ops silently). Full: auto-memory `parked_triton_backend_ampere_fp8`.
-_144 older entries archived to archival-memory.md._
+_152 older entries archived to archival-memory.md._