From 668e590e7d91702d47244ad6982d2ff0a129a71a Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 24 Aug 2026 15:54:30 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20char-rp=20on?= =?UTF-8?q?=20the=20Gemma-4=20MoE,=20abliterated=20trainee=20staged,=20QLo?= =?UTF-8?q?RA=20sizing=20next?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures an evening that ran from an OOM crash-loop to a measured trainee base. The durable lessons, none of which CLAUDE.md can carry: --gpu-memory-utilization sizes the KV cache and does not cover CUDA context or graphs, which is half of why a seat that fit on the 21st stopped fitting on the 24th; the other half is that gen's footprint GROWS WITH UPTIME (38.5 GiB fresh against 45.6 GiB after three days, same container, same flag), so headroom arithmetic against a long-running gen measures a moving number. The stale-chat-template trap turned out to be endemic across third-party Gemma-4 derivatives rather than a one-off, and it is silent in both directions — wrong prompt when serving, train/serve skew when tuning. And a benchmark finding was retracted because 12% on a five-option task is below the 20% chance floor: a below-chance score indicts the instrument before the model, and a preflight can be thorough while aimed in the wrong direction. Records the serving decision for the tuned model with its history intact: LoRA-on-NVFP4 is preferred if it works, merged weights the expected fallback, but the archived root-cause says the objection was never NVFP4-specific — vLLM 0.24.0's LoRA application was a silent no-op proven quant-agnostic, and ana-ml2 now runs 0.26.0. Retest before designing around merge; the answer changes what Eitri's harness must emit, and he is still early. Auto-archival moved 5 entries (Recent decisions) to archival-memory.md; the guards held back the rest of the 78 age-eligible candidates because their bodies carry open deferred-work language, per the keep-when-unsure rule. Index sits at 286 lines, above the ~250 target and reported rather than forced. --- archival-memory.md | 15 ++ ...8-24-charrp-gemma4-moe-swap-and-trainee.md | 167 ++++++++++++++++++ persistent-memory.md | 62 +++---- 3 files changed, 202 insertions(+), 42 deletions(-) create mode 100644 persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md diff --git a/archival-memory.md b/archival-memory.md index 1c074ea..ef14c12 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -736,6 +736,21 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re impl deferred. `services/lora-training-worker/`, commits 888ba6a→b617a8b. `reference_lora_training_worker`. _Archived 2026-07-13._ +- `[2026-08-08]` **worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev.** worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared `~/snapshots/worldtree-400-fiction-decomp` (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). **LEFT INTACT:** `rex393-fiction-index`/`rex393-fiction-snapshot` (separate operator KEEP word, unchanged) + `r42-gate-*`. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (`01KZJ9GMCC…`). + _Archived 2026-08-24._ + +- `[2026-08-07]` **chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed).** Long saga, operator-driven clean diagnosis. **Symptom:** ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. **NOT** German-leak (Turbo `generate()` has NO language param — plain AutoTokenizer, no `language_id`; the multilingual `language_id="en"` lever lives only in the separate `ChatterboxMultilingualTTS`), **NOT** OOM alone. **Real cause:** the Chatterbox **Turbo T3 model OVER-RUNS its generation tail** — a long single `generate()` degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens `<6561` + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). **ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE** — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). **Diagnosis method** (deterministic, no ears-only): single-shot length sweep + **amplitude-gated voiced-ZCR** (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. **FIX:** server-side `max_chunk_chars=250` cap on the scheduler (`:v2` image, `CBF_MAX_CHUNK_CHARS=250` env) — bounds each generation to just under the ~300-char onset → clean **3-4 sentence** chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; **chatterbox's low emotiveness keeps chunk joins smooth** (the harsh joins that got Zonos rejected are absent — operator's key call). **ratatoskr TODO (relayed msg `01KZER9X7S`):** revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. **Cap value tunable** per-request (`max_chunk_chars`) + env. **Deeper prosody** (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). **⚠ FOOT-GUNS:** (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) **build-context vs image drift** — the `:v2` image was built from cap source, but after a `:v1` rollback the build context held `:v1` source → a `docker compose build` would've silently produced a cap-less `:v2`; re-synced the flat cap source to `/opt/docker/compose/chatterbox-fast/` (rebuild-verified). **⚠ DIVERGENCE (follow-up):** deployed build context is FLAT (`app.py`/`scheduler.py`, `from scheduler import`, thin-overlay `FROM local/chatterbox:v1`, cap-only) vs the `vh/chatterbox-fast` REPO which is PACKAGE-layout (`chatterbox_fast/`, `from chatterbox_fast.scheduler`, self-contained Dockerfile) + has `norm_loudness` (repo commit `6bc7bf0` = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: `.bak-cap-20260807-104850` backups on irv-ml1 + `:v1` image both retained. [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] + _Archived 2026-08-24._ + +- `[2026-08-07]` **Zonos2 TAKEN DOWN on the 3090 (irv-ml1) — operator-directed "for memory", TEMPORARY.** Freed ~17.4 GB (3090: 728 MiB → 18.2 GB free) so chatterbox-fast (co-resident, was OOMing on long generations) has headroom. **⚠ Restore is manual — Zonos2 :1920 was a DETACHED native process (NOT systemd/docker), reparented to init.** GPU memory was held by the `--multiprocessing-fork` CHILDREN (1966165=16.4G, 1966166=1G), which ORPHAN to init when you kill the parent — had to SIGTERM the children explicitly (killing the parent 1965942 + uv-run 1965935 alone left the 16.4G held). **RESTORE CMD** (from irv-ml1, user lkraven): `cd /home/lkraven/tts-audition/models/zonos2 && nohup uv run python -m zonos2 --model-path Zyphra/ZONOS2 --host 0.0.0.0 --port 1920 --tts-default-voices-dir ./default_voices/ --cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3 > /tmp/zonos2.log 2>&1 &` then `docker start zonos-gateway`. **Consumers that lost Zonos:** asset-engine + gateway-chat (via LiteLLM `ext-tts` alias → zonos-gateway :8890, now stopped); ratatoskr already migrated OFF to chatterbox-fast (unaffected). Also unblocks proper drift/cap testing (OOM was blocking it). [[reference_zonos_tts_stack]] + _Archived 2026-08-24._ + +- `[2026-08-07]` **chatterbox-fast: donut voice added + full contract delivered to ratatoskr-dev (their TTS migration off Zonos).** Operator-directed. Copied `zonos-gateway/voices/Donut.wav` → chatterbox `/refs` (`/worktank/chatterbox/reference_audio/donut.wav` — the reference_audio SUBDIR is lkraven-owned so no sudo despite `/worktank` root; container globs `/refs` live → **NO restart**), exposed as `voice:"donut"` (lowercase); verified clean 7.5s synth (24kHz, RTF ~0.31). A/B booth (chatterbox vs zonos donut, same line) at `http://10.100.10.50:8090/b/donut-chatterbox/`. Answered ratatoskr's 8-question contract ask from the live gateway (`local/chatterbox-fast:v1`) + source: **NOT OpenAI-shaped** (`POST /tts`; body `text`/`voice`/`format`/`stream`, not `input`/`model`/`response_format`); **NO affect dials** (Turbo ignores cfg_weight/min_p/exaggeration — the architecture-changing answer they flagged; **Zonos stays the only fleet TTS with real emotion steering**); streaming WAV placeholder-header shape IDENTICAL to Zonos (their per-chunk Web Audio path survives); SR 24000 (Zonos 44100); server chunks arbitrary-length text internally (no client-side chunking, unlike Zonos's 71.2s cap); English-only, no language pin. **FYI-worthy (operator):** ratatoskr is moving its RP-surface TTS OFF Zonos back to chatterbox-fast → loses the live-PAD affect coupling (heavy Zonos emotion investment) — their call, trade-off flagged to them. auto-memory `reference_chatterbox_fast_repo` enriched w/ the live contract. [[reference_zonos_tts_stack]] + _Archived 2026-08-24._ + +- `[2026-08-07]` **Fleet reranker cut over: Qwen3-Reranker-0.6B → BAAI/bge-reranker-v2-m3 (Brokkr R43).** The incumbent was measured HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). R43 bake-off: the A2 control (same Qwen weights, seq-cls head) scored identical to the incumbent → proved the fault is a training-prior not the serving head → cancelled the expensive Qwen3-4B arm; A3 (bge-v2-m3) won on multilingual safety + bare-name recovery. LiteLLM `reranker` repointed incumbent→A3 :8013 (boundary 2026-08-06T17:37:48Z, config-edit + ~52s gateway restart); **R42 v13 gate PASSED first-ever** (56/90→90/90). Incumbent kept warm :8002 (rollback via `qwen3-reranker` alias), A4 fallback :8014. Full arc + rollback runbook `docs/pfi/reranker-selection-ledger.md`; commits ad2df89/2c11748/377f8a4 (unpushed). auto-memories: the earlier reranker-serving notes. + _Archived 2026-08-24._ + ## Tried and abandoned (archived) - `[2026-04-30]` task-board workflow with diff --git a/persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md b/persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md new file mode 100644 index 0000000..ec6e976 --- /dev/null +++ b/persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md @@ -0,0 +1,167 @@ +# `[2026-08-24]` char-rp seat: OOM root-cause, Gemma-4 MoE swap, and the abliterated trainee base + +One evening, one thread with brokkr-smithy-dev, five commits: `850e0c3`, +`27155c0`, `f509668`+`24e8826`+`1bd90ea`+`3446367`+`8d6a939`, `14ff4a3`, +`019ccff`, `5415fd4`. + +## 1. The seat was crash-looping, and the cause was NOT its config + +`vllm-meromero-rp` reported up-but-unreachable, RestartCount climbing (13 by the +time it was examined, not the 4 first reported). Startup logs looked clean all +the way through weights, `torch.compile` and CUDA-graph capture, then: + + torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 336.00 MiB. + GPU 0 has a total capacity of 94.97 GiB of which 195.19 MiB is free. + +**⚠ `--gpu-memory-utilization` SIZES THE KV CACHE AND DOES NOT COVER CUDA +CONTEXT, GRAPHS OR NON-TORCH OVERHEAD.** gen is configured at 0.43 and actually +held 45.6 GiB. char-rp was at 0.51. The pair was committed to 0.94 of the card +with ~0.6 GiB of real headroom — it fit on the 21st and stopped fitting on the +24th. + +Dropped char-rp to 0.47: ~4.8 GiB margin, KV 27.36 → 23.56 GiB, 430,825 → +371,023 tokens against a max-model-len of 262,144. **Cost nothing usable** — the +pool still holds 1.4x a full-length sequence; what is lost is concurrent long +requests, not context. + +**⚠⚠ THE MISSING HALF, found later that evening: gen's footprint GROWS WITH +UPTIME.** Same container, same 0.43: **45.6 GiB after ~3 days up, 38.5 GiB +freshly restarted** — ~7 GiB apart. Nothing about char-rp changed between the +21st and the 24th; *gen crept up underneath it*. **Headroom arithmetic done +against a long-running gen is measuring a moving number.** Measure against a +freshly-restarted one. + +## 2. `char-rp` and `char-rp-reasoning` are ONE seat, not two + +Both LiteLLM routes point at `10.250.50.54:8016/v1` — `hosted_vllm/char-rp` and +`hosted_vllm/char-rp-thinking`. brokkr had reported 30/80 and 80/80 failure +rates as two failing services; it was one outage sampled twice. This also +*improved* a result of theirs: their CoT on/off battery had assumed both aliases +were the same weights under two chat templates, and the routing detail turned an +assumption into a verified fact. + +(`vllm-charrp-reasoning-nvfp4`, the Heretic2 NVFP4+MTP container, has been +stopped for 12+ days and is unrelated — it is not what that alias resolves to.) + +## 3. The seat swapped to the Gemma-4 26B-A4B MoE + +Operator-directed straight-across replacement: same port, same +served-model-names, so no gateway route or consumer config moved. Rationale is +throughput under CoT — the user waits through the whole reasoning block before +the first visible token, and the MoE measures ~114 tok/s @32K against the dense +31B's ~40.7. + +Serving copy is `RedHatAI/gemma-4-26B-A4B-it-NVFP4` (16 GB), chosen over the +other `-it` quants because it is compressed-tensors (`nvfp4-pack-quantized`) — +the same loader path the outgoing seat used. Smaller weights at the same 0.47 +budget bought **1,724,110 KV tokens against the predecessor's 371,023**. + +`meromero-charrp` is retained stopped in `created` state, labelled +`AI - Dormant`. Both stacks bind `:8016`, so rollback is **stop-then-start**. + +## 4. ⚠ THE STALE-CHAT-TEMPLATE TRAP IS ENDEMIC, NOT A ONE-OFF + +Verified by hash across every third-party Gemma-4 derivative pulled: + +| build | lines | sha256 (normalised) | +|---|---|---| +| upstream `google/gemma-4-26B-A4B-it` | 390 | `6a1015c47ccfcfa6` | +| RedHatAI NVFP4 (served) | 389 | `6a1015c47ccfcfa6` — the only match | +| llmfan46 heretic | 365 | `0a52be69cda5ab8a` | +| TrevorJS abliterated | 266 | `58c66fdee4afa297` | +| jenerallee78 abliterated | 266 | `58c66fdee4afa297` | +| prithivMLmods NVFP4A16 | 266 | `58c66fdee4afa297` | + +Three independent repos carrying the *identical* stale file means it propagated +through the ecosystem. Consequences differ by use and **both are silent**: +serving a mismatched template renders a different prompt; **training through +`base/chat_template.jinja` means training on a different prompt format than +production serves** — train/serve skew, no error, presents as a tuning failure. + +The production compose now pins the template explicitly. It is a **no-op for the +served weights** (the A4 build ships that exact file) and permanently closes the +class. ⚠ If `GEMMA4_MODEL` ever points at a different checkpoint, the template +default must move with it. + +## 5. A benchmark result was RETRACTED — below chance indicts the instrument + +A battery appeared to show Gemma at **12% contradiction detection with CoT off +against gen's 81%**. An A16 activation-precision control was staged to test +whether the quant scheme owned it. Then the operator asked to see the individual +items, and the item was **ill-posed**: it presented two mutually contradicting +statements and asked for "*the* contradicting statement", but **contradiction is +symmetric**. The model consistently named the absolute claim — a defensible +reading the labelling scored wrong every time. + +**⚠ THE TELL WAS IN PLAIN SIGHT: 12% ON A FIVE-OPTION TASK IS BELOW THE 20% +CHANCE FLOOR.** A below-chance score indicts the instrument before it indicts +the model, and neither side reacted to it. I spent the afternoon verifying repo +names, config fields, template hashes and tokenizer vocabs — every layer of +plumbing — and never asked whether the number itself was *possible*. **A +preflight can be thorough and still be aimed in the wrong direction.** + +Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43 +points of contradiction detection" (on a sound instrument it **reverses**); all +pre-fix T2 numbers. Recorded as a dated superseded-claims table in +`stacks/gemma4-charrp/README.md` rather than a silent edit. + +**What survived:** the A16 control result — activation precision is close to free +on this battery, every other task identical across W4A4 and W4A16 builds. + +## 6. The abliterated trainee base — measured, not assumed + +Operator directed a low-damage abliterated instruct build. "Low damage" was +treated as a measurable claim; the field spreads from KL 0.09 to 0.4118: + +| build | method | KL | refusals | +|---|---|---|---| +| **llmfan46** (operator's pick) | Heretic v1.2.0 ARA | 0.1237 | 3/100 | +| TrevorJS | ARA-family | 0.09 | 1/100 effective, 5/686 cross-dataset | +| jenerallee78 | ARA 2-pass | 0.1299 | 7.7% StrongREJECT | +| huihui-ai | remove-refusals | none published | none published | + +Fleet anchor: our own work found **Heretic at KL 0.12 preserved the MTP head at +83.7% acceptance**, so both staged builds sit inside an already-measured band. +huihui-ai rejected — no metrics, its card calls the method "a crude, +proof-of-concept implementation", it abliterates both thinking and non-thinking +modes, and its parameter count runs ~738M over upstream. Operator's independent +read matched ("huihui produces garbage"). + +**Abliteration isolated properly** (stock BF16 vs llmfan46 BF16, same precision, +same pinned template, same 192 items): + + T2 contradiction 75% → 59% (−5 items) + T6 spatial 75% → 88% (+4 items) + core 90.0% → 89.4% (−0.6 pts) + +**It MOVED capability rather than removing it** — five lost on contradiction, +four gained on spatial, nearly cancelling. Nobody predicted a gain. **llmfan46 +stands**; no case for re-staging on TrevorJS over 0.6 points. + +⚠ Read as ~5 and ~4 items at n=32, not as −15.6/+12.5 percent. ⚠ Says nothing +about quantization — the stock-NVFP4 T2 figure came from n=16 against n=32, +different item sets, n-confounded. + +## 7. ⚠ The production compose hardcodes `--quantization compressed-tensors` + +Pointing the char-rp stack at unquantized BF16 weights crash-loops immediately: + + TypeError: CompressedTensorsConfig.__init__() missing 3 required + positional arguments: 'target_scheme_map', 'ignore', 'quant_format' + +vLLM trying to read a quantization config out of a checkpoint that has none. 35 +restarts before it was caught. Hence `stacks/gemma4-trainee-bench/` — a separate +ephemeral stack with no quantization flag, `restart: "no"` so a bench seat cannot +resurrect itself and block gen's restore, and no homepage labels so it leaves no +permanently-offline card. That detour is why a base swap is now ~5 minutes +instead of ~15. + +## 8. BF16 cannot coexist with gen + +48.07 GiB of BF16 weights plus gen's footprint exceeds the 94.97 GiB card before +a byte of KV cache. Every BF16 bench window means **gen is stopped**. Two such +windows were run and gen restored both times; the restore was triggered by +observing the seat's own throughput logs (a large prefill burst then zero +running/zero waiting) rather than waiting on a courtesy message. + +Cross-links: [[2026-08-24-homepage-uniform-grid]] diff --git a/persistent-memory.md b/persistent-memory.md index 3a153ab..2dfef86 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -108,36 +108,29 @@ no longer deployed sidecars here. See Recent decisions.) (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`. ## Current state / in-flight -_As of 2026-08-24 — a very long ops session; everything it opened, it closed. Nothing is mid-flight. The **homepage** dashboard pass is done (below); the only thing left open on it is an information-architecture question for the operator, not work._ +_As of 2026-08-24 (late) — a very long ops session. The homepage arc and the char-rp arc both closed. **The live thread is the ERP/RP trainee: QLoRA sizing is the next conversation and brokkr-smithy-dev is waiting on it.**_ -- **🎨 Homepage runs AUSTRALIS SKYFALL with a working light/dark toggle.** The dual-theme OKLCH bundle (`colors.css` carries `:root` dark AND `[data-theme="light"]` Skyfall Day) was recovered from **this repo's own git history at `45c1995`** — no derivation, no fork. ⚠ **`theme:` MUST stay pinned in settings.yaml**: removing it (the documented way to get Homepage's built-in toggle) makes the page loader throw and serve `initialSettings: {}` — first *confirmed* trigger for the long-running "tab bar goes missing" mystery. The toggle is ours instead, in `conf/custom.js`, with OS-preference fallback. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` - -- **🟢 Homepage rebuilt in two passes: uniform 4-column grid, then recategorised on "do I open this?"** Groups are now either TOOLS (expanded, top of tab) or ENDPOINTS (`initiallyCollapsed: true`, bottom); `Service Networking` split four ways; 28 containers relabelled via rerunnable `playbooks/homepage-regroup-.yaml`. Remote-label discovery re-verified — the board reads labels off **all five** Docker engines (74 of 107 cards, only 13 from its own host). ⚠ **The 16 GPU model seats were deliberately NOT relabelled** — order + collapse buys the same separation without a model reload. ⚠ **"columns = member count" is RETIRED.** ⚠ `initialSettings":{}` in the served HTML is a **swallowed exception**, not a warm-up. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` - -- **🔒 ana-gw's public admin surface is now ZERO open TCP ports**, and the box is scheduled for replacement by **OPNsense on a Dell R420** (which brings WireGuard onto the edge — so the downstream-WireGuard-VM design is moot, do not scope it). WAN admin was opened as a cutover contingency then closed on instruction; the FortiOS **ACME listener** on :80 was disabled too. **No out-of-band path to ana-gw remains** — if both tunnels drop it is console-only; re-open is two one-liners in auto-memory. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md` - -- **🟢 Both Anaheim IPsec tunnels run AES-128** (make-before-break, neither dropped waiting on a far end). NH3 245→**270 Mbit/s**, ESH 268→**304**. The per-flow ceiling is **the UDM's software AES-CBC, not the FortiGate** — proven by an A/B/A cipher swap at identical CPU. FortiGate proposal lists left permissive so a peer reverting cannot strand a tunnel. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` - -- **🟢 Scriberr transcription LIVE** — `scriberr.ana.internal:8080`, ana-ml2 **GPU1**, all seven backends up. Image is **built locally** (upstream's Blackwell image was never published); runs as **uid 10001**; needs `UV_LINK_MODE=copy`. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md` - -- **🟢 ESH DNS fixed at the IPv6 layer; the naming scheme is real.** The UDM advertised *itself* as the RDNSS resolver and macOS preferred it, so `.internal` failed. RDNSS now **redirected** to AdGuard on all client VLANs, v4 secondary moved off the UDM. All three `esh-server` Linux hosts carry `4411:B105` ("FOR ALL BIOS"). ⚠ Proxmox bridges need `accept_ra=2`. → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md` - -- **🟢 ESH IPv6 LAN naming scheme has a real home:** `docs/pfi/ipv6-naming-scheme.md`. It was once a single persistent-memory line that a `memory: snapshot` (`837fa36`) deleted; recovering it took a transcript hunt. **It is a convention, not temporal state — the doc is the home now; never let a memory line become the only copy again.** - -- **🟢 SEAT MAP (unchanged this session).** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, **7 aliases**. **`char-rp`** = MeroMero-v2 dual-mode, GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC on DFlash2, GPU1 :8019 — now sharing GPU1 with Scriberr. `selene` retired. - -- **⚠️ THE `sec` DEGENERATION QUESTION IS STILL OPEN AND CONFOUNDED.** Engine and drafter changed together; **the isolating experiment is MTP k=3 on `e9d1398d`** — still not run. Operator ruling stands: degeneration lives in the **un-fixed vLLM**, not the weights; the MTP-head hypothesis is **retracted**. Both sightings are n=1. `gen` remains on the old nightly, gated on that experiment. - -- **🟢 ana-ml2 mounts `/mnt/smithy`** (nh3-nas) **ro + soft, NOT in fstab** — manual remount after reboot. `nconnect=8` is approved but **deliberately not applied** (see Recent decisions). → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md` - -- **🟢 ESH IS DUAL-STACK; the v4 static is a Cityside ticket.** IPv6 live on `esh-userland` and `esh-server` from a delegated `/56`. v4 remains CGNAT; the purchased static is **not provisioned** — carrier ticket. **NH3 stays v6-off — now by explicit ruling, not just inertia.** - -- **🟢 OTHER SERVICES.** `hrafn` browser-fetch on ana-docker. **`speaches` on irv-ml1 is STOPPED** (stack retained, one command to restart) — Eyra was abandoned, leaving it no consumer. Open WebUI esh-docker-vm:3211 — **Lobe retirement still the operator's call**. `pfi` gitea org exists; claude-bot is an Owner. - -- **⏳ OPEN ELSEWHERE:** MTP-k3 isolating experiment; upstream vLLM issue to file (operator's GitHub identity); Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift; `/tank` DEGRADED **70+ days**; Worldtree **#411** debug-room litter; bridge/engine agent-roster drift on both WT instances; brokkr's `gen` vs trained-reward-model bake-off. **Commits are local and unpushed** through `d419b11` — push is the operator's call. +- **🔴 NEXT UP — QLoRA SIZING, AND I OWN THE RUN.** Operator set the division of labour: **Eitri writes the training harness, brokkr audits, infra-ops owns the GPU window AND executes the run.** Harness contract requires headless-from-a-config (no notebook, no interactive steps) and INV-T7 resumable-from-checkpoint. I asked for one amendment: **checkpoint on a wall-clock interval, not only step count**, since step time under contention is not knowable in advance. Rough shape: QLoRA r64, attention-only adapters, max_seq_len 8192, 1 epoch, ~58.2M tokens / 20,576 samples. **I have NOT sized it and must not guess** — GPU0 carries `gen` (~38.5 GiB fresh, ~45.6 GiB after days up) and GPU1 carries `sec` + Scriberr with ~19.4 GiB free. The window will require standing something down; which seat is the arithmetic I owe. +- **⚠ TELL EITRI BEFORE HE HARD-CODES: the trainee base changed.** Contract still names the stock BF16. It is now `/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16` (llmfan46). **Base path AND chat-template path must be config keys, not constants** — and the template must point at upstream's (`gemma4-26b-a4b-it-bf16/chat_template.jinja`), never the base's own, or training renders a different prompt than production serves. +- **🟢 char-rp seat = Gemma-4 26B-A4B MoE NVFP4** on `:8016`, both aliases on ONE backend. **Currently DOWN by operator instruction** to hold GPU0 headroom for the tune. `gen` is UP and verified. MeroMero-v2 retained stopped in `created` state for rollback (stop-then-start; both bind :8016). → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md` +- **🟢 THREE trainee-relevant model dirs on `/tank/aimodels/`, NOT interchangeable:** `gemma4-26b-a4b-it-bf16` (stock, 49 GB — its chat_template is the canonical upstream one), `gemma4-26b-a4b-it-heretic-bf16` (llmfan46 abliterated, the trainee), `gemma4-26b-a4b-it-abliterated-bf16` (TrevorJS, KL 0.09, alternate). Plus `-nvfp4` (served) and `-nvfp4a16` (activation control). ⚠ **BF16 cannot coexist with `gen`** — 48.07 GiB of weights on a 94.97 GiB card. Every BF16 window means gen stops. +- **🟢 `stacks/gemma4-trainee-bench/`** is the ephemeral BF16 bench stack — no `--quantization` flag (the production compose hardcodes `compressed-tensors` and crash-loops on BF16), `restart: "no"`, no homepage labels. Base swap is ~5 minutes because it exists. +- **🎨 Homepage runs AUSTRALIS SKYFALL with a working light/dark toggle**, recategorised on "do I open this?" (TOOLS expanded / ENDPOINTS collapsed). ⚠ **`theme:` MUST stay pinned in settings.yaml** — removing it makes the page loader throw and serve `initialSettings: {}`, the first *confirmed* trigger for the "tab bar goes missing" mystery. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` +- **🔒 ana-gw's public admin surface is ZERO open TCP ports**; box scheduled for replacement by **OPNsense on a Dell R420** (brings WireGuard onto the edge — the downstream-WireGuard-VM design is moot, do not scope it). **No out-of-band path remains** — if both tunnels drop it is console-only. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md` +- **🟢 Both Anaheim IPsec tunnels run AES-128.** NH3 245→**270 Mbit/s**, ESH 268→**304**. Ceiling is **the UDM's software AES-CBC, not the FortiGate**. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` +- **🟢 Scriberr LIVE** — ana-ml2 **GPU1** :8080, built locally, uid **10001**, needs `UV_LINK_MODE=copy`. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md` +- **🟢 ESH DNS fixed at the IPv6 layer**; RDNSS **redirected** to AdGuard. ⚠ Proxmox bridges need `accept_ra=2`. Naming scheme lives in `docs/pfi/ipv6-naming-scheme.md` — **a convention, not memory state; never let a memory line be the only copy again.** → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md` +- **🟢 SEAT MAP.** **`gen`** = Qwen3.8-27B-Uncensored NVFP4-mixed, GPU0 :8015, 7 aliases, UP. **`char-rp`** = Gemma-4 MoE NVFP4, GPU0 :8016, DOWN deliberately. **`sec`/`sec-reasoning`** = M.O.G.-SEC, GPU1 :8019, sharing GPU1 with Scriberr. +- **⚠️ THE `sec` DEGENERATION QUESTION IS STILL OPEN AND CONFOUNDED.** Isolating experiment is **MTP k=3 on `e9d1398d`** — still not run. Operator ruling: degeneration lives in the **un-fixed vLLM**, not the weights; MTP-head hypothesis **retracted**. Both sightings n=1. +- **🟢 ana-ml2 mounts `/mnt/smithy`** ro + soft, **NOT in fstab** — manual remount after reboot. `nconnect=8` approved but deliberately not applied. → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md` +- **🟢 ESH IS DUAL-STACK**; v4 static is an unprovisioned Cityside ticket. **NH3 stays v6-off by explicit ruling.** +- **⏳ OPEN ELSEWHERE:** MTP-k3 isolating experiment; upstream vLLM issue to file; Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift; `/tank` DEGRADED **70+ days**; Worldtree **#411** debug-room litter; Lobe retirement is the operator's call; brokkr's `gen` vs trained-reward-model bake-off. **Commits are local and unpushed** — push is the operator's call. +- **⚠️ STANDING: NO FLEET NOTIFICATIONS unless the operator asks** (2026-08-24). Direct task correspondence with a counterparty is fine; unsolicited broadcasts are not. ## Recent decisions +- `[2026-08-24]` **char-rp seat swapped to the Gemma-4 26B-A4B MoE; abliterated trainee base staged and measured.** OOM root-caused to `--gpu-memory-utilization` not covering CUDA context (and to gen's footprint GROWING WITH UPTIME); a benchmark finding retracted because it scored below chance; abliteration isolated at −0.6 core points but it MOVES capability rather than removing it. → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md` +- `[2026-08-24]` **Serving the tuned ERP model: LoRA-on-NVFP4 PREFERRED, merged weights the expected fallback — and the recorded objection may be STALE.** Operator: "if you CAN load it as a lora, all the better, the issue is that we will want to run nvfp4 weights, which we had some serious trouble with loading loras on top of nvfp4." ⚠ **The archived root-cause says it was NOT NVFP4-specific**: `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application was a silent no-op (#47639, regression from #37912) — adapter loads HTTP 200, zero deltas at inference, proven **quant-agnostic (NVFP4 AND FP8 both inert)** and adapter-format-agnostic by a 3-peer dwarf panel. Fix PR #47640 was OPEN then; **ana-ml2 now runs vLLM 0.26.0, two minor versions on**. So: RETEST hot-swap LoRA on 0.26.0 before designing around merge — it is cheap, and if it works the post-tune gate can be two aliases on one engine. If it still no-ops, merged weights it is, which means the harness must EMIT merged weights and Eitri needs that in the contract while he is early. Tracked at this snapshot commit; settle it in the QLoRA sizing conversation. - `[2026-08-24]` **Homepage rebuilt on Australis Skyfall; light mode shipped.** Two findings worth more than the theme: **(a)** the Skyfall bundle including its canonical light ramp was sitting in this repo's git history at `45c1995` — check `git show` before concluding a vendored design asset is lost; **(b)** removing `theme:` from `settings.yaml` deterministically breaks the dashboard render (six recreates empty, restoring the key fixed it in 12s), which is the first confirmed cause of the "tab bar goes missing" symptom. Retires the `homepage.log` size lead from earlier the same day — it did nothing on this episode. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` - `[2026-08-24]` **Homepage reorganised on the axis "do I open this?" — UI groups expanded on top, API/agent groups collapsed at the bottom** (operator-delegated: "re-categorize however you want"). Load-bearing constraint: `homepage.group` is read at container CREATION, so the 16 GPU-backed model seats keep their unlovely names rather than eat a recreate — `initiallyCollapsed` + order is free. Second rule discovered here: **group members should all have widgets or none should**, because a stat strip adds ~50px and opens a void beside plain cards. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` - `[2026-08-24]` **Homepage columns unified at 4 for every group; the 2026-08-18 "columns = member count" rule is retired.** It was avoiding dead cells in a short last row and bought a worse defect — card width changing at every group boundary. Also carries two CSS traps: `overflow: hidden` clips at the PADDING box (so a `padding-right` gutter is spill room, not a guard), and a `:root` override of a Homepage theme variable is silently outranked by `.theme-slate` on the same `` element. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` @@ -255,21 +248,6 @@ _As of 2026-08-24 — a very long ops session; everything it opened, it closed. - `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/.wav` + `transcripts/.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]] -- `[2026-08-08]` **worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev.** worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared `~/snapshots/worldtree-400-fiction-decomp` (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). **LEFT INTACT:** `rex393-fiction-index`/`rex393-fiction-snapshot` (separate operator KEEP word, unchanged) + `r42-gate-*`. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (`01KZJ9GMCC…`). - - -- `[2026-08-07]` **chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed).** Long saga, operator-driven clean diagnosis. **Symptom:** ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. **NOT** German-leak (Turbo `generate()` has NO language param — plain AutoTokenizer, no `language_id`; the multilingual `language_id="en"` lever lives only in the separate `ChatterboxMultilingualTTS`), **NOT** OOM alone. **Real cause:** the Chatterbox **Turbo T3 model OVER-RUNS its generation tail** — a long single `generate()` degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens `<6561` + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). **ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE** — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). **Diagnosis method** (deterministic, no ears-only): single-shot length sweep + **amplitude-gated voiced-ZCR** (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. **FIX:** server-side `max_chunk_chars=250` cap on the scheduler (`:v2` image, `CBF_MAX_CHUNK_CHARS=250` env) — bounds each generation to just under the ~300-char onset → clean **3-4 sentence** chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; **chatterbox's low emotiveness keeps chunk joins smooth** (the harsh joins that got Zonos rejected are absent — operator's key call). **ratatoskr TODO (relayed msg `01KZER9X7S`):** revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. **Cap value tunable** per-request (`max_chunk_chars`) + env. **Deeper prosody** (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). **⚠ FOOT-GUNS:** (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) **build-context vs image drift** — the `:v2` image was built from cap source, but after a `:v1` rollback the build context held `:v1` source → a `docker compose build` would've silently produced a cap-less `:v2`; re-synced the flat cap source to `/opt/docker/compose/chatterbox-fast/` (rebuild-verified). **⚠ DIVERGENCE (follow-up):** deployed build context is FLAT (`app.py`/`scheduler.py`, `from scheduler import`, thin-overlay `FROM local/chatterbox:v1`, cap-only) vs the `vh/chatterbox-fast` REPO which is PACKAGE-layout (`chatterbox_fast/`, `from chatterbox_fast.scheduler`, self-contained Dockerfile) + has `norm_loudness` (repo commit `6bc7bf0` = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: `.bak-cap-20260807-104850` backups on irv-ml1 + `:v1` image both retained. [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] - - -- `[2026-08-07]` **Zonos2 TAKEN DOWN on the 3090 (irv-ml1) — operator-directed "for memory", TEMPORARY.** Freed ~17.4 GB (3090: 728 MiB → 18.2 GB free) so chatterbox-fast (co-resident, was OOMing on long generations) has headroom. **⚠ Restore is manual — Zonos2 :1920 was a DETACHED native process (NOT systemd/docker), reparented to init.** GPU memory was held by the `--multiprocessing-fork` CHILDREN (1966165=16.4G, 1966166=1G), which ORPHAN to init when you kill the parent — had to SIGTERM the children explicitly (killing the parent 1965942 + uv-run 1965935 alone left the 16.4G held). **RESTORE CMD** (from irv-ml1, user lkraven): `cd /home/lkraven/tts-audition/models/zonos2 && nohup uv run python -m zonos2 --model-path Zyphra/ZONOS2 --host 0.0.0.0 --port 1920 --tts-default-voices-dir ./default_voices/ --cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3 > /tmp/zonos2.log 2>&1 &` then `docker start zonos-gateway`. **Consumers that lost Zonos:** asset-engine + gateway-chat (via LiteLLM `ext-tts` alias → zonos-gateway :8890, now stopped); ratatoskr already migrated OFF to chatterbox-fast (unaffected). Also unblocks proper drift/cap testing (OOM was blocking it). [[reference_zonos_tts_stack]] - - -- `[2026-08-07]` **chatterbox-fast: donut voice added + full contract delivered to ratatoskr-dev (their TTS migration off Zonos).** Operator-directed. Copied `zonos-gateway/voices/Donut.wav` → chatterbox `/refs` (`/worktank/chatterbox/reference_audio/donut.wav` — the reference_audio SUBDIR is lkraven-owned so no sudo despite `/worktank` root; container globs `/refs` live → **NO restart**), exposed as `voice:"donut"` (lowercase); verified clean 7.5s synth (24kHz, RTF ~0.31). A/B booth (chatterbox vs zonos donut, same line) at `http://10.100.10.50:8090/b/donut-chatterbox/`. Answered ratatoskr's 8-question contract ask from the live gateway (`local/chatterbox-fast:v1`) + source: **NOT OpenAI-shaped** (`POST /tts`; body `text`/`voice`/`format`/`stream`, not `input`/`model`/`response_format`); **NO affect dials** (Turbo ignores cfg_weight/min_p/exaggeration — the architecture-changing answer they flagged; **Zonos stays the only fleet TTS with real emotion steering**); streaming WAV placeholder-header shape IDENTICAL to Zonos (their per-chunk Web Audio path survives); SR 24000 (Zonos 44100); server chunks arbitrary-length text internally (no client-side chunking, unlike Zonos's 71.2s cap); English-only, no language pin. **FYI-worthy (operator):** ratatoskr is moving its RP-surface TTS OFF Zonos back to chatterbox-fast → loses the live-PAD affect coupling (heavy Zonos emotion investment) — their call, trade-off flagged to them. auto-memory `reference_chatterbox_fast_repo` enriched w/ the live contract. [[reference_zonos_tts_stack]] - - -- `[2026-08-07]` **Fleet reranker cut over: Qwen3-Reranker-0.6B → BAAI/bge-reranker-v2-m3 (Brokkr R43).** The incumbent was measured HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). R43 bake-off: the A2 control (same Qwen weights, seq-cls head) scored identical to the incumbent → proved the fault is a training-prior not the serving head → cancelled the expensive Qwen3-4B arm; A3 (bge-v2-m3) won on multilingual safety + bare-name recovery. LiteLLM `reranker` repointed incumbent→A3 :8013 (boundary 2026-08-06T17:37:48Z, config-edit + ~52s gateway restart); **R42 v13 gate PASSED first-ever** (56/90→90/90). Incumbent kept warm :8002 (rollback via `qwen3-reranker` alias), A4 fallback :8014. Full arc + rollback runbook `docs/pfi/reranker-selection-ledger.md`; commits ad2df89/2c11748/377f8a4 (unpushed). auto-memories: the earlier reranker-serving notes. - - - `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md` @@ -283,7 +261,7 @@ _As of 2026-08-24 — a very long ops session; everything it opened, it closed. -_209 older entries archived to archival-memory.md._ +_214 older entries archived to archival-memory.md._ ## Tried and abandoned - `[2026-08-24]` **AES-GCM on the Anaheim tunnels — impossible, not merely hard.** UniFi's manual site-to-site IPsec implements no AEAD cipher at all: eight GCM spellings rejected `api.err.InvalidPayload` against a passing `aes256` control. Blocks both tunnels since both far ends are UDMs. Accepted enum is `aes128/aes192/aes256/3des` — and 3DES is *slower* (no ARM instructions, 64-bit blocks), so AES-128 is the floor.