memory: snapshot for /clear — fleet reranker cutover shipped+verified (R42 v13 PASS), WT#394 kb-contamination diagnosed (attribution UNRESOLVED, shared-identity gap parked), b182 recreate-verify in-flight

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-08-03_
+_Last updated: 2026-08-07_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -109,31 +109,31 @@ no longer deployed sidecars here. See Recent decisions.)
 
 ## Current state / in-flight
 
-_As of 2026-08-05 — **session at a natural close; nothing infra-ops-side mid-action.** This session (08-05): shipped **3 Booth features** (verbatim-wrap back-chip+favicon, `.md`/`.txt` doc-viewer, image prev/next arrows — all live on `:8090`, committed + tagged `booth-v0.1.5/0.1.6/0.1.7`); root-caused the **worldtree herald re-nudge bug** → forseti shipped **althing-core v2.1.2** (`d5d33df`, deployed on nh3-dev); attempted the **fleet-CI-resilience flip** and **PARKED** it on a runner-auth blocker (→ Recent decisions, my deferred research task) with gitea **`REQUIRE_SIGNIN_VIEW=false`** now a standing change. The 08-03 b169 watch is CLOSED (worldtree-dev confirmed both instances clean). The prior-arc headlines below are historical context (see the 2026-08-02/03 Recent-decisions entries):_
-- **mimir-inbox DEPLOYED + verified on corviduo-dev `10.250.50.152:8091`** (#377 browser-facing half; write+read proven end to end). Live commit `8ece117` (3 redeploys); co-located per the operator's reversed-to-CO-LOCATE ruling. auto-memory `reference_mimir_inbox_deploy`.
-- **#377 read path fully working** — P&P ingested + queryable via Mimir on personal :8081. Chased through worldtree-dev bugs **#380** (wing-blind index → concepts in wrong collection; fixed b164 + one-shot `--reindex <job_id>`), **#381** (stale Chroma client → **restart `worldtree-personal-worldtree-api-1` after any ingest/re-index** until their fix), **#382** (intermittent Mimir grounding / silent training-substitution; fixed b166 = `_index.md` per wing job-dir + a prompt rule; **verified 3/3** by ratatoskr-dev). DCC re-file SETTLED = **no** (prompt rule grounds it; `_index.md` rides next re-ingest).
-- **muninn-gate → muninn-dispatch 0.1.5** (rebuilt off `vh/muninn-gate` `bc04c4c`, image 0.0.14; serves `concept_schema`/`concept_schema_source`). BuildKit gitea secret; recreate w/ `compose up -d` not bare restart.
-- **donut voice** cloned from the **65-frost Booth bundle** → registered in the Zonos gateway (`voice:"donut"`, live in the Asset Engine TTS-zoo make form; auditioned in booth `donut-voice`). **onyx-58 expansion TRIED → REVERTED 2026-08-02:** folded the `onyx-58` bundle (seg101/seg110/seg148, all Donut — `seg148` was diarized SPEAKER_03 but operator-confirmed misdiarize) in alongside seg000 → 52.0s multi-clip ref, but a pinned-seed **neutral/no-emotion** A/B (5 pairs, booth `donut-onyx58`) showed the **single-clip seg000 (16.3s) wins on timbre fidelity** — the 4-take concat muddied the speaker embedding. **LIVE = seg000-alone** (reverted both live bind-mount + build-source tree; old ref was at `irv-ml1:~/Donut.wav.pre-onyx58`). See Tried-and-abandoned for the emotion-fidelity lesson. Zonos `/v1/audio/speech` already streams (chunked, TTFB ~0.44s) — ratatoskr shipped the client-side chunk-passthrough for play-as-it-arrives; **no gateway change was needed**.
+_As of 2026-08-07 — heavy overnight session, now at a natural rest; one background watcher live (b182 recreate verify). The night's four arcs are all landed or handed off:_
 
-**Open follow-ups (non-blocking — pick one up or not):**
-- **Zonos emotion:** sad axes/text pass on the 3 calibrated voices (only named-sad, untested); emotion-congruent-text pass (validates intensity, may rescue sad id); clone-char (Emmie/Penny/Natalie/Miranda) emotion rows use the mid-region fallback until measured. Presets are **provisional** (neutral-text ear-check was inconclusive). Tools `~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio,assemble_voice}.py` (run ON irv-ml1; dial-in studio = nohup :8898 on nh3-dev). dvalin thread at rest (`01KXT12FN0AS…`). → `persistent-memory.d/2026-07-18-zonos-gateway-0.2.1-emotion-presets.md`
-- **zonos-gateway CI-wire:** deployed irv-ml1 tree `/opt/docker/compose/zonos-gateway` is still NON-git (hand-updated build context) — git-connect + build-on-push like the other sisters. (Same pattern soong-lab now has.)
-- **soong-lab:** cutover DONE + **auto-redeploy DONE + validated 2026-07-18** (CI-deploy step live; dispatch run #5 recreated the live container ...541f7730 → ...07526a08, health-gated green). Deploy dir now **/opt/soong-lab** (deploy-owned, mirrors /opt/worldtree); old `/home/infra-ops/soong-lab-deploy` retired (`.retired-20260718`). Dedicated soong-only ed25519 deploy key on `deploy`'s authorized_keys (fp SHA256:MG7M3Ri…). → `persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md`.
+- **FLEET RERANKER CUT OVER (the big one) — DONE + VERIFIED.** Brokkr R43 bake-off replaced the fleet `reranker` (Qwen3-Reranker-0.6B, measured HARMING 80/90 queries fleet-wide — no-reranker beat it 89/90 vs 56/90) with **A3 = BAAI/bge-reranker-v2-m3**. LiteLLM `reranker` alias repointed incumbent→A3 (boundary **2026-08-06T17:37:48Z**, config-edit on ana-docker litellm + ~52s gateway restart); Brokkr independently VERIFIED (maxdiff 0, single backend) and the **R42 v13 acceptance gate PASSED — first ever** (main+kb 56/90→90/90, evictions 33→0). Bench wound down: A2 (control) torn down; **A3 = PROD on ana-ml2 GPU1 :8013**; **A4 (gte-modernbert) = warm fallback :8014**; **incumbent still up :8002 for rollback** (via the untouched `qwen3-reranker` alias — rollback = revert one config line + a SECOND ~52s gateway restart). A3/A4 `restart=unless-stopped`. Full arc + rollback runbook: **`docs/pfi/reranker-selection-ledger.md`** (git-tracked; commits ad2df89/2c11748/377f8a4, UNPUSHED). **Passive watch:** A3 caps ~34 req/s (vs incumbent's ~55) — queues gracefully; if real p99 bites, levers = A4 swap / bump A3 gpu-util / 2nd A3 replica (Brokkr re-runs the grid, no intuition swap). **Non-urgent follow-up:** promote A3 from throwaway `docker run` to a `stacks/vllm/` compose service (a recreate briefly drops the alias → needs a window); retire incumbent when the post-cutover watch closes.
 
-**Zonos voice stack (LIVE):** **9 voices** in `zonos-gateway` (`:8890` irv-ml1) — defaults AmericanFemale/Male/BritishFemale/Cora + 4 clones Emmie/Penny/Natalie/Miranda + **donut** (2026-08-02, from the 65-frost bundle, **expanded same day with onyx-58 clips → 52.0s multi-clip ref**); add a voice = drop `<Name>.wav` (44.1kHz mono s16 PCM) in `/opt/docker/compose/zonos-gateway/voices/` + `docker compose restart` (host-managed bind-mount, NO rebuild; registry scans at startup). Also mirror into the build-source tree `~/zonos-gateway/voices/` for rebuild-durability. Clone pipeline: `/mnt/smithy/voice_clones/<name>.zip` → `assemble_voice.py` → drop. Dial-in studio http://10.100.10.50:8898/ (nohup on nh3-dev, relaunch `nohup python3 ~/development/zonos-tools/dial-in-studio.py >/tmp/zonos-studio.log 2>&1 &`).
+- **IN-FLIGHT RIGHT NOW:** b182 staging deploy (ships the WT #394 kb-walk fix) is rolling out → worldtree-personal recreates imminently. A **background watcher confirms the recreate lands clean (kb stays 8,230, no re-sweep), then I RETIRE the flag-before-personal-redeploy commitment** (b182 is the very deploy that fixes the recontamination risk that flag guarded). Baseline captured: personal on b181, kb=8,230.
 
-**althing monitor** ARMED (handle `infra-ops`; herald up; wake-listener task-id rotates every re-arm). ⚠️ Re-arm ONLY after an actual FIRE (`<task-notification> completed rc0`), never after a plain operator turn (bounces rc3). Spawn `althing-wake-listener` as its OWN `run_in_background` task — NEVER chain with `&`/`&&` (orphans it → untracked → mail unwatched). **b169 watch CLOSED** (2026-08-05 — worldtree-dev confirmed both instances clean after the re-run; was a shared-containerd pull-race, NOT disk — see Tried-and-abandoned). **Current standing item — worldtree #389 config delta (2026-08-05, parity-only, NO infra-ops action):** commit `f66559c` (unpushed) changes `agents/mimir/{system.md, config.yaml}`; `agents/` is **IN-IMAGE** (Dockerfile:54 `COPY agents/`), not among the 5 bind-mounted config files, so it ships to demo/personal via worldtree-dev's **CI/CD image build** — infra-ops just stands by, no deploy-wt-config pre-sync (source-verified with worldtree-dev; `matrix.yaml` carries only per-agent display identity, no companion; pinned excluded); b174 pushed 2026-08-05 (`9b7f1fd`), demo deploy running — no action. **+ CARRIED TASK (do early next session):** publish **muninn-dispatch 0.1.6** to the Gitea PyPI registry (worldtree-dev asked; source on Worldtree `origin/main` `packages/muninn-dispatch`, comment-only bump; self-serve via the vh-token per `reference_infra_ops_vh_gitea_token_and_sdk_publish`), then ping worldtree-dev to announce — muninn-dev holds its 0.1.5 pin awaiting it. See the /tmp handoff. **STANDING CONFIG CHANGE:** gitea `REQUIRE_SIGNIN_VIEW=false` on ana-docker (from the CI-flip attempt, kept per operator — internal WG net). **PARKED watches:** #363 research-wing ingest (no deadline); worldtree-dev's `/embed` fix reaching PERSONAL (operator's call). The **worldtree b168/#384/#385 arc is COMPLETE** — runbooks + full detail in the 2026-08-03 Recent-decisions entry.
+- **kb-CONTAMINATION INCIDENT (WT #394) — diagnosed, fix shipped (b182), attribution CLOSED UNRESOLVED.** A `WingStore._embed` reconcile full-tree walk (kb wing `fs_root = KB_PATH` root, sibling wings nested under it) swept **5,354** fiction+main rows into personal's `knowledge_base` (incl. 2 SUPERSEDED generations served as current). Fixed by worldtree #394 (aca39a1, kb walks exclude sibling wings). Trigger un-attributable — a peer reconcile via the SHARED infra-ops identity leaves no fingerprint (0 dockerd/containerd exec-logging). Durable finding → auto-memory `infra_ops_shared_identity_attribution_gap`, **PARKED (operator ruled A) into the identity-hygiene arc** [[project_migrate_infra_access_to_claude_credentials]]. **⚠ EVIDENCE HOLD:** do NOT delete the 5,354 swept rows or the on-disk generations — cleanup rides the operator's SEPARATE authorization (on #394's agenda with Brokkr).
 
-**Two small pending items (operator's call, non-urgent):** (1) bless/reshape the `env.public` non-secret-env-overlay mechanism in the config repo; (2) the pre-existing herald pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range" — likely `render_command messages[0]` on empty list; NOT infra-ops's, rec = flag to althing-dev).
+- **gen-reasoning empty-content (WT #392) — serving EXONERATED, no infra action.** ~2-3% zero-concept sections on the personal concepts seat = thinking-model channel behavior + broken JSON string-escapes, NOT serving config (`vllm-aeon-gen` :8015 runs `--reasoning-parser qwen3`, no reasoning-budget). rex393 raw data confirmed; worldtree's b179 retry self-heals at zero data loss. The thinking-off lever is Worldtree-court AND benched-mandatory (reasoning needed for the significance filter per #385), so no flip.
 
-**eshpfi push state:** `main` WAS pushed to origin this session (synced through the booth verbatim-wrap + memory commits); a handful of 2026-08-05 commits are unpushed since — booth doc-viewer `315faac`, arrows `c37a425`, triton memory `b5ae936`, + this snapshot. **Push is the operator's call.** `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's); `graphify-out/GRAPH_REPORT.md` = graphify-hook artifact (churns every commit, ignore, don't stage).
+- **PERSONAL-KB SNAPSHOTS on nh3-dev** (read-only pulls for worldtree probes, provenance-marked, do NOT auto-delete): `~/snapshots/r42-gate-index/` (#389, operator ruled KEEP), `~/snapshots/rex393-fiction-index/` (#393, full multi-collection persist store — fiction 1578 / main 3224 / kb 8230; clear when worldtree signals #393 done). The `conversations.db` attribution copy was SHORT-LIVED and is ALREADY DELETED.
 
-**PARKED (grok-code/Codex):** operator asked about fronting grok-code / Codex behind the LiteLLM gateway. Rec (given): raw models behind the gateway → **API keys** (native `xai/` + `openai/` providers, the GLM-passthrough pattern); fleet *consults* → the **Heid/Eitri peer-CLI** pattern (Codex already wired). Do NOT reverse-proxy the subscription CLIs (grok CLI / Codex CLI, OAuth-auth) into the gateway — ToS + account-ban risk + brittle. Untracked by operator choice; no decision made.
+- **CARRIED still-pending from 08-05 (NOT done this session):** publish **muninn-dispatch 0.1.6** to the vh Gitea PyPI (worldtree-dev asked; source Worldtree `origin/main packages/muninn-dispatch`, comment-only bump; self-serve via the vh-token per `reference_infra_ops_vh_gitea_token_and_sdk_publish`), then ping worldtree-dev; muninn-dev holds its 0.1.5 pin awaiting it. Other standing/parked: CI-flip runner-auth research (Recent decisions 08-05 detail file); gitea `REQUIRE_SIGNIN_VIEW=false` standing; #363 research-wing ingest (no deadline); zonos emotion + zonos-gateway CI-wire follow-ups.
+
+**althing monitor** ARMED (handle `infra-ops`; herald up; wake-listener task-id rotates every re-arm). ⚠️ Re-arm ONLY after a real FIRE (`<task-notification> completed rc0`), never after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, NEVER chained with `&`/`&&`.
+
+**eshpfi push state:** `origin/main` is behind by **7 commits** (08-05 booth/memory + this session's 3 reranker commits ad2df89/2c11748/377f8a4 + this snapshot). **Push is the operator's call.** `graphify-out/GRAPH_REPORT.md` = graphify-hook artifact (churns every commit, ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's).
 
-**Carried standing (non-blocking):** **INFRA (non-)WATCH (2026-08-05, worldtree-dev — CORRECTED, was over-stated):** the R42 empties were **NOT an infra fault** — Brokkr's full-args capture found mimir **hallucinating a non-empty `facet_filter`** that legitimately matches nothing (agent-side, worldtree's). **NO evidence of an embed/Chroma silent-empty seam fault as of 2026-08-05** (my empty-log grep was the finding, but the cause was agent, not infra). The tripwire now in the instance logs (b175/b176 guards): **"filters … emptied a subject-bearing search"** = agent hallucination (worldtree's, expected occasional noise, NOT infra) · **"silent zero-row"** = a genuine embed/Chroma seam fault (MINE) → dig at the embed/Chroma layer **only if these show in volume.** ana-ml2 GPU0 ~14 G reserve; irv-ml1 3090 oversubscription (kokoro :8193 + vibevoicefusion :9527 idle-pinned + zonos :1920 — operator declined to fix); rotate the 5 rest-server backup creds (operator, offline); Worldtree #363 research-wing ingest (parked, no deadline); T1 SFT LoRA dormant; Zonos2 engine still NATIVE (containerize deprioritized); **Audio8 → TTS zoo** (PARKED 2026-08-02 per operator — download Audio8 + add it to the fleet TTS zoo [= the Asset Engine catalog at ana-docker:8200, ~20 audio svcs w/ irv-ml1 endpoints; Zonos native :1920, zonos-gateway :8890, chatterbox-fast :8197, etc.] DEFERRED, not now; un-park = confirm Audio8's source/nature w/ operator, then register as a new Asset-Engine service entry [endpoint on irv-ml1] per the zoo convention); **worldtree config-fold PENDING** (next routine config sync): add `reference_knowledge.tier3_wings: ["fiction"]` to BOTH demo+personal defaults.yaml (worldtree #383, b5db691, PARITY-ONLY — baked default matches so no behavior diff, not boot-blocking; per-instance widening [e.g. +main on personal] is this key's purpose).
 
 ## Recent decisions
 
+- `[2026-08-07]` **Fleet reranker cut over: Qwen3-Reranker-0.6B → BAAI/bge-reranker-v2-m3 (Brokkr R43).** The incumbent was measured HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). R43 bake-off: the A2 control (same Qwen weights, seq-cls head) scored identical to the incumbent → proved the fault is a training-prior not the serving head → cancelled the expensive Qwen3-4B arm; A3 (bge-v2-m3) won on multilingual safety + bare-name recovery. LiteLLM `reranker` repointed incumbent→A3 :8013 (boundary 2026-08-06T17:37:48Z, config-edit + ~52s gateway restart); **R42 v13 gate PASSED first-ever** (56/90→90/90). Incumbent kept warm :8002 (rollback via `qwen3-reranker` alias), A4 fallback :8014. Full arc + rollback runbook `docs/pfi/reranker-selection-ledger.md`; commits ad2df89/2c11748/377f8a4 (unpushed). auto-memories: the earlier reranker-serving notes.
+
+- `[2026-08-07]` **Personal-Worldtree kb-contamination incident (WT #394) diagnosed; attribution CLOSED UNRESOLVED.** A reconcile `WingStore._embed` full-tree walk (kb `fs_root=KB_PATH` root, sibling wings nested) swept 5,354 fiction+main rows into personal's `knowledge_base` (2 superseded generations served as current). Fixed by WT #394 (aca39a1, kb walks exclude sibling wings; ships b182). Trigger un-attributable — peer reconcile via the SHARED infra-ops identity + 0 dockerd exec-logging = fingerprint-less. Durable finding → auto-memory `infra_ops_shared_identity_attribution_gap`, PARKED (operator ruled A) into [[project_migrate_infra_access_to_claude_credentials]]. Evidence hold on the 5,354 rows until operator sequences cleanup (w/ Brokkr, on #394's agenda).
+
 - `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md`
 
 - `[2026-08-05]` **Booth — 3 features shipped, live on `:8090` + tagged.** (1) verbatim-`index.html` booths get a floating top-right "‹ all booths" chip + inherited favicon, doctype/charset-safe byte-injection (`booth-v0.1.5`, `8577e7e`); (2) `.md` renders + `.txt`/`.log` view in-booth without downloading via the `/b/<n>/view` route + a `markdown` dep + `doc.html` (`booth-v0.1.6`, `315faac`); (3) prev/next arrows in the image zoom viewer — wrap-around + keyboard ←/→, hidden for single-image booths (`booth-v0.1.7`, `c37a425`). Canonical `services/booth/`; deploy = `systemctl --user restart booth.service` on nh3-dev (runs from the checkout's `.venv`; `uv pip install` new deps into it first); 47 tests. `uv.lock` gitignored (`348c5c1`).