diff --git a/persistent-memory.md b/persistent-memory.md index d755b81..2ea6aad 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-08-07_ +_Last updated: 2026-08-08_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -109,23 +109,21 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-08-07 — heavy overnight session landed; b182 deployed and the operator-authorized personal-KB orphan sweep EXECUTED + verified this (post-/clear) session. The night's four arcs are all landed or handed off:_ +_As of 2026-08-08 — long session; all major arcs LANDED (full detail per arc is in Recent decisions). Genuinely in-flight now is thin:_ -- **FLEET RERANKER CUT OVER (the big one) — DONE + VERIFIED.** Brokkr R43 bake-off replaced the fleet `reranker` (Qwen3-Reranker-0.6B, measured HARMING 80/90 queries fleet-wide — no-reranker beat it 89/90 vs 56/90) with **A3 = BAAI/bge-reranker-v2-m3**. LiteLLM `reranker` alias repointed incumbent→A3 (boundary **2026-08-06T17:37:48Z**, config-edit on ana-docker litellm + ~52s gateway restart); Brokkr independently VERIFIED (maxdiff 0, single backend) and the **R42 v13 acceptance gate PASSED — first ever** (main+kb 56/90→90/90, evictions 33→0). Bench wound down: A2 (control) torn down; **A3 = PROD on ana-ml2 GPU1 :8013**; **A4 (gte-modernbert) = warm fallback :8014**; **incumbent still up :8002 for rollback** (via the untouched `qwen3-reranker` alias — rollback = revert one config line + a SECOND ~52s gateway restart). A3/A4 `restart=unless-stopped`. Full arc + rollback runbook: **`docs/pfi/reranker-selection-ledger.md`** (git-tracked; commits ad2df89/2c11748/377f8a4, UNPUSHED). **Passive watch:** A3 caps ~34 req/s (vs incumbent's ~55) — queues gracefully; if real p99 bites, levers = A4 swap / bump A3 gpu-util / 2nd A3 replica (Brokkr re-runs the grid, no intuition swap). **Non-urgent follow-up:** promote A3 from throwaway `docker run` to a `stacks/vllm/` compose service (a recreate briefly drops the alias → needs a window); retire incumbent when the post-cutover watch closes. +- **AWAITING peers (non-blocking):** (1) **ratatoskr-dev** to apply their side of the chatterbox-fast fix — revert sampling knobs to default (top_k 80→1000, temp 0.5→0.8) + send full text (server now caps generations at 250 chars, chunks internally); relayed msg `01KZER9X7S`. (2) **worldtree-dev** running the #400 exact-lexical rank decomposition against the Chroma snapshot I pulled to `nh3-dev:~/snapshots/worldtree-400-fiction-decomp/chroma/`. -- **b182 DEPLOYED + personal-KB orphan sweep DONE (2026-08-07 ~14:0xZ).** personal-api recreated onto b182 (`gitea…/worldtree:d6e5d6ec7b00`, git `d6e5d6e`, image `sha256:3f82b147…`, healthy, recreated 13:55:16Z; identical to the image demo serves — worldtree-dev re-tagged `staging/v1.0.0b182` onto `d6e5d6ec7b00` after the mid-train-memory-commit staging-tag miss). First-boot kb held at 8,230 (no re-sweep — the #394 fix works). All 3 preconditions verified (image/kb/orphan) → per operator's AUTO-EXECUTE ruling (no final-GO pause) ran `reconcile --repair` (uid 1000, index-rows-only): **kb 8,230 → 2,876** (−5,354 orphans); fresh reconcile confirms orphan=0/missing=0/stale=0. flag-before-personal-redeploy commitment RETIRED (b182 is the fix that guard protected). Report posted to worldtree-dev thread `01KZE6TGRHAW` (msg `01KZE8WT6W…`). **⚠ `--repair` exited rc=1** on a NON-blocking worldtree-side bug: after the sweep, reconcile.py:166 `ws._repo.index.add(report.uncommitted_in_git)` threw `FileNotFoundError` on phantom path `docs/specs/saga-taxonomy-redesign-plan.md` (a tracked-but-deleted entry in the kb wing's 1,847 uncommitted set; GitPython can't `lstat` it). No commit ran, wing git state stable (uncommitted still 1,847). Flagged to worldtree-dev as (1) reconcile robustness gap + (2) pre-existing kb-wing git inconsistency — their code/data call, I did NOT touch reconcile.py or the wing. Sweep GATES Brokkr's 16-subject floor re-run + ratatoskr's #393 baseline (worldtree coordinates). +- **Zonos2 PARKED DOWN on the 3090** (irv-ml1; operator ruled "leave down for now" 2026-08-08) — frees ~17 GB for chatterbox-fast; asset-engine + gateway-chat TTS idle meanwhile. **Manual restore command saved** in the Zonos-down Recent-decisions entry + auto-memory `reference_zonos_tts_stack` (detached native proc; kill the multiprocessing-fork CHILDREN, not just the parent). -- **kb-CONTAMINATION INCIDENT (WT #394) — diagnosed, fix shipped (b182), attribution CLOSED UNRESOLVED.** A `WingStore._embed` reconcile full-tree walk (kb wing `fs_root = KB_PATH` root, sibling wings nested under it) swept **5,354** fiction+main rows into personal's `knowledge_base` (incl. 2 SUPERSEDED generations served as current). Fixed by worldtree #394 (aca39a1, kb walks exclude sibling wings). Trigger un-attributable — a peer reconcile via the SHARED infra-ops identity leaves no fingerprint (0 dockerd/containerd exec-logging). Durable finding → auto-memory `infra_ops_shared_identity_attribution_gap`, **PARKED (operator ruled A) into the identity-hygiene arc** [[project_migrate_infra_access_to_claude_credentials]]. **EVIDENCE HOLD — INDEX-ROW half LIFTED + EXECUTED 2026-08-07:** operator authorized (relayed via worldtree-dev thread `01KZE6TGRHAW`); the 5,354 swept **index rows** are now DELETED via `reconcile --repair` post-b182 (kb 8,230→2,876, orphan=0 — see the b182 sweep entry above). **The FILE half of the hold STILL STANDS:** do NOT delete the on-disk generation dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`) — file-retention hold is separate and unlifted. +- **FLEET RERANKER** = A3 (bge-reranker-v2-m3) PROD on ana-ml2 GPU1 :8013, VERIFIED (R42 v13 gate PASSED first-ever, 56/90→90/90). Passive watch: A3 caps ~34 req/s; levers = A4 fallback :8014 / gpu-util / 2nd replica. Incumbent :8002 warm for rollback (revert 1 config line + ~52s gateway restart). Non-urgent follow-up: promote A3 throwaway `docker run` → `stacks/vllm/` compose service (brief alias-drop window). Full arc: `docs/pfi/reranker-selection-ledger.md`. -- **gen-reasoning empty-content (WT #392) — serving EXONERATED, no infra action.** ~2-3% zero-concept sections on the personal concepts seat = thinking-model channel behavior + broken JSON string-escapes, NOT serving config (`vllm-aeon-gen` :8015 runs `--reasoning-parser qwen3`, no reasoning-budget). rex393 raw data confirmed; worldtree's b179 retry self-heals at zero data loss. The thinking-off lever is Worldtree-court AND benched-mandatory (reasoning needed for the significance filter per #385), so no flip. +- **EVIDENCE HOLD (partial):** WT #394 index-row half LIFTED + swept (kb 8,230→2,876); the **FILE half STILL STANDS** — do NOT delete the on-disk generation dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`). Also un-actioned worldtree-side: the `reconcile --repair` rc=1 git-staging bug (phantom `docs/specs/saga-taxonomy-redesign-plan.md`) — flagged, their code. -- **PERSONAL-KB SNAPSHOTS on nh3-dev** (read-only pulls for worldtree probes, provenance-marked, do NOT auto-delete): `~/snapshots/r42-gate-index/` (#389, operator ruled KEEP), `~/snapshots/rex393-fiction-index/` (#393, full multi-collection persist store — fiction 1578 / main 3224 / kb 8230; clear when worldtree signals #393 done), `~/snapshots/worldtree-400-fiction-decomp/chroma/` (#400 exact-lexical rank decomposition, pulled 2026-08-08T01:52:19Z — full `.chroma` persist store, POST-#394 state: fiction 1578 / main 3224 / **kb 2876** / ingested_concepts 0; clear when worldtree signals #400 done). The `conversations.db` attribution copy was SHORT-LIVED and is ALREADY DELETED. +- **OPEN FOLLOW-UPS:** chatterbox-fast **deployed flat-build-context vs package-repo divergence** (reconcile so a repo rebuild matches deploy — repo commit `6bc7bf0` has cap+norm_loudness in package layout; deployed is flat cap-only). Standing/parked: CI-flip runner-auth research, gitea `REQUIRE_SIGNIN_VIEW=false`, #363 research-wing ingest (no deadline), zonos-gateway CI-wire. -- **muninn-dispatch 0.1.6 PUBLISHED to vh Gitea PyPI (2026-08-07) — worldtree-dev's carried 08-05 ask, DONE.** Comment-only bump. Built from a clean `git archive origin/main packages/muninn-dispatch` (commit `578f8fc` "release 0.1.6 — provenance sweep of shipped-code comments", on the b182 line, origin/main tip `d6e5d6e`) — NOT the operator's dirty working tree; hatchling, deps=[] stdlib-only (INV-377-5). `uv publish` via the vh-token; index lists 0.1.6 (whl+sdist), clean-venv install + import + `importlib.metadata==0.1.6` all verified. Pinged muninn-dev (lift the 0.1.5 pin) + worldtree-dev (done). **Runbook note:** monorepo subpackage → source via `git archive` of the Worldtree subpath (the sdk runbook's API-archive is for standalone repos); otherwise identical to `reference_infra_ops_vh_gitea_token_and_sdk_publish`. **Other standing/parked:** CI-flip runner-auth research (Recent decisions 08-05 detail file); gitea `REQUIRE_SIGNIN_VIEW=false` standing; #363 research-wing ingest (no deadline); zonos emotion + zonos-gateway CI-wire follow-ups. +**althing monitor** ARMED (handle `infra-ops`, task `b1uat4njl`; herald up). ⚠️ Re-arm ONLY after a real FIRE (` completed rc0`), NEVER after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, **NEVER chained with `&`/`&&`** (orphans it to init — bit me this session; reclaim with `althing-cli stop-monitor` then re-arm cleanly). -**althing monitor** ARMED (handle `infra-ops`; herald up; wake-listener task-id rotates every re-arm). ⚠️ Re-arm ONLY after a real FIRE (` completed rc0`), never after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, NEVER chained with `&`/`&&`. - -**eshpfi push state:** `origin/main` is behind by **7 commits** (08-05 booth/memory + this session's 3 reranker commits ad2df89/2c11748/377f8a4 + this snapshot). **Push is the operator's call.** `graphify-out/GRAPH_REPORT.md` = graphify-hook artifact (churns every commit, ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's). +**eshpfi push state:** `origin/main` behind by ~13 commits (prior 7 + this session's 6 memory commits: 7ea8dd3/9147bc9/2e0bb85/2d7eb90/6332f14/6399a5a + this snapshot). **Push is the operator's call.** `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED (pre-existing, operator's). ## Recent decisions