From b4ef0600d224a2233fa6555eab56b997491b3752 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 2 Aug 2026 22:54:32 -0700 Subject: [PATCH] memory: BrokenPipe wing-git watch CLOSED (round-2 commits clean, chown enabled) worldtree-dev closed the watch with git evidence: both round-2 wing commits landed clean under exec -u 1000; loose-object chown enabled it. Self-healed as predicted; last thread of the reindex-as-root fallout closed. --- persistent-memory.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/persistent-memory.md b/persistent-memory.md index a946938..02845ae 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -122,7 +122,7 @@ _As of 2026-08-02 — **session at a natural close; nothing infra-ops-side block **Zonos voice stack (LIVE):** **9 voices** in `zonos-gateway` (`:8890` irv-ml1) — defaults AmericanFemale/Male/BritishFemale/Cora + 4 clones Emmie/Penny/Natalie/Miranda + **donut** (2026-08-02, from the 65-frost bundle, **expanded same day with onyx-58 clips → 52.0s multi-clip ref**); add a voice = drop `.wav` (44.1kHz mono s16 PCM) in `/opt/docker/compose/zonos-gateway/voices/` + `docker compose restart` (host-managed bind-mount, NO rebuild; registry scans at startup). Also mirror into the build-source tree `~/zonos-gateway/voices/` for rebuild-durability. Clone pipeline: `/mnt/smithy/voice_clones/.zip` → `assemble_voice.py` → drop. Dial-in studio http://10.100.10.50:8898/ (nohup on nh3-dev, relaunch `nohup python3 ~/development/zonos-tools/dial-in-studio.py >/tmp/zonos-studio.log 2>&1 &`). -**althing monitor** ARMED (handle `infra-ops`; herald up; wake-listener task-id rotates every re-arm). ⚠️ Re-arm ONLY after an actual FIRE (` completed rc0`), never after a plain operator turn (bounces rc3). Spawn `althing-wake-listener` as its OWN `run_in_background` task — NEVER chain with `&`/`&&` (orphans it → untracked → mail unwatched; I fumbled this ~4× this session, each time = stop-monitor + re-arm clean). **Open watches:** worldtree-dev may ping when their `/embed` code fix reaches PERSONAL (still 503 on b146 until a staging promotion — operator's call); #363 research-wing ingest PARKED; **#381 restart DONE (2026-08-02):** DCC job `mimir-6351554e8e8f` completed (145 sections/97 concepts, wing=fiction); restarted `worldtree-personal-worldtree-api-1` (plain restart, healthz/readyz 200 ~28s); confirmed retrieval-visibility via a Mimir session — `search_library` on `wings:["fiction"]` returned 10 DCC/Princess-Donut hits (fresh Chroma client sees the new writes). Scoped as RETRIEVAL, not grounding (thread `01KZ229N5Q2Y`, replied); grounding verify handed to ratatoskr-dev. **Runbook confirmed:** admin token (`~/.config/worldtree/personal-admin-token`, wildcard scope) → Mimir session (agent_id=`mimir`, `record_tool_intermediates=true`) → post message → read `/sessions/{id}/tool-events` + parse SSE `tool_result` → DELETE session. Message endpoint STREAMS SSE (not JSON). **providers.yaml pre-sync DONE (2026-08-02):** #384/#385 b168 (commit 293f8f3) needed a boot-gating `summarization` capability block (vastblueai-gateway/gen, non-thinking, temp 0.5, 8192, after `classification`) + a deep-reasoning description refresh. Synced BOTH hunks into demo + personal `providers.yaml` via `deploy-wt-config` (health-gated, both api healthy ~18-21s); instance-configs repo commit `53349f8`; replied GO (`01KZ2NMER3EP78X95CMZGP76BK`) → they push main (demo auto-deploy) + cut personal staging tag. **GOTCHAS for next time:** (1) the source commit was UNPUSHED at sync time — `git show ` 404s and a gitea `raw?ref=` silently falls back to the default branch; verify the commit exists before trusting a fetch, else ask the peer to paste hunks. (2) worldtree-dev's hunk paste indented the deep-reasoning `description:` at 8 spaces but the block keys are 4-space → invalid YAML; corrected to 4-space (always YAML-validate after a paste-sourced edit). **deploy-wt-config runbook:** `diff`/`deploy --file providers.yaml` (repo `~/development/worldtree-instance-configs`, per-instance dirs demo/personal/pinned; deploy = host write + api/matrix restart + 90s health-gate + auto-rollback); demo+personal providers.yaml are byte-identical. See [[reference_worldtree_instance_configs_repo]]. b168 PUSHED to gitea (main 8920adf..293f8f3 + tags v1.0.0b168 / staging/v1.0.0b168); demo auto-deploys via main, personal via staging tag; worldtree-dev verified my host 4-space == commit byte-for-byte. **#384 recovery on personal DONE (2026-08-02):** b168 (`293f8f3c6077`) confirmed live on all personal containers (preflight — checked `.Config.Image` before executing) → reindex `sudo docker exec worldtree-personal-worldtree-muninn-1 python -m core.muninn --reindex mimir-6351554e8e8f` (exit 0, `indexed=97 skipped=0`) → #381 restart (healthz/readyz 200) → metadata verify 5/5 fiction rows carry non-empty note_id+path (Tier-3 backfill confirmed). Reported `01KZ2PEGZ0DVQSJN4PWMGRD198`; ratatoskr-dev runs first-light re-verify. **⚠️ OPEN (worldtree-dev's call):** reindex logged `Wing git commit failed … BrokenPipeError (reconcile-visible)` / `kb_ingest_degraded phase=indexing` — Chroma rows + on-disk concept/note files landed, only the wing git-versioning step degraded. DISPOSITION (worldtree-dev 2026-08-02): WATCH-ITEM, NO reconcile — the DCC round-2 re-run (via mimir-inbox, post-verify) re-commits the same wing dir and self-heals the missed commit; escalate ONLY if round-2's wing commit also fails (two-in-a-row). Retrieval/provenance unaffected. **Chroma-verify runbook:** `sudo docker exec -i python -` (MUST pass `-i` or stdin never reaches `python -`) → `chromadb.PersistentClient('/data/kb/.chroma').get_collection('fiction').get(where={'job_id':}, include=['metadatas'])`; fiction count 166 (69 P&P + 97 DCC). **PENDING production delete-sweep (heads-up 2026-08-02):** worldtree-dev will request (as a request to me / a supervised script, sequenced AFTER ratatoskr's verify) an explicit delete-by-job_id of the **785 April-era DCC orphan rows** in the **'main'** collection (`job_id=b59c147c5ce0`, no source_identity metadata → predate identity tracking, so supersession can't reach them; the #384 stray-row sweep removed `source_identity=76cb5829035553bf`, NOT these). This is a PRODUCTION-COLLECTION MASS DELETE → handle supervised: confirm exact job_id + expected count, snapshot/back up the collection first, and loop the operator BEFORE executing. Not yet requested. +**althing monitor** ARMED (handle `infra-ops`; herald up; wake-listener task-id rotates every re-arm). ⚠️ Re-arm ONLY after an actual FIRE (` completed rc0`), never after a plain operator turn (bounces rc3). Spawn `althing-wake-listener` as its OWN `run_in_background` task — NEVER chain with `&`/`&&` (orphans it → untracked → mail unwatched; I fumbled this ~4× this session, each time = stop-monitor + re-arm clean). **Open watches:** worldtree-dev may ping when their `/embed` code fix reaches PERSONAL (still 503 on b146 until a staging promotion — operator's call); #363 research-wing ingest PARKED; **#381 restart DONE (2026-08-02):** DCC job `mimir-6351554e8e8f` completed (145 sections/97 concepts, wing=fiction); restarted `worldtree-personal-worldtree-api-1` (plain restart, healthz/readyz 200 ~28s); confirmed retrieval-visibility via a Mimir session — `search_library` on `wings:["fiction"]` returned 10 DCC/Princess-Donut hits (fresh Chroma client sees the new writes). Scoped as RETRIEVAL, not grounding (thread `01KZ229N5Q2Y`, replied); grounding verify handed to ratatoskr-dev. **Runbook confirmed:** admin token (`~/.config/worldtree/personal-admin-token`, wildcard scope) → Mimir session (agent_id=`mimir`, `record_tool_intermediates=true`) → post message → read `/sessions/{id}/tool-events` + parse SSE `tool_result` → DELETE session. Message endpoint STREAMS SSE (not JSON). **providers.yaml pre-sync DONE (2026-08-02):** #384/#385 b168 (commit 293f8f3) needed a boot-gating `summarization` capability block (vastblueai-gateway/gen, non-thinking, temp 0.5, 8192, after `classification`) + a deep-reasoning description refresh. Synced BOTH hunks into demo + personal `providers.yaml` via `deploy-wt-config` (health-gated, both api healthy ~18-21s); instance-configs repo commit `53349f8`; replied GO (`01KZ2NMER3EP78X95CMZGP76BK`) → they push main (demo auto-deploy) + cut personal staging tag. **GOTCHAS for next time:** (1) the source commit was UNPUSHED at sync time — `git show ` 404s and a gitea `raw?ref=` silently falls back to the default branch; verify the commit exists before trusting a fetch, else ask the peer to paste hunks. (2) worldtree-dev's hunk paste indented the deep-reasoning `description:` at 8 spaces but the block keys are 4-space → invalid YAML; corrected to 4-space (always YAML-validate after a paste-sourced edit). **deploy-wt-config runbook:** `diff`/`deploy --file providers.yaml` (repo `~/development/worldtree-instance-configs`, per-instance dirs demo/personal/pinned; deploy = host write + api/matrix restart + 90s health-gate + auto-rollback); demo+personal providers.yaml are byte-identical. See [[reference_worldtree_instance_configs_repo]]. b168 PUSHED to gitea (main 8920adf..293f8f3 + tags v1.0.0b168 / staging/v1.0.0b168); demo auto-deploys via main, personal via staging tag; worldtree-dev verified my host 4-space == commit byte-for-byte. **#384 recovery on personal DONE (2026-08-02):** b168 (`293f8f3c6077`) confirmed live on all personal containers (preflight — checked `.Config.Image` before executing) → reindex `sudo docker exec worldtree-personal-worldtree-muninn-1 python -m core.muninn --reindex mimir-6351554e8e8f` (exit 0, `indexed=97 skipped=0`) → #381 restart (healthz/readyz 200) → metadata verify 5/5 fiction rows carry non-empty note_id+path (Tier-3 backfill confirmed). Reported `01KZ2PEGZ0DVQSJN4PWMGRD198`; ratatoskr-dev runs first-light re-verify. **⚠️ OPEN (worldtree-dev's call):** reindex logged `Wing git commit failed … BrokenPipeError (reconcile-visible)` / `kb_ingest_degraded phase=indexing` — Chroma rows + on-disk concept/note files landed, only the wing git-versioning step degraded. DISPOSITION (worldtree-dev 2026-08-02): WATCH-ITEM, NO reconcile — the DCC round-2 re-run (via mimir-inbox, post-verify) re-commits the same wing dir and self-heals the missed commit; escalate ONLY if round-2's wing commit also fails (two-in-a-row). Retrieval/provenance unaffected. → **CLOSED (worldtree-dev 2026-08-03):** round-2's wing commits landed clean (verified `exec -u 1000`); wing HEAD = "Muninn ingest: mimir-6351554e8e8f" with P&P beneath; my loose-object chown almost certainly enabled the commit. Self-healed as predicted — watch-item retired. **Chroma-verify runbook:** `sudo docker exec -i python -` (MUST pass `-i` or stdin never reaches `python -`) → `chromadb.PersistentClient('/data/kb/.chroma').get_collection('fiction').get(where={'job_id':}, include=['metadatas'])`; fiction count 166 (69 P&P + 97 DCC). **PENDING production delete-sweep (heads-up 2026-08-02):** worldtree-dev will request (as a request to me / a supervised script, sequenced AFTER ratatoskr's verify) an explicit delete-by-job_id of the **785 April-era DCC orphan rows** in the **'main'** collection (`job_id=b59c147c5ce0`, no source_identity metadata → predate identity tracking, so supersession can't reach them; the #384 stray-row sweep removed `source_identity=76cb5829035553bf`, NOT these). This is a PRODUCTION-COLLECTION MASS DELETE → handle supervised: confirm exact job_id + expected count, snapshot/back up the collection first, and loop the operator BEFORE executing. Not yet requested. **Two small pending items (operator's call, non-urgent):** (1) bless/reshape the `env.public` non-secret-env-overlay mechanism in the config repo; (2) the pre-existing herald pane-route errors on `worldtree-codex` + `eitri-smithy-dev` ("route-error: list index out of range" — likely `render_command messages[0]` on empty list; NOT infra-ops's, rec = flag to althing-dev).