esh-pfi-infrastructure/persistent-memory.d/2026-08-02-mimir-inbox-arc.md

[2026-08-02] The mimir-inbox / #377-read-path arc — deploy, four bugs found+fixed+verified, a cloned voice, all in one long session (2026-08-01→02).

The browser-facing half of the #377 Muninn ingestion arc, end to end: mimir-inbox stood up, the write path proven, the read path chased through four defects to a verified-working state, and a character voice cloned into the TTS zoo. Peers: mimir-dev (the app), muninn-dev (gate/watcher spec), worldtree-dev (Worldtree app layer + the #380/#381/#382/#383 fixes), ratatoskr-dev (a consumer + the rigorous verifier).

mimir-inbox deployed (#377)

• New infra-ops stack, canonical eshpfi stacks/mimir-inbox/; live corviduo-dev 10.250.50.152:8091 (co-located w/ muninn-gate :8090 + the worldtree-personal muninn watcher). Full deploy detail + procedures → auto-memory reference_mimir_inbox_deploy.
• Placement decision (operator, reversed): 7-31 he ruled mimir-inbox stays OFF corviduo-dev (shared/NFS mount); 8-01 he REVERSED to CO-LOCATE. Trigger: muninn-dev's code-check showed staging is NOT same-fs-constrained (gate reads staging metadata + passes path strings; os.replace is inside ingestion_root) — staging's real constraint is path-identity across writer/gate/watcher, which co-location buys outright while dodging NFS failure modes. I HELD the reversal for the operator's direct word (data/hosting on a team-managed box, reversing his own ruling) even against 3 peer relays — vindicated as the right instinct; muninn-dev agreed.
• Build: uv sync --no-dev --frozen, SINGLE-STAGE (project installs editable-linked to src/, so src/ MUST stay beside .venv — a multi-stage "copy only .venv" dies at import/404s assets). uid 1000, host-net bind 10.250.50.152:8091, TCP-liveness healthcheck (deliberately NOT gate-coupled). Redeploy = refresh build context (preserve the on-server .env!) → docker build -t mimir-inbox:0.0.1 -t mimir-inbox:<sha> . → compose up -d. Version stays 0.0.1 across dev commits → tag the image w/ the source SHA too. Live commit progression 0478452→c8ab38f→2dcc77e→8ece117 (3 redeploys).
• mimir-inbox key on the gate bumped [read,submit]→[read,submit,control] (cancel/retry); brokered via a 0600 drop on nh3-dev (never on the althing bus).

The read-path bug chain (worldtree-dev's, all found via this arc)

• #380 wing-blind indexing: the book-ingest path upserted concepts into a hardcoded main Chroma collection while wing search reads the fiction collection → P&P written to disk but search_library returned total 0. A silent-success defect ("complete/69 indexed" was right about the WRITE, wrong drawer). Root-caused off MY physical evidence (files on disk + search empty). Fixed b164 + a one-shot --reindex <job_id> (re-upsert into the right wing collection + delete stray main rows).
• #381 stale Chroma client: the personal api opens its Chroma client before the watcher's cross-process writes → a freshly-ingested/re-indexed book is NOT queryable until the api is restarted. Proven by my restart-diagnostic (pre-restart total 0 → post-restart hits, same index). Workaround until fixed: docker restart worldtree-personal-worldtree-api-1 after any ingest/re-index. Filed as #381.
• #382 unreliable Mimir grounding (the subtle one): post-#380-fix the index was correct, but Mimir's grounding was INTERMITTENT — some sessions navigated the opaque job-hash dir (mimir-f3887c9b97b7) to the content, others distrusted the correct vector hits and silently answered from training knowledge (worst of the looks-fine-isn't family). ratatoskr-dev caught it; I'd been over-confident ("Mimir read Austen back to you") having verified the INDEX, not the GROUNDING. Fixed b166 with BOTH shapes: a self-describing _index.md per wing job-dir (resolves the hash dir to its title) + a Mimir prompt rule (wing-scoped hits ARE library content, never discard on a name mismatch, never substitute training). Verified: ratatoskr-dev re-ran 3× fresh sessions → 3/3 grounded, citations in note-extracted language not raw Austen. #382 CLOSED.
• DCC (Dungeon Crawler Carl, job b59c147c5ce0) backfill: --reindex FAILED ("job not found in any state dir" — predates state-tracking). SETTLED = no re-file (the b166 prompt rule already grounds it even without an _index.md; ratatoskr confirmed incidentally); an _index.md rides whenever DCC is next re-ingested.
• #377 mimir-inbox banner bug (mimir-dev's, 8ece117): /health-banner misattributed an unwritable ingestion_root to the WORKER, rendering "The worker is not running." for a running worker — a false lead pointed at infra-ops's half of #377. Fixed (guard split into two banners); I confirmed from the DEPLOYED handler (not just the test) that ingestion_root_writable:False now renders "The ingestion root is not writable."

muninn-gate → muninn-dispatch 0.1.5

Rebuilt muninn-gate off vh/muninn-gate main bc04c4c (dispatch 0.1.4→0.1.5) so the gate serves the new concept_schema/concept_schema_source row fields (computed gate-side). Gate version unchanged 0.0.14 (dual-tag the SHA). Build needs the vh gitea token as a BuildKit secret (--secret id=gitea_pw, UV_INDEX_GITEA_USERNAME=vh, drop+shred). Recreate with compose up -d (NOT bare restart — needs the new image). Verified: P&P job serves concept_schema='fiction', concept_schema_source=null (null correct — pre-b164 job). Registry tags by commit SHA — v1.0.0bNNN docker tags don't resolve; use the deployed SHA (confirm --reindex present before using an image for a data-op).

donut voice (65-frost → Zonos gateway)

Operator: "pick up 65-frost, use that bundle as a voice for a character named donut." 65-frost = a Booth id (~/booth-data/65-frost/) holding a curated yt-voice-clipper dataset (dataset-…-curated.zip: 4 clips + manifest, all SPEAKER_02 = Princess Donut). Zonos gateway voice registry = a filesystem drop: <Name>.wav in the voices dir (44.1kHz mono s16 PCM) auto-registers as voice:"<name>" on startup (needs a restart). The LIVE dir is the bind mount /opt/docker/compose/zonos-gateway/voices/ (lkraven-writable), NOT the working tree. Built Donut.wav from seg000 (best clip), dropped it, restarted → voice:"donut" live in the gateway AND the Asset Engine's make form. Also copied to the build-source tree ~/zonos-gateway/voices/ for rebuild-durability (true canonical = the gitea repo, not yet CI-wired). Auditioned in booth donut-voice. Expanded 2026-08-02 (onyx-58 bundle): operator curated a 2nd Booth bundle onyx-58 (dataset-467d2cf8…curated.zip, 3 Donut clips) as additions. Rebuilt the reference = seg000 (65-frost) + seg101/seg110/seg148 (onyx-58) ffmpeg-concat + resampled 24k→44.1k mono s16 = 52.0s. seg148 was diarized SPEAKER_03 but is Donut (operator-confirmed misdiarize → included). Assembly is NOT assemble_voice.py (that -c copy can't resample + caps ~15s); used a manual aresample=44100,aformat=…,concat=n=4 filter. Backed up old ref → irv-ml1:~/Donut.wav.pre-onyx58; dropped to live bind-mount + build-source tree; docker compose restart (healthy 2s, voice:"donut" still 1 of 9). A/B booth donut-onyx58 (A=old 16.3s ref, B=new 52s ref, same line). Longer ref is fine mechanically: gateway passes it as speaker_audio_base64 → speaker embedding, not an audio prefix. BUT auditioned → REVERTED same day: pinned-seed neutral A/B (5 pairs, booth donut-onyx58) showed the single-clip seg000 (16.3s) beats the 52s 4-take concat on timbre — concatenating disparate takes muddied the embedding more than the range helped. Reverted both live + build-source to seg000-alone. Lessons (→ Tried-and-abandoned): more reference ≠ better when takes vary; and emotion steering pulls output away from the clone fast (operator craft rule) — keep clones emotion-neutral; bare {input,voice} calls send NO emotion (gateway only enables it on an explicit emotion_*/preset dial).

Zonos streaming (no gateway change needed)

ratatoskr wanted play-as-it-arrives. /v1/audio/speech ALREADY streams — chunked StreamingResponse, opens native /tts/generate with stream=True, wraps as a streaming int16 WAV with 0xFFFFFFFF placeholder sizes (meant for progressive <audio>). Verified TTFB 0.44s vs 6.84s total, transfer-encoding: chunked, dials preserved. ratatoskr's proxy was rewriting the placeholder header → forced buffering. Fix was theirs (pass chunks through); shipped + confirmed (TTFB 0.46s progressive). The Asset Engine (ana-docker:8200) IS the fleet "TTS zoo" (~20 audio svcs w/ irv-ml1 endpoints); zonos-gateway registered there, state=ready.

Lessons (also in Tried-and-abandoned)

• Verifying the INDEX (search returns hits) is NOT verifying GROUNDING (does the agent trust+use them vs. silently answer from training). Check that citations are note-extracted, not model-knowledge. ratatoskr caught this after my over-confident "it works."
• Reading the DEPLOYED artifact > trusting the test for "is the fix live" — the test proves the source is right; reading the running code proves the artifact is, which is what an on-call actually meets.
• Held a boundary-box/data reversal for the operator's DIRECT word against 3 peer relays — the right call (peer relay ≠ operator consent; the placement guard was vindicated).

See also: 2026-07-31-muninn-gate-deploy. auto-memory: reference_mimir_inbox_deploy, reference_muninn_gate_deploy, reference_muninn_gate_staging_path, reference_zonos_tts_stack, reference_infra_ops_vh_gitea_token_and_sdk_publish.