From 1ab5465293e647b85b5e758e18f45f40cd011eb1 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 2 Aug 2026 07:33:49 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20mimir-inbox/#3?= =?UTF-8?q?77=20read-path=20arc=20(deploy=20+=20#380/#381/#382=20+=20donut?= =?UTF-8?q?=20voice);=20archived=2011=20old=20entries?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- archival-memory.md | 167 +++++++++++++++++- ...onal-worldtree-ci-deploy-that-fails-85s.md | 4 - ...-systemd-user-daemon-that-shells-out-to.md | 4 - ...-04-on-prem-t1-train-that-keeps-any-ana.md | 4 - ...mir-deploy-blocker-resolved-mid-session.md | 4 - ...-breaks-morpheus-audio-token-generation.md | 4 - ...gen-memory-extractor-bind-green-lit-for.md | 7 - ...en-memory-extractor-bind-host-synced-on.md | 46 ----- ...sper-datasets-aren-t-actually-whispered.md | 4 - ...ts-off-the-shelf-voice-pipeline-shipped.md | 10 -- ...-09-sentence-chunking-tts-loses-prosody.md | 3 - ...parked-items-closed-phantom-qwen3-6-35b.md | 16 -- .../2026-08-02-mimir-inbox-arc.md | 32 ++++ persistent-memory.md | 48 ++--- 14 files changed, 213 insertions(+), 140 deletions(-) delete mode 100644 persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md delete mode 100644 persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md delete mode 100644 persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md delete mode 100644 persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md delete mode 100644 persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md delete mode 100644 persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md delete mode 100644 persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md delete mode 100644 persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md delete mode 100644 persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md delete mode 100644 persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md delete mode 100644 persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md create mode 100644 persistent-memory.d/2026-08-02-mimir-inbox-arc.md diff --git a/archival-memory.md b/archival-memory.md index c20ead6..9846954 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -1316,4 +1316,169 @@ _76 older entries archived to archival-memory.md._ - `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) — but `ssh infra-ops@ana-docker` HAS NOPASSWD root.** A `sudo cp` as lkraven silently failed → nearly punted the rest-server recovery. Reach for `infra-ops@ana-docker` for sudo ops. - _Archived 2026-07-08._ \ No newline at end of file + _Archived 2026-07-08._ +## Archived 2026-08-02 — Recent decisions (archived) + +### 2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session + +- `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` + + `envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal), + YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my + parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.) + +_Archived 2026-08-02._ + +### 2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b + +- `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker + docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB + registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the + 2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to + remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because + daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never + reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd + opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`, + `rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the + frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime + (zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares + `live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start, + never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers + once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon; + log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running + containers). + +_Archived 2026-08-02._ + +### 2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped + +- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into + gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17, + vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack + (01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829); + max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and + `repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture + no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone + voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514). + **Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not + A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s). + +_Archived 2026-08-02._ + +### 2026-07-09-granite-gen-memory-extractor-bind-host-synced-on + +- `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed, + #335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer`→`gen` (overrides + intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on + `summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit + both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen** + (b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted + `worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL + NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1` + after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config + INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven + safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed). + StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON: + the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never + updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) — + pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass` + parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider + description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix + (operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING + defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs + when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway + config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD; + live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09): + character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving + character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping + principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the + dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent + but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44, + UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly + from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) + + mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's + `load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character + + character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window + VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev + to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked + config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly + re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard + + character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning + downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying + (both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37** + (char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09): + mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx + 131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45 + canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push + of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity + block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev), + monitor armed. See [[reference_corviduo_dev_emergency_ops]]. + +_Archived 2026-08-02._ + +### 2026-07-09-granite-gen-memory-extractor-bind-green-lit-for + +- `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).** + Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty + idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching). + Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08). + **This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the + operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread + 01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]]. + +_Archived 2026-08-02._ + +## Archived 2026-08-02 — Tried and abandoned (archived) + +### 2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s + +- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" + is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but + fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push + finishes. FIX: re-run once built, or gate on `workflow_run: completed`. + +_Archived 2026-08-02._ + +### 2026-07-04-on-prem-t1-train-that-keeps-any-ana + +- `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe + ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast + on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the + 566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`. + +_Archived 2026-08-02._ + +### 2026-07-04-a-systemd-user-daemon-that-shells-out-to + +- `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools + needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The + althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO + routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`. + +_Archived 2026-08-02._ + +### 2026-07-09-fp8-breaks-morpheus-audio-token-generation + +- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC + frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks- + TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the + bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real. + +_Archived 2026-08-02._ + +### 2026-07-09-sentence-chunking-tts-loses-prosody + +- `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that + spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning + (penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path). + +_Archived 2026-08-02._ + +### 2026-07-09-hf-whisper-datasets-aren-t-actually-whispered + +- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8 + (not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the + cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) — + that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting. + +_Archived 2026-08-02._ + diff --git a/persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md b/persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md deleted file mode 100644 index aaffaaa..0000000 --- a/persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md +++ /dev/null @@ -1,4 +0,0 @@ -- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" - is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but - fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push - finishes. FIX: re-run once built, or gate on `workflow_run: completed`. diff --git a/persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md b/persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md deleted file mode 100644 index f391305..0000000 --- a/persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md +++ /dev/null @@ -1,4 +0,0 @@ -- `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools - needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The - althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO - routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`. diff --git a/persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md b/persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md deleted file mode 100644 index c5a2fcb..0000000 --- a/persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md +++ /dev/null @@ -1,4 +0,0 @@ -- `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe - ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast - on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the - 566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`. diff --git a/persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md b/persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md deleted file mode 100644 index ddd10c6..0000000 --- a/persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md +++ /dev/null @@ -1,4 +0,0 @@ -- `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` + - `envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal), - YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my - parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.) diff --git a/persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md b/persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md deleted file mode 100644 index 4ed80d1..0000000 --- a/persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md +++ /dev/null @@ -1,4 +0,0 @@ -- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC - frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks- - TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the - bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real. diff --git a/persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md b/persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md deleted file mode 100644 index c7fae06..0000000 --- a/persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md +++ /dev/null @@ -1,7 +0,0 @@ -- `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).** - Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty - idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching). - Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08). - **This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the - operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread - 01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]]. diff --git a/persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md b/persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md deleted file mode 100644 index f000c4e..0000000 --- a/persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md +++ /dev/null @@ -1,46 +0,0 @@ -- `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed, - #335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer`→`gen` (overrides - intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on - `summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit - both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen** - (b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted - `worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL - NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1` - after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config - INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven - safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed). - StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON: - the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never - updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) — - pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass` - parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider - description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix - (operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING - defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs - when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway - config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD; - live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09): - character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving - character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping - principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the - dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent - but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44, - UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly - from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) + - mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's - `load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character + - character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window - VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev - to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked - config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly - re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard - + character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning - downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying - (both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37** - (char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09): - mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx - 131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45 - canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push - of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity - block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev), - monitor armed. See [[reference_corviduo_dev_emergency_ops]]. diff --git a/persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md b/persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md deleted file mode 100644 index 4c546ac..0000000 --- a/persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md +++ /dev/null @@ -1,4 +0,0 @@ -- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8 - (not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the - cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) — - that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting. diff --git a/persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md b/persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md deleted file mode 100644 index 7450511..0000000 --- a/persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md +++ /dev/null @@ -1,10 +0,0 @@ -- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into - gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17, - vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack - (01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829); - max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and - `repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture - no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone - voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514). - **Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not - A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s). diff --git a/persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md b/persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md deleted file mode 100644 index 0dec87f..0000000 --- a/persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md +++ /dev/null @@ -1,3 +0,0 @@ -- `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that - spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning - (penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path). diff --git a/persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md b/persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md deleted file mode 100644 index ac18f7f..0000000 --- a/persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md +++ /dev/null @@ -1,16 +0,0 @@ -- `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker - docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB - registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the - 2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to - remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because - daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never - reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd - opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`, - `rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the - frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime - (zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares - `live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start, - never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers - once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon; - log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running - containers). diff --git a/persistent-memory.d/2026-08-02-mimir-inbox-arc.md b/persistent-memory.d/2026-08-02-mimir-inbox-arc.md new file mode 100644 index 0000000..df961ef --- /dev/null +++ b/persistent-memory.d/2026-08-02-mimir-inbox-arc.md @@ -0,0 +1,32 @@ +`[2026-08-02]` **The mimir-inbox / #377-read-path arc — deploy, four bugs found+fixed+verified, a cloned voice, all in one long session (2026-08-01→02).** + +The browser-facing half of the #377 Muninn ingestion arc, end to end: mimir-inbox stood up, the write path proven, the read path chased through four defects to a verified-working state, and a character voice cloned into the TTS zoo. Peers: mimir-dev (the app), muninn-dev (gate/watcher spec), worldtree-dev (Worldtree app layer + the #380/#381/#382/#383 fixes), ratatoskr-dev (a consumer + the rigorous verifier). + +## mimir-inbox deployed (#377) +- **New infra-ops stack, canonical eshpfi `stacks/mimir-inbox/`; live corviduo-dev `10.250.50.152:8091`** (co-located w/ muninn-gate :8090 + the worldtree-personal muninn watcher). Full deploy detail + procedures → auto-memory `reference_mimir_inbox_deploy`. +- **Placement decision (operator, reversed):** 7-31 he ruled mimir-inbox stays OFF corviduo-dev (shared/NFS mount); 8-01 he REVERSED to CO-LOCATE. Trigger: muninn-dev's code-check showed staging is NOT same-fs-constrained (gate reads staging metadata + passes path strings; `os.replace` is inside `ingestion_root`) — staging's real constraint is **path-identity across writer/gate/watcher**, which co-location buys outright while dodging NFS failure modes. I HELD the reversal for the operator's direct word (data/hosting on a team-managed box, reversing his own ruling) even against 3 peer relays — vindicated as the right instinct; muninn-dev agreed. +- Build: **`uv sync --no-dev --frozen`, SINGLE-STAGE** (project installs editable-linked to `src/`, so src/ MUST stay beside .venv — a multi-stage "copy only .venv" dies at import/404s assets). uid 1000, host-net bind 10.250.50.152:8091, TCP-liveness healthcheck (deliberately NOT gate-coupled). Redeploy = refresh build context (**preserve the on-server `.env`!**) → `docker build -t mimir-inbox:0.0.1 -t mimir-inbox: .` → `compose up -d`. Version stays 0.0.1 across dev commits → tag the image w/ the source SHA too. Live commit progression `0478452`→`c8ab38f`→`2dcc77e`→**`8ece117`** (3 redeploys). +- mimir-inbox key on the gate bumped [read,submit]→**[read,submit,control]** (cancel/retry); brokered via a 0600 drop on nh3-dev (never on the althing bus). + +## The read-path bug chain (worldtree-dev's, all found via this arc) +- **#380 wing-blind indexing:** the book-ingest path upserted concepts into a hardcoded `main` Chroma collection while wing search reads the `fiction` collection → P&P written to disk but `search_library` returned total 0. A silent-success defect ("complete/69 indexed" was right about the WRITE, wrong drawer). Root-caused off MY physical evidence (files on disk + search empty). Fixed b164 + a one-shot `--reindex ` (re-upsert into the right wing collection + delete stray `main` rows). +- **#381 stale Chroma client:** the personal api opens its Chroma client before the watcher's cross-process writes → **a freshly-ingested/re-indexed book is NOT queryable until the api is restarted.** Proven by my restart-diagnostic (pre-restart total 0 → post-restart hits, same index). Workaround until fixed: `docker restart worldtree-personal-worldtree-api-1` after any ingest/re-index. Filed as #381. +- **#382 unreliable Mimir grounding (the subtle one):** post-#380-fix the index was correct, but Mimir's grounding was INTERMITTENT — some sessions navigated the opaque job-hash dir (`mimir-f3887c9b97b7`) to the content, others distrusted the correct vector hits and **silently answered from training knowledge** (worst of the looks-fine-isn't family). ratatoskr-dev caught it; I'd been over-confident ("Mimir read Austen back to you") having verified the INDEX, not the GROUNDING. Fixed b166 with BOTH shapes: a self-describing `_index.md` per wing job-dir (resolves the hash dir to its title) + a Mimir prompt rule (wing-scoped hits ARE library content, never discard on a name mismatch, never substitute training). **Verified: ratatoskr-dev re-ran 3× fresh sessions → 3/3 grounded**, citations in note-extracted language not raw Austen. #382 CLOSED. +- **DCC (Dungeon Crawler Carl, job `b59c147c5ce0`) backfill:** `--reindex` FAILED ("job not found in any state dir" — predates state-tracking). SETTLED = **no re-file** (the b166 prompt rule already grounds it even without an `_index.md`; ratatoskr confirmed incidentally); an `_index.md` rides whenever DCC is next re-ingested. +- **#377 mimir-inbox banner bug (mimir-dev's, `8ece117`):** `/health-banner` misattributed an unwritable `ingestion_root` to the WORKER, rendering "The worker is not running." for a running worker — a false lead pointed at infra-ops's half of #377. Fixed (guard split into two banners); I confirmed from the DEPLOYED handler (not just the test) that `ingestion_root_writable:False` now renders "The ingestion root is not writable." + +## muninn-gate → muninn-dispatch 0.1.5 +Rebuilt `muninn-gate` off `vh/muninn-gate` main `bc04c4c` (dispatch 0.1.4→0.1.5) so the gate serves the new `concept_schema`/`concept_schema_source` row fields (computed gate-side). Gate version unchanged 0.0.14 (dual-tag the SHA). Build needs the vh gitea token as a BuildKit secret (`--secret id=gitea_pw`, UV_INDEX_GITEA_USERNAME=vh, drop+shred). Recreate with `compose up -d` (NOT bare restart — needs the new image). Verified: P&P job serves `concept_schema='fiction'`, `concept_schema_source=null` (null correct — pre-b164 job). Registry tags by commit SHA — `v1.0.0bNNN` docker tags don't resolve; use the deployed SHA (confirm `--reindex` present before using an image for a data-op). + +## donut voice (65-frost → Zonos gateway) +Operator: "pick up 65-frost, use that bundle as a voice for a character named donut." 65-frost = a **Booth id** (`~/booth-data/65-frost/`) holding a curated yt-voice-clipper dataset (`dataset-…-curated.zip`: 4 clips + manifest, all SPEAKER_02 = Princess Donut). **Zonos gateway voice registry = a filesystem drop:** `.wav` in the voices dir (44.1kHz mono s16 PCM) auto-registers as `voice:""` on **startup** (needs a restart). The LIVE dir is the bind mount `/opt/docker/compose/zonos-gateway/voices/` (lkraven-writable), NOT the working tree. Built `Donut.wav` from seg000 (best clip), dropped it, restarted → `voice:"donut"` live in the gateway AND the Asset Engine's make form. Also copied to the build-source tree `~/zonos-gateway/voices/` for rebuild-durability (true canonical = the gitea repo, not yet CI-wired). Auditioned in booth `donut-voice`. + +## Zonos streaming (no gateway change needed) +ratatoskr wanted play-as-it-arrives. `/v1/audio/speech` ALREADY streams — chunked `StreamingResponse`, opens native `/tts/generate` with `stream=True`, wraps as a streaming int16 WAV with `0xFFFFFFFF` placeholder sizes (meant for progressive `