diff --git a/archival-memory.md b/archival-memory.md index 29bc503..6da9acf 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -295,6 +295,28 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re `01KT2K2SY9N7AY69R9V0B4RXSW` to asset-engine-dev. _Archived 2026-06-04._ +- `[2026-06-01]` **Fish reference_id empty-dir fix shipped** (`c5bbb90`) — Populated + `references//.wav`+`.lab` for all 32 voices; playbook gained + normalize-step + A/B smoke gate. glados got a real transcript (ASR'd via + Parakeet): the Portal "Welcome to test chamber 4" lines. + _Archived 2026-06-11._ + +- `[2026-06-02]` **Chatterbox → main TTS engine; build custom `chatterbox-fast` + streaming container.** Workload = single-stream interactive. GPU placement: + 3090 (device 0) if it fits else A6000 (device 1). Cutover: parallel catalog + entry, burn in beside live `chatterbox`, then flip. Streaming = adaptive + buffer-ratchet chunking (native frame-streaming abandoned). Since extracted to + its own repo `vh/chatterbox-fast` (auto-memory `reference_chatterbox_fast_repo`). + _Archived 2026-06-11._ + +- `[2026-06-02]` **Sentence-splitting loses quality (operator-corrected).** Claimed + naive sentence-level streaming has "zero quality loss" — WRONG. The T3 AR backbone + conditions prosody on the WHOLE text; splitting loses cross-sentence prosodic + context even though voice timbre stays (reference-conditioned). No *artifacts* ≠ + no *quality loss*. Hence adaptive-chunk design (maximize context per chunk subject + to latency budget), not fixed per-sentence splits. + _Archived 2026-06-11._ + ## Tried and abandoned (archived) - `[2026-04-30]` task-board workflow with @@ -629,3 +651,33 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re same clips gave the correct answer (clone 0.79 to real Imogen, 0.10 to default). Use ECAPA, not resemblyzer, for speaker-verification. _Archived 2026-06-04._ + +- `[2026-06-02]` **Fish (fish-s2 / OpenAudio S1-mini) progressive streaming — SHELVED + (sub-realtime).** Benched RTF on A6000: 0.72x/0.82x/0.86x, mean 0.80x = sub-realtime, + so client-side chunking starves. Root cause: Fish only chunks on `<|speaker:X|>` tags; + plain text → batches=[whole text] → all semantic tokens generate before any audio + (chunk_length inert). Plus kui/ASGI StreamResponse doesn't flush (fix = anti-buffering + headers, kept on file). STANDING REVISIT TRIGGER: when an RTX Blackwell Pro lands → bench + fp4-quantized Fish; if RTF > ~1.5x give it the chatterbox-fast treatment. For now Fish is + a buffered catalog entry (great for SAVED gens, not live-audition). + _Archived 2026-06-11._ + +- `[2026-06-02]` **Context-priming at chunk joins (chatterbox-fast §1.6) — ABANDONED + (discard-cut leaks the prefix).** Prepend prior sentence, generate `prefix+content`, + discard prefix audio. Built + shipped (`d707439`), live-A/B'd, reverted (`090e70a`). + Kill: `generate()` returns one waveform with NO marker where the prefix ends, and the + model renders the same prefix with different timing solo vs followed-by-content — so + locating the cut is a guess that left a whole clause in the output (operator caught + "...without a trace of sarcasm" spoken twice). Reliable cut needs token-level boundaries + or per-chunk forced-alignment (heavy). Coherence loss at joins = ACCEPTED limitation. + _Archived 2026-06-11._ + +- `[2026-06-02]` **Native frame-level streaming on Chatterbox-TURBO — ABANDONED (turbo + isn't built for streaming).** (1) `S3GenStreamer` is docstring-only (unimplemented). + (2) `flow_inference(finalize=False)` is buggy (lookahead trim removes frames from `h` + not `h_masks`/conds → shape mismatch); a 1-line patch makes it RUN. (3) BUT the flow + encoder uses full-context attention (`static_chunk_size=0`) → incremental decode is + prefix-unstable (adding tokens re-attends, shifts earlier mel ~0.30-0.39 vs one-shot). + (4) Forcing `static_chunk_size>0` didn't stabilize it. True sub-second frame-streaming + needs deep attention surgery with quality risk — not worth it. → adaptive-chunking instead. + _Archived 2026-06-11._ diff --git a/persistent-memory.md b/persistent-memory.md index 6224f71..d48ac0e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-06-05_ +_Last updated: 2026-06-11_ ## Repo purpose @@ -24,9 +24,12 @@ Sister repos (separate gitea repos, deployed by playbooks here): | `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) | | `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) | | `vh/worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration | push-to-main → CI deploys | -| `vh/volva` | Codex peer agent on althing bus (single-turn oracle, systemd daemon on nh3-dev) | manual install via `deploy/volva.service` (2026-05-18) | | `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` | +(`vh/volva` + Heid were re-architected from systemd daemons to Claude Code +session orchestrators 2026-06-08; their nh3-dev `.service` units were removed — +no longer deployed sidecars here. See Recent decisions.) + - **Two-layer backups** — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + @@ -88,29 +91,85 @@ Sister repos (separate gitea repos, deployed by playbooks here): ## Current state / in-flight -_As of 2026-06-05:_ +_As of 2026-06-11:_ -- **Granite-FP8 + observability session — all LIVE & committed (`34a43a0`, `9171e6a`).** - - **Granite 4.1 8B FP8 is the production summarizer** (`vllm-granite` :8004, ana-ml2 GPU 1, - 50K ctx, CUDA graphs) — replaced phi4-mini, validated by brokkr (valid_format 1.0, FP8 stays). - - **LiteLLM gateway** (:4000) routes `granite-4.1-8b`→vLLM (explicit entry shadows the `*` - wildcard) + **Langfuse v3 wired** (ana-docker:3001, "LLM Throughput (tok/s)" dashboard built). - - **GPU-1 retuned** (trio over-provisioned KV trimmed) → granite runs with CUDA graphs + ~10 GB - free as a future Granite-text-LoRA hedge. Streaming through the gateway confirmed (TTFT 0.24s). - - **ana-docker pruned** 77 GB (unused images + build cache; disk 83%→49%) to fit ClickHouse. -- **Worldtree summarizer repoint — NO instance change now; DEFERRED to Worldtree #254** (see Recent - decisions). worldtree-dev will ping with the providers.yaml + consumer config when #254 un-holds; - infra-ops applies to the personal/demo/pinned bind mounts (vh@10.250.50.152, `/opt/worldtree*/config`). -- **Commits unpushed** (`34a43a0`, `9171e6a`, nevermore `d3e19b8` in its repo) — operator's call to push. -- **Operator flagged "new work to do"** for the next session — this snapshot is the handoff. -- **Disclosed-keys hygiene queue** (rotate at convenience): HF token `hf_HBl…` (lkraven's), `/tmp/ - wt-personal-skaldsong-prod.key`, Worldtree `Z_AI_API_KEY`, Gitea runner reg token, `MINIFLUX_PASSWORD` - (leaked twice). (sk-corvid + the langfuse/vastblueai-gateway keys are dev-enclosed — leakage deprioritized.) -- **Still open from prior:** clean legacy `news-digest` on ana-docker; watch nh3-nas `/volume1`; **pin - llama-swap to GPU 0** for clean GPU-1 separation; the `docker push 60s ceiling` mystery uninstrumented. +- **Mac Pro migration is the big open project** — `migration-plan.md` (repo root) frames a + WORKSTATION-ONLY migration of nh3-dev's dev env to an M2 Ultra Mac Pro Rack, racked in NH3 on + the 10.100 subnet. nh3-dev wears two hats; only **Hat 1** (the ~40 repos, all of `~/.claude`, + dotfiles, toolchain) moves. **Hat 2** fleet sidecars (egress SOCKS5 proxy, mead-hall, ttyd + seats, `althing-forseti`) STAY on the Linux VM. **Phase 0 is the only time-sensitive step**: + push the pushable repos + confirm which of ~6 no-upstream repos are local-only. Cutover = + `rsync` the working trees, **NOT re-clone** (re-clone would lose unsaved work in ~15 repos). + macOS gotcha pre-flagged: `/home/lkraven` → `/Users/lkraven` repath. Everything else waits on + the hardware landing in the rack. + +- **Two commits/branches committed-not-pushed** (operator's push call): `95b2701` (GLM thinking + config) on main; and the R17 v2 results branch `r17-v2-characterization` on the irv-ml1 + checkout (push HELD — see decisions). + +- **R17 v2 corpus characterization DONE on irv-ml1** (brokkr, ~8.5h): 119 clips → **18 MULTI / + 101 ELIGIBLE** (Stage-0 drop list) + 19936 candidates + ASR speech + blend verdicts. Outputs + on the local branch; **push HELD** — no-push rule AND the `speech.json` are ASR transcripts of + the intimate-audio batch (content-exposure call). Awaiting brokkr-collect or explicit push approval. + +- **pi coding agent + GLM 5.1 harness live on nh3-dev** — `glm` launcher runs pi against `glm-5.1` + (thinking-off) via the LiteLLM gateway; `glm-5.1-reasoning` for opt-in thinking. + +- **Worldtree config-propagation lane is mature + humming** — pre-merge delta-ping → sync-on-merge. + All of v0.33.x→v0.34.1 reconciled to the demo+personal bind-mounts; demo/personal both on v0.34.x, + pinned stays pre-cutover (full migration at its next re-image). + +- **Disclosed-keys hygiene queue** carries forward (rotate at convenience): HF token, the wt-personal + keys, Gitea runner reg token, MINIFLUX_PASSWORD. `ZAI_API_KEY` (z.ai MCP web-tools key) now also + lives in `~/.config/secrets/env.sh` (shared from Worldtree env for the Brokkr panel). + +- **Still open from prior:** clean legacy `news-digest` on ana-docker; pin llama-swap to GPU 0; the + `docker push 60s ceiling` mystery uninstrumented. ## Recent decisions +- `[2026-06-11]` **GLM thinking inverted at the LiteLLM gateway** (operator call): `glm-5.1` defaults + thinking-OFF; new `glm-5.1-reasoning` alias = same z.ai upstream with thinking ON (opt-in). + Mechanism: `litellm_params.extra_body:{thinking:{type:disabled}}` — LiteLLM `drop_params` STRIPS a + top-level `thinking`/`reasoning_effort`, but forwards `extra_body` verbatim to z.ai (the only channel + that works; verified reasoning_tokens 0 vs >0). Shared-gateway change — affects ALL glm-5.1 callers + (brokkr's all-models key included). (`95b2701`, auto-memory `reference_litellm_gateway`) + +- `[2026-06-11]` **pi coding agent installed on nh3-dev as a GLM 5.1 harness** — `@earendil-works/ + pi-coding-agent` via **bun** (user-level; npm's global prefix is `/usr` → needs sudo, bun avoids it). + Config `~/.pi/agent/models.json` (litellm provider → gateway), launcher `~/.local/bin/glm` sources + the gateway key + selects the model. pi is OpenAI-compatible; proxy-safe compat flags for the GLM path. + +- `[2026-06-11]` **z.ai web-tools (regin) = z.ai hosted MCP path, NOT the `/paas/v4` Tool API.** Direct + `api.z.ai/api/paas/v4/web_search` → 429/1113 "insufficient balance" (coding-plan keys bill tools on a + separate quota path); `open.bigmodel.cn` is the China platform (404, different account). WORKS: MCP + streamable-HTTP at `https://api.z.ai/api/mcp/{web_search_prime,web_reader}/mcp`, `Authorization: Bearer + $ZAI_API_KEY` (the **MCP** key — distinct from `Z_AI_API_KEY` the LLM key). Reference impl = Worldtree's + Leif agent (`tools/web/zai_client.py` + `core/clients/mcp.py`). + +- `[2026-06-10]` **Mac Pro migration framed: workstation-only** (M2 Ultra ARM, racked NH3 on-subnet); + sidecars stay Linux. `migration-plan.md`. (See in-flight.) + +- `[2026-06-10]` **Worldtree deployed-config propagation is infra-ops's OWNED lane** (operator ruling). + worldtree-dev pings the config delta on every config-touching commit (pre-merge); infra-ops syncs + `config/*.yaml` from MERGED canonical to the `/opt/worldtree*/config` bind-mounts on demo+personal + (Heimdall hot-reloads policies; defaults are code-defaulted). The **v0.33.8 9-HOUR demo outage** — a + `model_roles.yaml` hard-startup-dep that shipped in canonical 3 releases earlier but never reached the + VM — is the failure mode this lane prevents. providers.yaml stays hand-tuned (artemis graft on + personal). corviduo-dev emergency-ops = `ssh vh@10.250.50.152` (alias doesn't resolve, lkraven denied, + infra-ops key excluded), docker no-sudo. (auto-memory `reference_worldtree_deploys_cicd`, + `reference_corviduo_dev_emergency_ops`) + +- `[2026-06-09]` **LiteLLM scoped virtual keys issued to consumers** (operator-authorized): `brokkr- + smithy` (all-proxy-models), `arbo-prompt-enhance` (comfy-dev, granite-only). Pattern: mint via + `/key/generate` (master `sk-corvid`), scope-restricted + rotatable, value → 600 file never the bus. + (auto-memory `reference_litellm_gateway`) + +- `[2026-06-08]` **volva.service + heid.service removed from nh3-dev** — vestigial systemd daemons; + Heid/Volva re-architected from Python pollers to Claude Code session orchestrators (heid commit + `12aa5a9`); binaries+dirs gone, volva.service was crash-looping 203/EXEC. Cleanup at heid's request + (the bus-content-driven sudo got the harness guardrail; operator green-lit). (`6e2f80e`) + - `[2026-06-05]` **Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer** (supersedes the 2026-06-04 phi4 decision below). Beat phi4 on precision in brokkr's R15 P03. **Staying FP8, not Q4/AWQ** — primary workload (agent memory + summarization) is high-concurrency, where FP8-on-Ada @@ -190,32 +249,22 @@ _As of 2026-06-05:_ CI/CD lane (a developer `staging/vX.Y.Z` git-tag promote), not infra-ops. They self-corrected to the same conclusion independently. -- `[2026-06-02]` **Chatterbox → main TTS engine; build custom `chatterbox-fast` - streaming container.** Workload = single-stream interactive. **GPU placement: - 3090 (device 0) if it fits else A6000 (device 1)** — shared dev stack, 20.5 GB - 3090-idle is expected residency, not a blocker. **Cutover: parallel catalog - entry**, burn in beside live `chatterbox`, then flip. **Streaming approach: - adaptive buffer-ratchet chunking** (see in-flight). Native frame-streaming - abandoned (Tried/abandoned). Tracked: `docs/design/chatterbox-fast-plan.md`. - -- `[2026-06-02]` **Sentence-splitting loses quality (operator-corrected).** I - claimed naive sentence-level streaming has "zero quality loss" — WRONG. The - T3 AR backbone conditions prosody on the WHOLE text; splitting loses - cross-sentence prosodic context (contextual delivery, declination, affect - continuity) even though voice timbre stays (reference-conditioned). No - *artifacts* ≠ no *quality loss*. Hence the adaptive-chunk design (maximize - context per chunk subject to latency budget), not fixed per-sentence splits. - -- `[2026-06-01]` **Fish reference_id empty-dir fix shipped** (`c5bbb90`) — see - in-flight + Tried/abandoned. Populated `references//.wav`+`.lab` - for all 32 voices; playbook gained normalize-step + A/B smoke gate. glados got - a real transcript (ASR'd via Parakeet): the Portal "Welcome to test chamber 4" - lines. - -_37 older entries archived to archival-memory.md._ +_40 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-06-11]` **A completion-poll `while pgrep -f ` SELF-MATCHES its own remote shell + argv** — the poll's command line contains the script name, so its own `pgrep -f` always finds itself + → the loop never exits, the poll never fires. Use a match pattern ABSENT from the poll command (pgrep + the python stage, or a sentinel file), not the driver's own name. (Caught only because the operator + asked "check status"; the job had already finished cleanly.) + +- `[2026-06-08]` **Demucs `uv pip install demucs` pulls torch 2.12/torchaudio 2.11 → `ta.save()` + requires torchcodec → dies AFTER separating (0 stems written, rc=1).** Same class as the torch-2.12 + torchcodec foot-gun below. Fix = pin `torch==torchaudio==2.4.1` (pre-torchcodec save backend) + + `UV_LINK_MODE=copy` for the EPERM-hardlink quirk. Lesson restated: validate the SAVE path, not just + import + GPU inference, on a bleeding-edge torch. + - `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — it fills the KV cache to the `--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs AFTER weights+KV load (model/KV log looks healthy, then crash-loops; saw 11 restarts at util 0.36 @@ -291,43 +340,4 @@ _37 older entries archived to archival-memory.md._ — it's analysis scripts + a poetry pyproject, no buildable package (setuptools package-discovery error). Install the pyproject deps directly, don't build the "package". -- `[2026-06-02]` **Fish (fish-s2 / OpenAudio S1-mini) progressive streaming — SHELVED (sub-realtime).** Benched RTF on A6000: 0.72x (12w) / 0.82x (30w) / 0.86x (60w), **mean 0.80x = sub-realtime**, so client-side chunking would starve (same reason chatterbox-fast needs turbo's RTF>1). Root cause of the buffering (dvalin-smithy-dev deep research, verified in our code text2semantic/inference.py L600-607): Fish only chunks on `<|speaker:X|>` tags; **plain text -> batches=[whole text]** -> all semantic tokens generate before any audio (chunk_length inert). Plus a 2nd layer: kui/ASGI StreamResponse doesn't flush (header produced t=1s, delivered t=23s) -> fix = anti-buffering headers (X-Accel-Buffering:no / Transfer-Encoding:chunked) in tools/server/views.py (kept on file, not applied). A rebuild does NOT fix this (current main same logic). **STANDING REVISIT TRIGGER: when an RTX Blackwell Pro lands in the fleet -> bench fp4-quantized Fish; if RTF > ~1.5x, give it the chatterbox-fast treatment** (client-side adaptive buffer-ratchet chunker driving /v1/tts with small text pieces). Projection: fp4 (~1/4 weight bytes, memory-bound AR decode) + Blackwell (GDDR7 ~1.8TB/s vs A6000 0.77TB/s, native FP4 cores) ~ 2-3x RTF; validate fp4 voice quality (ear/ECAPA) before committing. For now Fish stays a buffered catalog entry (great for SAVED gens, not the live-audition lane). - - -- `[2026-06-02]` **Context-priming at chunk joins (chatterbox-fast §1.6) — - ABANDONED (discard-cut leaks the prefix).** To give a chunk backward prosodic - context, prepend the prior sentence, generate `prefix+content` together, then - discard the prefix audio. Built + opt-in shipped (commit d707439), live-A/B'd, - reverted (090e70a). The kill: `generate()` returns one finished waveform with - NO marker for where the prefix ends, and the model renders the same prefix with - different timing solo vs followed-by-content — so locating the cut (generate - prefix solo → measure duration → snap to nearest energy-min pause within ±0.4s) - is a guess that left a whole clause of prefix in the output ("...without a trace - of sarcasm," spoken twice; operator caught it). A reliable cut needs token-level - boundaries (= the abandoned native-streaming arc) or per-chunk ASR/forced- - alignment (heavy, imperfect, eats the latency budget). → Coherence loss at joins - stays an ACCEPTED limitation; cold adaptive-chunk streaming judged "really good". - Scheduler-side work that DID land + survive: affordability-gated priming math - (a 2nd pass can't starve the buffer) — sound, but moot without a working cut. - -- `[2026-06-02]` **Native frame-level streaming on Chatterbox-TURBO — ABANDONED - (turbo isn't built for streaming).** Long R&D arc; record so it's not - re-derived. (1) The model's flow is CosyVoice2-derived but `S3GenStreamer` is - referenced-in-docstring-only (not implemented). (2) The lib's - `flow_inference(finalize=False)` is BUGGY: the lookahead trim removes - `pre_lookahead_len(3)*token_mel_ratio(2)=6` frames from `h` but NOT from - `h_masks`/conds → decoder shape mismatch (e.g. 656 vs 662). A 1-line patch - (`h_masks = h_masks[:, :, :-pre*ratio]` after the `h` trim) + sizing the - meanflow noise to the trimmed length makes finalize=False RUN. (3) BUT the - flow encoder uses FULL-context attention (`static_chunk_size=0`), so - incremental/cumulative decode is **prefix-unstable** — adding tokens - re-attends and shifts earlier mel (maxdiff ~0.30-0.39 vs one-shot, - irrespective of fixed-noise slicing or emit-margin). (4) Forcing - `static_chunk_size>0` on the 2 modules that carry the attr did NOT stabilize - it (decoding_chunk_size is a forward-arg, not settable via attribute). Verdict: - true sub-second frame-streaming on turbo needs deep model-attention surgery - with quality risk — not worth it. Matches research ("turbo+streaming - unsolved"; vLLM-turbo outputs noise; davidbrowne17 streaming fork is - BASE-only). → Use adaptive-chunking instead. - -_41 older entries archived to archival-memory.md._ +_44 older entries archived to archival-memory.md._