From 19308ff5aa9dd853ac700f89946ade99f72e56f1 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 1 Jun 2026 22:21:17 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20chatterbox-fas?= =?UTF-8?q?t=20streaming=20build=20(adaptive=20buffer-ratchet=20chunking);?= =?UTF-8?q?=20native-turbo-streaming=20abandoned;=20fish=20reference=5Fid?= =?UTF-8?q?=20fix=20+=20glados=20transcript=20+=20NFS/smithy=20+=20referen?= =?UTF-8?q?ces-togglable;=20add=20design=20doc?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design/chatterbox-fast.md | 137 +++++++++++++++++++++++++++++++++ persistent-memory.md | 114 +++++++++++++++++++++++---- 2 files changed, 237 insertions(+), 14 deletions(-) create mode 100644 docs/design/chatterbox-fast.md diff --git a/docs/design/chatterbox-fast.md b/docs/design/chatterbox-fast.md new file mode 100644 index 0000000..08465ed --- /dev/null +++ b/docs/design/chatterbox-fast.md @@ -0,0 +1,137 @@ +# Chatterbox-Fast — Streaming TTS Engine (design) + +**Status:** Draft / pre-contract design. Spike-validated 2026-06-02. +**Owner:** infra-ops · **Workload (operator-confirmed):** single-stream interactive. + +## 1. Goal + +Make Chatterbox-Turbo our **primary interactive TTS engine** by cutting +time-to-first-audio from today's ~2.5 s to ~0.3–0.5 s via **true +incremental streaming**, while preserving turbo's quality, inline +paralinguistic tags, and voice cloning. + +## 2. Why now — the proven result + +Spike (2026-06-02, turbo on the A6000), proper CUDA-sync timing: + +| | time-to-first-audio | +|---|---| +| today (devnen, no real streaming) | **~2.5 s** | +| windowed generate, K=20 tokens | **0.31 s** (delivers ~0.96 s audio) | +| K=30 | 0.41 s | +| K=50 | 0.57 s | + +~**8× TTFB win.** Decode cost is ~constant **0.12 s** regardless of chunk +size (turbo's 2-step decoder), so first-chunk time is dominated by +generating the first K tokens — smaller K = faster first audio. Full +generation runs at **4.2× realtime**, so once the first chunk plays the +generator stays well ahead of playback and the stream never starves. + +## 3. Non-goals + +- High-concurrency **batch throughput** — that's a separate base-on-vLLM + lane; deferred (research: vLLM port doesn't support turbo). +- **Multilingual** — turbo is EN-only; out of scope. +- **Quantization** — research flags it high-risk (gibberish < Q8, Q8-CUDA + broken); deprioritized. + +## 4. Background — turbo internals (from the spike) + +- `generate()` = `t3.inference_turbo()` (AR loop, all tokens) → + `s3gen.inference(..., n_cfm_timesteps=2)` (flow + HiFT vocoder, all tokens). +- `inference_turbo` is a **plain-Python `for` loop** with a KV cache, a + `max_gen_len` param, and a stop-token break — cleanly hookable. +- `s3gen` decode is cheap per-window (~0.12 s constant). + +## 5. Architecture + +A **lean, purpose-built FastAPI server on the `chatterbox` library** — +*not* a fork of devnen. devnen buffers the entire synthesis before +emitting (proven: opus/mp3/wav all return first byte at full-synth time); +we need to own the generate loop. Components: + +1. **Model holder** — `ChatterboxTurboTTS` loaded once at startup, warmed. +2. **Streaming generate** — windowed wrapper around `inference_turbo`: + yields token windows; each window decoded via `s3gen` → audio chunk → + streamed. (~40 lines on top of the lib, per the spike.) +3. **Voice management** — predefined voices (dir of wavs) + clone refs + (the `prepare_conditionals` path); reuse chatterbox's `Conditionals`. +4. **HTTP API** — `POST /tts` (streaming + non-streaming), optional + OpenAI-compat `/v1/audio/speech`, `/health`. +5. **Watermark** — Resemble PerTh (mandatory); applied per-chunk or post. + +### Decision 1 — streaming transport +**Recommend: HTTP chunked transfer of raw PCM** (client plays chunks as +they arrive). Lowest latency, trivial client. Offer opus for +bandwidth-constrained callers. *Not* websocket (one-way; overkill). + +### Decision 2 — seam handling (the key productionization detail) +Independent per-window decode (as in the spike) can leave faint **seams** +at chunk boundaries because the vocoder has receptive-field context. +**Approach: overlap-discard** — decode each window with a small lookback +of the previous window's trailing tokens, discard that overlap's audio, +keep only the new window's output. (The `davidbrowne17/chatterbox-streaming` +fork uses this pattern.) Tune the overlap for inaudible seams vs latency. +**Validate** by ear + a spectral seam check. + +### Decision 3 — chunk schedule +First chunk **small** (K≈20–25 → ~0.3 s first audio); subsequent chunks +**larger** (K≈50–100) for decode efficiency, since after chunk 1 we're +ahead of playback. A simple ramp. + +## 6. Performance levers (fold in, measure each) + +- **bf16** (Ampere-safe), **TF32** (matmul), **SDPA/flash** backend on the + Llama backbone — low-risk, measure the delta. +- **torch.compile** — **DEFER.** Research flags a real batch-1 regression + risk (documented 0.85× at batch-1). Benchmark separately; adopt only if + it beats eager on our hardware. Not on the critical path. + +## 7. Deployment + +- New stack **`chatterbox-fast` deployed ALONGSIDE** the existing + `chatterbox` (zero disruption; A/B then cut over). +- **Port:** 8197 (next free on irv-ml1). +- **GPU placement (decided 2026-06-02):** **3090 (device 0) if it fits, + else A6000 (device 1).** The GPU stack is a shared dev stack — workloads + float across cards, so the 20.5 GB-at-idle on the 3090 is expected + residency, not a blocker. Fit is borderline: turbo is ~2.5 GB but the + 3090 currently shows ~3.5 GB free, so the deploy step **tries the 3090, + falls back to the A6000 (device 1, ~30 GB free, shares with Fish) on + OOM.** Pin via `device_ids` in compose per fleet convention. +- **From-source Dockerfile** (chatterbox lib + our server), pinned. + +## 8. Benchmark / A-B gate (deploy guard, like the Fish reference_id gate) + +- **first-audio (TTFB)** under target (e.g. < 0.6 s on the deployment GPU). +- **realtime factor** maintained (> 3×). +- **quality parity** vs current chatterbox — ECAPA speaker-sim for clone + voices, listen test for predefined, spectral **seam** check. +- Wire as a hard gate in the deploy playbook. + +## 9. Risks / open questions + +1. **Seam artifacts** — mitigation: overlap-discard decode; validate by ear + spectral. +2. **torch.compile batch-1 regression** — mitigation: benchmark, optional. +3. **3090's 20.5 GB-at-idle** — RESOLVED (non-issue): shared dev stack, expected residency. Placement decided (§7): 3090-if-fits-else-A6000. +4. **PerTh watermark on short chunks** — confirm no artifacts per-chunk. +5. **Paralinguistic tags across chunk boundaries** — confirm a tag split + across windows doesn't break delivery. + +## 10. Build plan (phases) + +0. **Spike** — DONE, proven (§2). +1. **Streaming server MVP** — windowed generate + overlap-discard seam + handling + `/tts` streaming endpoint; bench first-audio + seam quality. +2. **Parity + perf** — predefined + clone voice management; bf16/TF32/SDPA; + per-chunk watermark. +3. **Containerize + deploy** — from-source Dockerfile; deploy `chatterbox-fast` + alongside; wire the A-B gate. +4. **Cutover** — switch the catalog route; burn-in; deprecate the old stack. + +## 11. Open decisions for operator + +- ~~GPU placement~~ — **DECIDED (2026-06-02):** 3090 if it fits, else A6000 (§7). +- ~~Cutover strategy~~ — **DECIDED (2026-06-02): parallel catalog entry**, + burn-in beside the live `chatterbox`, then flip the route once it earns + trust. Phases 1–3 are cutover-agnostic; the flip happens in Phase 4. diff --git a/persistent-memory.md b/persistent-memory.md index 9865316..f36b53c 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-06-01_ +_Last updated: 2026-06-02_ ## Repo purpose @@ -87,8 +87,34 @@ Sister repos (separate gitea repos, deployed by playbooks here): ## Current state / in-flight -_As of 2026-06-01:_ +_As of 2026-06-02:_ +- **PRIMARY FOCUS — building `chatterbox-fast`, a custom streaming TTS + container; Chatterbox is becoming our MAIN TTS engine.** Operator-authorized + high-effort build (incl. custom container from source). **Plan-of-attack: + `/tmp/chatterbox-fast-plan.md`** (write/refresh before /clear — it carries + the full executable detail). Design doc: `docs/design/chatterbox-fast.md`. + - **Goal:** cut time-to-first-audio from ~2.5s → sub-second via streaming, + keep turbo quality. Workload = single-stream interactive. + - **Chosen approach = adaptive buffer-ratchet chunking** (operator's idea): + generate the FIRST sentence alone for instant response (~0.66s first-audio + measured), then while it plays, generate the largest sentence-aligned chunk + that fits in ~80% of the bought playback time; chunks grow ~3× each (driven + by Chatterbox's ~3.8× realtime headroom), so after 2-3 chunks the rest of + the paragraph is one big chunk with near-full context. Context loss confined + to 2-3 joins at natural sentence-pause boundaries. THIS is "sentence-level + done right" and supersedes naive per-sentence splitting (which loses + cross-sentence prosodic context = real quality loss, operator-corrected). + Only works because RTF>1 (Fish at <1× realtime starves — can't use this). + - **Native frame-level streaming on turbo = ABANDONED** (see Tried/abandoned). + - **Also to build for A/B:** base-chatterbox + davidbrowne17 streaming fork + (true frame-stream but base-model quality) — not yet installed. + - **A/B samples** (GLaDOS voice) on nh3-dev `~/chatterbox-ab/`: sentence-level + turbo, chunked-native (artifacty), chunked-oneshot. + - **Dev/test pattern:** one-off GPU-1 container from `local/chatterbox:v1`, + mount `/worktank/chatterbox/{cache,reference_audio}` + the test script; + `docker exec -i chatterbox python -` for lib introspection. lkraven is in + the `docker` group on irv-ml1 — NO sudo needed for docker. - **TTS fleet on irv-ml1 reworked this session; asset-engine catalog now 17 services** (CSM removed). Canonical `docs/asset-engine/services.yaml` + vendored `vh/asset-engine` both pushed (eshpfi `38d9e3b`; asset-engine @@ -109,15 +135,16 @@ _As of 2026-06-01:_ API). Research/non-commercial license. - **zonos (:8203)** down (adapter built, not deployed). **ComfyUI :8188** catalog-deferred link. -- **BLOCKED on asset-engine-dev — P1 "undefined" select bug** (default voice - picks send the literal string `"undefined"` → Chatterbox 404s, Fish falls to - default voice). ROOT-CAUSED: the form's Kokoro voice-blend widget - (`templates/partials/_fields.html`) runs `recompute()` at page load, reading - the Shoelace `.value` BEFORE hydration → writes "undefined" into - the submit field. Escalated with full root cause + 3-part fix (althing msg - `01KT2K2SY9N7AY69R9V0B4RXSW` → asset-engine-dev). **Workaround until fixed: - click the dropdown (fires change) or hit the API directly.** My half - (`blendable: false` catalog flag) is queued — see Recent decisions. +- **"undefined" select bug — RESOLVED.** asset-engine-dev shipped form-select + hardening (v0.1.9/.10) + a durable **per-field enable-toggle** (`togglable`, + v0.1.14/.16) — the real fix for the "form submits untouched fields" family. + My catalog half: opted fish-s2 `references` into `togglable: true` + (`catalog_version 1→2`, schema gained the `togglable` boolean; `c98a12b`, + pushed to origin). The earlier `blendable` idea was superseded by `togglable`. + dia2/chatterbox clone fields deliberately NOT toggled (dia2 defaults clone-on + as its stable voice). asset-engine CI was briefly red because the commit was + local-only until pushed — lesson: push catalog commits promptly (their CI + drift-checks against the remote). - **Fish "not British" had TWO independent root causes — BOTH now fixed.** The ECAPA-TDNN re-test (2026-06-01) showed Fish's cloning *engine* works: an Imogen reference scores **~0.79 cosine vs real `Imogen.wav`** vs **~0.10 vs @@ -147,9 +174,19 @@ _As of 2026-06-01:_ (CC BY 4.0, accent-tagged speaker IDs), Unmute voice-donations (CC0), EARS + expresso (CC BY-NC). Source for future clone voices. British-female Southern England speakers p225/p228/p229 staged into Fish as Imogen/Eleanor/Beatrice. -- **Parakeet ASR (:8765) is DOWN** — no container, but catalog says - `status: ready` (drift). Use Whisper-in-a-container (`faster-whisper`) for - transcripts instead; it's self-contained (CTranslate2, no torch). +- **Parakeet ASR (:8765) is now UP** (brought online 2026-06-02 to transcribe + the glados clip; operator wants it kept online). CPU-only (`gpu_device_id: + null`), `restart=unless-stopped`, healthy. API: POST multipart `file` to + `/transcribe` → `{"text":...}`. Image `local/parakeet:sherpa-onnx-v2` + pre-built; `docker compose up -d` in `/opt/docker/compose/parakeet`. +- **NFS share for Smithy — DONE.** `/volume1/smithy` on nh3-nas → `/mnt/smithy` + on nh3-dev (single export, RW, map-to-admin, scoped to nh3-dev only, fstab + `_netdev,nofail,x-systemd.automount`). Subdirs `datasets/{raw,manifest-store, + derived,holdout,quarantine}`; rename-atomicity verified. Smithy wired + storage-roots.yaml + E2E-verified. **Backup DEFERRED** per operator (datasets + regenerable; raw + manifest-store are the irreplaceable surface for when it's + wired — durable tier should land OFF nh3-nas/cross-site, since the NH3 restic + repo lives on the same NAS as the source). - **Worldtree healthy v0.29.13** (last-known); **Skaldsong v0.32.2** (ana-docker:8300, Kokoro SSE streaming); **artemis-31b-v1i** live on llama-swap + worldtree personal; **ttyd fleet driver seat** on nh3-dev @@ -168,6 +205,29 @@ _As of 2026-06-01:_ ## Recent decisions +- `[2026-06-02]` **Chatterbox → main TTS engine; build custom `chatterbox-fast` + streaming container.** Workload = single-stream interactive. **GPU placement: + 3090 (device 0) if it fits else A6000 (device 1)** — shared dev stack, 20.5 GB + 3090-idle is expected residency, not a blocker. **Cutover: parallel catalog + entry**, burn in beside live `chatterbox`, then flip. **Streaming approach: + adaptive buffer-ratchet chunking** (see in-flight). Native frame-streaming + abandoned (Tried/abandoned). Tracked: `docs/design/chatterbox-fast.md` + + `/tmp/chatterbox-fast-plan.md`. + +- `[2026-06-02]` **Sentence-splitting loses quality (operator-corrected).** I + claimed naive sentence-level streaming has "zero quality loss" — WRONG. The + T3 AR backbone conditions prosody on the WHOLE text; splitting loses + cross-sentence prosodic context (contextual delivery, declination, affect + continuity) even though voice timbre stays (reference-conditioned). No + *artifacts* ≠ no *quality loss*. Hence the adaptive-chunk design (maximize + context per chunk subject to latency budget), not fixed per-sentence splits. + +- `[2026-06-01]` **Fish reference_id empty-dir fix shipped** (`c5bbb90`) — see + in-flight + Tried/abandoned. Populated `references//.wav`+`.lab` + for all 32 voices; playbook gained normalize-step + A/B smoke gate. glados got + a real transcript (ASR'd via Parakeet): the Portal "Welcome to test chamber 4" + lines. + - `[2026-06-01]` **Fish cloning VERIFIED competent (ECAPA-TDNN)** — retracting the earlier "weak cloner" call. Isolated test: Imogen-referenced clone ~0.79 cosine to the real `Imogen.wav` vs ~0.10 for the no-reference default; @@ -262,6 +322,32 @@ _25 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-06-02]` **Native frame-level streaming on Chatterbox-TURBO — ABANDONED + (turbo isn't built for streaming).** Long R&D arc; record so it's not + re-derived. (1) The model's flow is CosyVoice2-derived but `S3GenStreamer` is + referenced-in-docstring-only (not implemented). (2) The lib's + `flow_inference(finalize=False)` is BUGGY: the lookahead trim removes + `pre_lookahead_len(3)*token_mel_ratio(2)=6` frames from `h` but NOT from + `h_masks`/conds → decoder shape mismatch (e.g. 656 vs 662). A 1-line patch + (`h_masks = h_masks[:, :, :-pre*ratio]` after the `h` trim) + sizing the + meanflow noise to the trimmed length makes finalize=False RUN. (3) BUT the + flow encoder uses FULL-context attention (`static_chunk_size=0`), so + incremental/cumulative decode is **prefix-unstable** — adding tokens + re-attends and shifts earlier mel (maxdiff ~0.30-0.39 vs one-shot, + irrespective of fixed-noise slicing or emit-margin). (4) Forcing + `static_chunk_size>0` on the 2 modules that carry the attr did NOT stabilize + it (decoding_chunk_size is a forward-arg, not settable via attribute). Verdict: + true sub-second frame-streaming on turbo needs deep model-attention surgery + with quality risk — not worth it. Matches research ("turbo+streaming + unsolved"; vLLM-turbo outputs noise; davidbrowne17 streaming fork is + BASE-only). → Use adaptive-chunking instead. + +- `[2026-06-02]` **Naive cumulative re-decode for streaming** — decode the + growing token prefix each chunk + emit the delta, assuming the causal flow + gives a stable prefix. It does NOT (full-context attention, see above); + maxdiff 0.30. Also the high-level `s3gen.inference` re-applies `trim_fade` at + the start every call. Don't go this way. + - `[2026-06-01]` **CSM bring-up** — upstream `phildougherty/sesame_csm_openai` Dockerfile pins NO `huggingface_hub` version, so it now resolves to 1.17.0 where `huggingface-cli` is fully removed (replaced by `hf`) → the build dies