diff --git a/persistent-memory.md b/persistent-memory.md index 951158f..41704b7 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -128,6 +128,8 @@ _As of 2026-08-08 — long session; all major arcs LANDED (full detail per arc i ## Recent decisions +- `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/.wav` + `transcripts/.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **PENDING: operator A/B ear-verdict → Phase 2** (containerize SGLang Omni serving dots.tts-soar on the 3090 alongside chatterbox; ratatoskr client cutover to `/v1/audio/speech`); wrapper-vs-Omni serving-layer choice deferred to Phase 2. **OPEN operator call:** corpus home = eshpfi `voices/` (my rec) vs spin-out `vh/voice-corpus`. [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]] + - `[2026-08-08]` **worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev.** worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared `~/snapshots/worldtree-400-fiction-decomp` (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). **LEFT INTACT:** `rex393-fiction-index`/`rex393-fiction-snapshot` (separate operator KEEP word, unchanged) + `r42-gate-*`. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (`01KZJ9GMCC…`). - `[2026-08-07]` **chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed).** Long saga, operator-driven clean diagnosis. **Symptom:** ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. **NOT** German-leak (Turbo `generate()` has NO language param — plain AutoTokenizer, no `language_id`; the multilingual `language_id="en"` lever lives only in the separate `ChatterboxMultilingualTTS`), **NOT** OOM alone. **Real cause:** the Chatterbox **Turbo T3 model OVER-RUNS its generation tail** — a long single `generate()` degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens `<6561` + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). **ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE** — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). **Diagnosis method** (deterministic, no ears-only): single-shot length sweep + **amplitude-gated voiced-ZCR** (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. **FIX:** server-side `max_chunk_chars=250` cap on the scheduler (`:v2` image, `CBF_MAX_CHUNK_CHARS=250` env) — bounds each generation to just under the ~300-char onset → clean **3-4 sentence** chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; **chatterbox's low emotiveness keeps chunk joins smooth** (the harsh joins that got Zonos rejected are absent — operator's key call). **ratatoskr TODO (relayed msg `01KZER9X7S`):** revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. **Cap value tunable** per-request (`max_chunk_chars`) + env. **Deeper prosody** (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). **⚠ FOOT-GUNS:** (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) **build-context vs image drift** — the `:v2` image was built from cap source, but after a `:v1` rollback the build context held `:v1` source → a `docker compose build` would've silently produced a cap-less `:v2`; re-synced the flat cap source to `/opt/docker/compose/chatterbox-fast/` (rebuild-verified). **⚠ DIVERGENCE (follow-up):** deployed build context is FLAT (`app.py`/`scheduler.py`, `from scheduler import`, thin-overlay `FROM local/chatterbox:v1`, cap-only) vs the `vh/chatterbox-fast` REPO which is PACKAGE-layout (`chatterbox_fast/`, `from chatterbox_fast.scheduler`, self-contained Dockerfile) + has `norm_loudness` (repo commit `6bc7bf0` = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: `.bak-cap-20260807-104850` backups on irv-ml1 + `:v1` image both retained. [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] diff --git a/voices/.gitignore b/voices/.gitignore new file mode 100644 index 0000000..7085938 --- /dev/null +++ b/voices/.gitignore @@ -0,0 +1,2 @@ +# Per-engine reference sets are build outputs — regenerate with derive.py. +derived/ diff --git a/voices/README.md b/voices/README.md new file mode 100644 index 0000000..05a65b9 --- /dev/null +++ b/voices/README.md @@ -0,0 +1,78 @@ +# Canonical voice corpus + +Engine-agnostic source of truth for cloned voice identities. Each voice is +stored **once** as a canonical source clip + an accurate transcript; per-engine +reference sets (dots.tts, chatterbox, zonos, …) are **derived** from it on +demand by [`derive.py`](derive.py). Adding a new TTS engine is "add a profile +to [`engines.yaml`](engines.yaml) and re-derive" — not "re-hunt every voice." + +## Why this exists + +TTS engines disagree on what a reference clip must be: + +| Engine | SR | Transcript? | Reference shape | +|---|---|---|---| +| **dots.tts** | 48kHz | **required** | ~≤10s, **sentence-bounded**, accurate transcript | +| chatterbox-fast | 24kHz | no | any length, audio-only | +| Zonos2 | 44.1kHz | no | any length, audio-only + emotion dials | + +Keeping one canonical source per voice + a derivation step means a voice cloned +for Zonos a year ago can be re-optimized for whatever engine comes next without +re-sourcing the audio. + +## The dots.tts sensitivity finding (load-bearing) + +dots.tts conditions each generation on (reference audio **+ its transcript**) and +**regurgitates reference content into the output** when the transcript is wrong +**or ends mid-clause**. Symptoms seen during the 2026-08 burn-in: a mismatched +transcript collapsed output to 0.16s; an over-long reference with a repetitive +transcript prefixed the output with reference lines; a transcript trimmed +mid-clause ("…we will") leaked a stray "we'll" into the output. The reliable +recipe — encoded in `derive.py` for the `dots` profile — is **trim to a clean +~≤10s clip ending on a sentence boundary (. ! ?) with an accurate transcript of +exactly that clip.** + +## Layout + +``` +voices/ + manifest.yaml # voice registry: canonical path, transcript, SR, provenance + engines.yaml # per-engine reference requirements + derive.py # canonical -> derived//.{wav,txt} + canonical/.wav # source clip, best available SR (git-tracked, small + curated) + transcripts/.txt # full accurate transcript of the canonical source + derived/ # per-engine reference sets (GITIGNORED — regenerable) + dots/.{wav,txt} + chatterbox/.wav +``` + +## Usage + +```bash +# derive dots-ready references for every voice (needs faster-whisper for the trim): +python derive.py dots + +# just two voices: +python derive.py dots donut glados + +# a no-transcript engine (copies canonical; resample = follow-up, see engines.yaml): +python derive.py chatterbox +``` + +`derived/` is gitignored — treat it as a build output. Deploy a derived set to a +live engine by copying `derived//` into that stack's refs dir +(e.g. dots' voices mount, chatterbox `/worktank/chatterbox/reference_audio/`). + +## Adding a voice + +1. Drop the best available source clip in `canonical/.wav` (highest SR, + cleanest, ~10–30s is plenty). +2. Add its row to `manifest.yaml` (SR, duration, provenance). +3. `python derive.py dots ` — writes the transcript + dots reference and, + if you wire it, a verify pass. + +## Provenance discipline + +Record where each source came from in `manifest.yaml`. Unknown origin is fine to +start (`origin unrecorded`) but should be filled in when known — a canonical +corpus is only as trustworthy as its provenance. diff --git a/voices/canonical/donut.wav b/voices/canonical/donut.wav new file mode 100644 index 0000000..aea646d Binary files /dev/null and b/voices/canonical/donut.wav differ diff --git a/voices/canonical/emmie.wav b/voices/canonical/emmie.wav new file mode 100644 index 0000000..a3ce463 Binary files /dev/null and b/voices/canonical/emmie.wav differ diff --git a/voices/canonical/glados.wav b/voices/canonical/glados.wav new file mode 100644 index 0000000..e33f3e4 Binary files /dev/null and b/voices/canonical/glados.wav differ diff --git a/voices/canonical/miranda.wav b/voices/canonical/miranda.wav new file mode 100644 index 0000000..a438faf Binary files /dev/null and b/voices/canonical/miranda.wav differ diff --git a/voices/derive.py b/voices/derive.py new file mode 100644 index 0000000..1247aa5 --- /dev/null +++ b/voices/derive.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python3 +"""Derive per-engine reference sets from the canonical voice corpus. + +Reads manifest.yaml + engines.yaml and, for a chosen engine, writes +derived//.wav (plus .txt when the engine needs a +transcript). + +Usage: + python derive.py [voice ...] # default: every voice in manifest + +Deps: pyyaml, soundfile. faster-whisper is imported lazily, only when an engine +sets ref_sentence_bounded (dots) — it picks a clean sentence-boundary trim and +its exact transcript. Run under a venv that has these (on irv-ml1 the dots + +whisper venvs already do). + +Known follow-up: `resample: true` engines (chatterbox, zonos) currently COPY the +canonical clip at its source SR rather than resampling — a proper resample step +(soundfile + a resampler) is a TODO. dots sets resample:false (it resamples +internally at load), so the dots path is complete. +""" +import sys +import wave +import pathlib +import shutil +import yaml + +ROOT = pathlib.Path(__file__).parent + + +def load(): + manifest = yaml.safe_load((ROOT / "manifest.yaml").read_text())["voices"] + engines = yaml.safe_load((ROOT / "engines.yaml").read_text())["engines"] + return manifest, engines + + +DANGLING = {"and", "but", "so", "or", "the", "a", "an", "that", "to", "my", + "because", "with", "of", "for", "as", "i", "we", "it", "is"} + + +def sentence_bounded_trim(src, target_s, model, min_s=6.0): + """Return (end_seconds, transcript) for a clip ending on a real sentence + boundary. + + Accumulates whisper segments and takes the FIRST point past `min_s` where the + running transcript ends in . ! ? — searching up to target_s+4 so a run-on + conversational source (no boundary early) still lands on a real sentence end + rather than a dangling clause. Only if the source has no boundary at all in + that window does it fall back to a best-effort trim with the trailing dangling + conjunction/article stripped — a partial-clause tail is exactly what dots.tts + regurgitates into its output. + """ + target_s = float(target_s) if target_s else 10.0 + hard_max = target_s + 4.0 + segs = list(model.transcribe(src, beam_size=5)[0]) + acc, best_end, best_txt = [], None, None + for s in segs: + if s.end > hard_max: + break + acc.append(s) + txt = " ".join(x.text.strip() for x in acc).strip() + if txt.endswith((".", "!", "?")): + best_end, best_txt = s.end, txt + if s.end >= min_s: + break + if best_end is not None: + return best_end, best_txt or "" + # no sentence boundary in-window — best effort, strip the dangling tail + end = acc[-1].end if acc else 0.0 + words = " ".join(x.text.strip() for x in acc).strip().rstrip(",").split() + while words and words[-1].lower().strip(",.") in DANGLING: + words.pop() + return end, " ".join(words) + + +def trim_wav(src, dst, end_s): + w = wave.open(str(src)) + sr = w.getframerate() + frames = w.readframes(int(end_s * sr)) + w.close() + o = wave.open(str(dst), "w") + o.setnchannels(1) + o.setsampwidth(2) + o.setframerate(sr) + o.writeframes(frames) + o.close() + + +def main(): + if len(sys.argv) < 2: + sys.exit("usage: derive.py [voice ...]") + engine = sys.argv[1] + manifest, engines = load() + if engine not in engines: + sys.exit(f"unknown engine '{engine}'; have {list(engines)}") + prof = engines[engine] + names = sys.argv[2:] or list(manifest) + + outdir = ROOT / "derived" / engine + outdir.mkdir(parents=True, exist_ok=True) + + model = None + if prof.get("ref_sentence_bounded"): + from faster_whisper import WhisperModel + model = WhisperModel("base.en", device="cpu", compute_type="int8") + + for v in names: + vc = manifest[v] + src = ROOT / vc["canonical"] + dst_wav = outdir / f"{v}.wav" + if prof.get("ref_sentence_bounded"): + end, txt = sentence_bounded_trim(str(src), prof.get("ref_max_seconds") or 10, model) + trim_wav(src, dst_wav, end) + if prof.get("needs_transcript"): + (outdir / f"{v}.txt").write_text(txt + "\n") + print(f"{engine}/{v}: {end:.1f}s sentence-bounded | {txt}") + else: + # TODO: resample to prof['sample_rate'] when resample:true + shutil.copy(src, dst_wav) + if prof.get("needs_transcript"): + (outdir / f"{v}.txt").write_text((ROOT / vc["transcript"]).read_text()) + print(f"{engine}/{v}: copied canonical ({vc.get('source_sr')}Hz) -> {dst_wav.name}") + + +if __name__ == "__main__": + main() diff --git a/voices/engines.yaml b/voices/engines.yaml new file mode 100644 index 0000000..5fe7d14 --- /dev/null +++ b/voices/engines.yaml @@ -0,0 +1,40 @@ +# Per-engine reference requirements. derive.py reads this to turn a canonical +# source + transcript into an engine-ready reference set under derived//. +# +# Fields: +# sample_rate native SR the engine wants +# resample true = derive.py should resample to sample_rate +# (NOTE: resample is a follow-up — see the resample TODO +# in derive.py; dots resamples internally so it's false there) +# needs_transcript engine requires a per-reference transcript file +# ref_max_seconds cap on derived reference length (null = uncapped) +# ref_sentence_bounded transcript/clip must end on a sentence boundary (. ! ?) +# — set for engines that leak reference content otherwise + +engines: + dots: + description: "dots.tts (rednote-hilab) — continuous-AR 48kHz zero-shot clone" + sample_rate: 48000 + resample: false # runtime auto-resamples at load; keep source SR + needs_transcript: true # REQUIRED and must be accurate + sentence-bounded + ref_max_seconds: 10 + ref_sentence_bounded: true + notes: > + Transcript accuracy AND sentence-boundary are load-bearing: a mismatched or + mid-clause transcript makes dots regurgitate reference audio into the output. + + chatterbox: + description: "chatterbox-fast (Turbo) — streaming 24kHz clone" + sample_rate: 24000 + resample: true + needs_transcript: false # audio-only clone; server globs its refs dir live + ref_max_seconds: null + ref_sentence_bounded: false + + zonos: + description: "Zonos2 — expressive 44.1kHz clone + emotion dials" + sample_rate: 44100 + resample: true + needs_transcript: false + ref_max_seconds: null + ref_sentence_bounded: false diff --git a/voices/manifest.yaml b/voices/manifest.yaml new file mode 100644 index 0000000..68f217a --- /dev/null +++ b/voices/manifest.yaml @@ -0,0 +1,35 @@ +# Canonical voice corpus registry. One row per voice; the canonical clip + its +# full transcript are the source of truth, engine-agnostic. derive.py reads this +# together with engines.yaml to produce per-engine reference sets. + +voices: + donut: + canonical: canonical/donut.wav + transcript: transcripts/donut.txt + source_sr: 44100 + duration_s: 16.3 + character: "sassy fairy-charm kid" + provenance: "cloned from the 65-frost Booth bundle (2026-08)" + + glados: + canonical: canonical/glados.wav + transcript: transcripts/glados.txt + source_sr: 16000 + duration_s: 25.0 + character: "GLaDOS — flat, deliberate, menacing-cheerful" + provenance: "Portal GLaDOS lines" + warning: "LOW-SR source (16kHz) — upgrade the canonical clip if a cleaner GLaDOS source surfaces" + + emmie: + canonical: canonical/emmie.wav + transcript: transcripts/emmie.txt + source_sr: 24000 + duration_s: 19.3 + provenance: "Zonos clone added 2026-07-17; origin unrecorded" + + miranda: + canonical: canonical/miranda.wav + transcript: transcripts/miranda.txt + source_sr: 24000 + duration_s: 16.3 + provenance: "Zonos clone added 2026-07-17; origin unrecorded" diff --git a/voices/transcripts/donut.txt b/voices/transcripts/donut.txt new file mode 100644 index 0000000..ebbd1c0 --- /dev/null +++ b/voices/transcripts/donut.txt @@ -0,0 +1 @@ +This is just not acceptable, Carl. I like my butterfly charm. It makes it so fairies like me, and it is pretty. It's part of my fit. I don't want to take it off. I don't see why I can't just wear two charms at the same time. Stupid angel of the caucus spaniel had like four or five tags. \ No newline at end of file diff --git a/voices/transcripts/emmie.txt b/voices/transcripts/emmie.txt new file mode 100644 index 0000000..ba2dd76 --- /dev/null +++ b/voices/transcripts/emmie.txt @@ -0,0 +1 @@ +I think I mentioned but I read your book because my my dear friend Nupa told me that I should and every now and again I would see you come up. I don't know. I take my job seriously I guess and so interviews to me felt a lot like chess and it required so much energy. \ No newline at end of file diff --git a/voices/transcripts/glados.txt b/voices/transcripts/glados.txt new file mode 100644 index 0000000..657b36a --- /dev/null +++ b/voices/transcripts/glados.txt @@ -0,0 +1 @@ +Welcome to test chamber 4. You're doing quite well. Once again, excellent work. As part of a required test protocol, we will not monitor the next test chamber. You will be entirely on your own. Good luck! As part of a required test protocol, our previous statement suggesting that we would not monitor this chamber was an outright fabrication. Good job! As part of a required test protocol, we will not monitor the next test protocol. \ No newline at end of file diff --git a/voices/transcripts/miranda.txt b/voices/transcripts/miranda.txt new file mode 100644 index 0000000..b98604e --- /dev/null +++ b/voices/transcripts/miranda.txt @@ -0,0 +1 @@ +It's great. I mean, it's definitely comforting to go back to Australia when I come from there. So, you know, I get to see my parents, I get to see my friends and hang out. And I know the city really well because this was my fourth movie that I did in... \ No newline at end of file