feat(voices): canonical voice corpus + dots.tts-optimized refs
Engine-agnostic voice corpus: canonical source clip + transcript per voice, per-engine reference sets derived by derive.py from engines.yaml profiles. First residents donut/glados/emmie/miranda optimized + verified clean for dots.tts (sentence-bounded ref + accurate transcript — dots leaks reference audio into output otherwise). canonical/ + transcripts/ tracked; derived/ gitignored (regenerable). Records the dots.tts burn-in in persistent-memory.
This commit is contained in:
@@ -128,6 +128,8 @@ _As of 2026-08-08 — long session; all major arcs LANDED (full detail per arc i
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **PENDING: operator A/B ear-verdict → Phase 2** (containerize SGLang Omni serving dots.tts-soar on the 3090 alongside chatterbox; ratatoskr client cutover to `/v1/audio/speech`); wrapper-vs-Omni serving-layer choice deferred to Phase 2. **OPEN operator call:** corpus home = eshpfi `voices/` (my rec) vs spin-out `vh/voice-corpus`. [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]]
|
||||
|
||||
- `[2026-08-08]` **worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev.** worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared `~/snapshots/worldtree-400-fiction-decomp` (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). **LEFT INTACT:** `rex393-fiction-index`/`rex393-fiction-snapshot` (separate operator KEEP word, unchanged) + `r42-gate-*`. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (`01KZJ9GMCC…`).
|
||||
|
||||
- `[2026-08-07]` **chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed).** Long saga, operator-driven clean diagnosis. **Symptom:** ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. **NOT** German-leak (Turbo `generate()` has NO language param — plain AutoTokenizer, no `language_id`; the multilingual `language_id="en"` lever lives only in the separate `ChatterboxMultilingualTTS`), **NOT** OOM alone. **Real cause:** the Chatterbox **Turbo T3 model OVER-RUNS its generation tail** — a long single `generate()` degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens `<6561` + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). **ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE** — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). **Diagnosis method** (deterministic, no ears-only): single-shot length sweep + **amplitude-gated voiced-ZCR** (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. **FIX:** server-side `max_chunk_chars=250` cap on the scheduler (`:v2` image, `CBF_MAX_CHUNK_CHARS=250` env) — bounds each generation to just under the ~300-char onset → clean **3-4 sentence** chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; **chatterbox's low emotiveness keeps chunk joins smooth** (the harsh joins that got Zonos rejected are absent — operator's key call). **ratatoskr TODO (relayed msg `01KZER9X7S`):** revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. **Cap value tunable** per-request (`max_chunk_chars`) + env. **Deeper prosody** (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). **⚠ FOOT-GUNS:** (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) **build-context vs image drift** — the `:v2` image was built from cap source, but after a `:v1` rollback the build context held `:v1` source → a `docker compose build` would've silently produced a cap-less `:v2`; re-synced the flat cap source to `/opt/docker/compose/chatterbox-fast/` (rebuild-verified). **⚠ DIVERGENCE (follow-up):** deployed build context is FLAT (`app.py`/`scheduler.py`, `from scheduler import`, thin-overlay `FROM local/chatterbox:v1`, cap-only) vs the `vh/chatterbox-fast` REPO which is PACKAGE-layout (`chatterbox_fast/`, `from chatterbox_fast.scheduler`, self-contained Dockerfile) + has `norm_loudness` (repo commit `6bc7bf0` = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: `.bak-cap-20260807-104850` backups on irv-ml1 + `:v1` image both retained. [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]]
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
# Per-engine reference sets are build outputs — regenerate with derive.py.
|
||||
derived/
|
||||
@@ -0,0 +1,78 @@
|
||||
# Canonical voice corpus
|
||||
|
||||
Engine-agnostic source of truth for cloned voice identities. Each voice is
|
||||
stored **once** as a canonical source clip + an accurate transcript; per-engine
|
||||
reference sets (dots.tts, chatterbox, zonos, …) are **derived** from it on
|
||||
demand by [`derive.py`](derive.py). Adding a new TTS engine is "add a profile
|
||||
to [`engines.yaml`](engines.yaml) and re-derive" — not "re-hunt every voice."
|
||||
|
||||
## Why this exists
|
||||
|
||||
TTS engines disagree on what a reference clip must be:
|
||||
|
||||
| Engine | SR | Transcript? | Reference shape |
|
||||
|---|---|---|---|
|
||||
| **dots.tts** | 48kHz | **required** | ~≤10s, **sentence-bounded**, accurate transcript |
|
||||
| chatterbox-fast | 24kHz | no | any length, audio-only |
|
||||
| Zonos2 | 44.1kHz | no | any length, audio-only + emotion dials |
|
||||
|
||||
Keeping one canonical source per voice + a derivation step means a voice cloned
|
||||
for Zonos a year ago can be re-optimized for whatever engine comes next without
|
||||
re-sourcing the audio.
|
||||
|
||||
## The dots.tts sensitivity finding (load-bearing)
|
||||
|
||||
dots.tts conditions each generation on (reference audio **+ its transcript**) and
|
||||
**regurgitates reference content into the output** when the transcript is wrong
|
||||
**or ends mid-clause**. Symptoms seen during the 2026-08 burn-in: a mismatched
|
||||
transcript collapsed output to 0.16s; an over-long reference with a repetitive
|
||||
transcript prefixed the output with reference lines; a transcript trimmed
|
||||
mid-clause ("…we will") leaked a stray "we'll" into the output. The reliable
|
||||
recipe — encoded in `derive.py` for the `dots` profile — is **trim to a clean
|
||||
~≤10s clip ending on a sentence boundary (. ! ?) with an accurate transcript of
|
||||
exactly that clip.**
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
voices/
|
||||
manifest.yaml # voice registry: canonical path, transcript, SR, provenance
|
||||
engines.yaml # per-engine reference requirements
|
||||
derive.py # canonical -> derived/<engine>/<voice>.{wav,txt}
|
||||
canonical/<v>.wav # source clip, best available SR (git-tracked, small + curated)
|
||||
transcripts/<v>.txt # full accurate transcript of the canonical source
|
||||
derived/ # per-engine reference sets (GITIGNORED — regenerable)
|
||||
dots/<v>.{wav,txt}
|
||||
chatterbox/<v>.wav
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# derive dots-ready references for every voice (needs faster-whisper for the trim):
|
||||
python derive.py dots
|
||||
|
||||
# just two voices:
|
||||
python derive.py dots donut glados
|
||||
|
||||
# a no-transcript engine (copies canonical; resample = follow-up, see engines.yaml):
|
||||
python derive.py chatterbox
|
||||
```
|
||||
|
||||
`derived/` is gitignored — treat it as a build output. Deploy a derived set to a
|
||||
live engine by copying `derived/<engine>/` into that stack's refs dir
|
||||
(e.g. dots' voices mount, chatterbox `/worktank/chatterbox/reference_audio/`).
|
||||
|
||||
## Adding a voice
|
||||
|
||||
1. Drop the best available source clip in `canonical/<name>.wav` (highest SR,
|
||||
cleanest, ~10–30s is plenty).
|
||||
2. Add its row to `manifest.yaml` (SR, duration, provenance).
|
||||
3. `python derive.py dots <name>` — writes the transcript + dots reference and,
|
||||
if you wire it, a verify pass.
|
||||
|
||||
## Provenance discipline
|
||||
|
||||
Record where each source came from in `manifest.yaml`. Unknown origin is fine to
|
||||
start (`origin unrecorded`) but should be filled in when known — a canonical
|
||||
corpus is only as trustworthy as its provenance.
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,125 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Derive per-engine reference sets from the canonical voice corpus.
|
||||
|
||||
Reads manifest.yaml + engines.yaml and, for a chosen engine, writes
|
||||
derived/<engine>/<voice>.wav (plus <voice>.txt when the engine needs a
|
||||
transcript).
|
||||
|
||||
Usage:
|
||||
python derive.py <engine> [voice ...] # default: every voice in manifest
|
||||
|
||||
Deps: pyyaml, soundfile. faster-whisper is imported lazily, only when an engine
|
||||
sets ref_sentence_bounded (dots) — it picks a clean sentence-boundary trim and
|
||||
its exact transcript. Run under a venv that has these (on irv-ml1 the dots +
|
||||
whisper venvs already do).
|
||||
|
||||
Known follow-up: `resample: true` engines (chatterbox, zonos) currently COPY the
|
||||
canonical clip at its source SR rather than resampling — a proper resample step
|
||||
(soundfile + a resampler) is a TODO. dots sets resample:false (it resamples
|
||||
internally at load), so the dots path is complete.
|
||||
"""
|
||||
import sys
|
||||
import wave
|
||||
import pathlib
|
||||
import shutil
|
||||
import yaml
|
||||
|
||||
ROOT = pathlib.Path(__file__).parent
|
||||
|
||||
|
||||
def load():
|
||||
manifest = yaml.safe_load((ROOT / "manifest.yaml").read_text())["voices"]
|
||||
engines = yaml.safe_load((ROOT / "engines.yaml").read_text())["engines"]
|
||||
return manifest, engines
|
||||
|
||||
|
||||
DANGLING = {"and", "but", "so", "or", "the", "a", "an", "that", "to", "my",
|
||||
"because", "with", "of", "for", "as", "i", "we", "it", "is"}
|
||||
|
||||
|
||||
def sentence_bounded_trim(src, target_s, model, min_s=6.0):
|
||||
"""Return (end_seconds, transcript) for a clip ending on a real sentence
|
||||
boundary.
|
||||
|
||||
Accumulates whisper segments and takes the FIRST point past `min_s` where the
|
||||
running transcript ends in . ! ? — searching up to target_s+4 so a run-on
|
||||
conversational source (no boundary early) still lands on a real sentence end
|
||||
rather than a dangling clause. Only if the source has no boundary at all in
|
||||
that window does it fall back to a best-effort trim with the trailing dangling
|
||||
conjunction/article stripped — a partial-clause tail is exactly what dots.tts
|
||||
regurgitates into its output.
|
||||
"""
|
||||
target_s = float(target_s) if target_s else 10.0
|
||||
hard_max = target_s + 4.0
|
||||
segs = list(model.transcribe(src, beam_size=5)[0])
|
||||
acc, best_end, best_txt = [], None, None
|
||||
for s in segs:
|
||||
if s.end > hard_max:
|
||||
break
|
||||
acc.append(s)
|
||||
txt = " ".join(x.text.strip() for x in acc).strip()
|
||||
if txt.endswith((".", "!", "?")):
|
||||
best_end, best_txt = s.end, txt
|
||||
if s.end >= min_s:
|
||||
break
|
||||
if best_end is not None:
|
||||
return best_end, best_txt or ""
|
||||
# no sentence boundary in-window — best effort, strip the dangling tail
|
||||
end = acc[-1].end if acc else 0.0
|
||||
words = " ".join(x.text.strip() for x in acc).strip().rstrip(",").split()
|
||||
while words and words[-1].lower().strip(",.") in DANGLING:
|
||||
words.pop()
|
||||
return end, " ".join(words)
|
||||
|
||||
|
||||
def trim_wav(src, dst, end_s):
|
||||
w = wave.open(str(src))
|
||||
sr = w.getframerate()
|
||||
frames = w.readframes(int(end_s * sr))
|
||||
w.close()
|
||||
o = wave.open(str(dst), "w")
|
||||
o.setnchannels(1)
|
||||
o.setsampwidth(2)
|
||||
o.setframerate(sr)
|
||||
o.writeframes(frames)
|
||||
o.close()
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
sys.exit("usage: derive.py <engine> [voice ...]")
|
||||
engine = sys.argv[1]
|
||||
manifest, engines = load()
|
||||
if engine not in engines:
|
||||
sys.exit(f"unknown engine '{engine}'; have {list(engines)}")
|
||||
prof = engines[engine]
|
||||
names = sys.argv[2:] or list(manifest)
|
||||
|
||||
outdir = ROOT / "derived" / engine
|
||||
outdir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
model = None
|
||||
if prof.get("ref_sentence_bounded"):
|
||||
from faster_whisper import WhisperModel
|
||||
model = WhisperModel("base.en", device="cpu", compute_type="int8")
|
||||
|
||||
for v in names:
|
||||
vc = manifest[v]
|
||||
src = ROOT / vc["canonical"]
|
||||
dst_wav = outdir / f"{v}.wav"
|
||||
if prof.get("ref_sentence_bounded"):
|
||||
end, txt = sentence_bounded_trim(str(src), prof.get("ref_max_seconds") or 10, model)
|
||||
trim_wav(src, dst_wav, end)
|
||||
if prof.get("needs_transcript"):
|
||||
(outdir / f"{v}.txt").write_text(txt + "\n")
|
||||
print(f"{engine}/{v}: {end:.1f}s sentence-bounded | {txt}")
|
||||
else:
|
||||
# TODO: resample to prof['sample_rate'] when resample:true
|
||||
shutil.copy(src, dst_wav)
|
||||
if prof.get("needs_transcript"):
|
||||
(outdir / f"{v}.txt").write_text((ROOT / vc["transcript"]).read_text())
|
||||
print(f"{engine}/{v}: copied canonical ({vc.get('source_sr')}Hz) -> {dst_wav.name}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,40 @@
|
||||
# Per-engine reference requirements. derive.py reads this to turn a canonical
|
||||
# source + transcript into an engine-ready reference set under derived/<engine>/.
|
||||
#
|
||||
# Fields:
|
||||
# sample_rate native SR the engine wants
|
||||
# resample true = derive.py should resample to sample_rate
|
||||
# (NOTE: resample is a follow-up — see the resample TODO
|
||||
# in derive.py; dots resamples internally so it's false there)
|
||||
# needs_transcript engine requires a per-reference transcript file
|
||||
# ref_max_seconds cap on derived reference length (null = uncapped)
|
||||
# ref_sentence_bounded transcript/clip must end on a sentence boundary (. ! ?)
|
||||
# — set for engines that leak reference content otherwise
|
||||
|
||||
engines:
|
||||
dots:
|
||||
description: "dots.tts (rednote-hilab) — continuous-AR 48kHz zero-shot clone"
|
||||
sample_rate: 48000
|
||||
resample: false # runtime auto-resamples at load; keep source SR
|
||||
needs_transcript: true # REQUIRED and must be accurate + sentence-bounded
|
||||
ref_max_seconds: 10
|
||||
ref_sentence_bounded: true
|
||||
notes: >
|
||||
Transcript accuracy AND sentence-boundary are load-bearing: a mismatched or
|
||||
mid-clause transcript makes dots regurgitate reference audio into the output.
|
||||
|
||||
chatterbox:
|
||||
description: "chatterbox-fast (Turbo) — streaming 24kHz clone"
|
||||
sample_rate: 24000
|
||||
resample: true
|
||||
needs_transcript: false # audio-only clone; server globs its refs dir live
|
||||
ref_max_seconds: null
|
||||
ref_sentence_bounded: false
|
||||
|
||||
zonos:
|
||||
description: "Zonos2 — expressive 44.1kHz clone + emotion dials"
|
||||
sample_rate: 44100
|
||||
resample: true
|
||||
needs_transcript: false
|
||||
ref_max_seconds: null
|
||||
ref_sentence_bounded: false
|
||||
@@ -0,0 +1,35 @@
|
||||
# Canonical voice corpus registry. One row per voice; the canonical clip + its
|
||||
# full transcript are the source of truth, engine-agnostic. derive.py reads this
|
||||
# together with engines.yaml to produce per-engine reference sets.
|
||||
|
||||
voices:
|
||||
donut:
|
||||
canonical: canonical/donut.wav
|
||||
transcript: transcripts/donut.txt
|
||||
source_sr: 44100
|
||||
duration_s: 16.3
|
||||
character: "sassy fairy-charm kid"
|
||||
provenance: "cloned from the 65-frost Booth bundle (2026-08)"
|
||||
|
||||
glados:
|
||||
canonical: canonical/glados.wav
|
||||
transcript: transcripts/glados.txt
|
||||
source_sr: 16000
|
||||
duration_s: 25.0
|
||||
character: "GLaDOS — flat, deliberate, menacing-cheerful"
|
||||
provenance: "Portal GLaDOS lines"
|
||||
warning: "LOW-SR source (16kHz) — upgrade the canonical clip if a cleaner GLaDOS source surfaces"
|
||||
|
||||
emmie:
|
||||
canonical: canonical/emmie.wav
|
||||
transcript: transcripts/emmie.txt
|
||||
source_sr: 24000
|
||||
duration_s: 19.3
|
||||
provenance: "Zonos clone added 2026-07-17; origin unrecorded"
|
||||
|
||||
miranda:
|
||||
canonical: canonical/miranda.wav
|
||||
transcript: transcripts/miranda.txt
|
||||
source_sr: 24000
|
||||
duration_s: 16.3
|
||||
provenance: "Zonos clone added 2026-07-17; origin unrecorded"
|
||||
@@ -0,0 +1 @@
|
||||
This is just not acceptable, Carl. I like my butterfly charm. It makes it so fairies like me, and it is pretty. It's part of my fit. I don't want to take it off. I don't see why I can't just wear two charms at the same time. Stupid angel of the caucus spaniel had like four or five tags.
|
||||
@@ -0,0 +1 @@
|
||||
I think I mentioned but I read your book because my my dear friend Nupa told me that I should and every now and again I would see you come up. I don't know. I take my job seriously I guess and so interviews to me felt a lot like chess and it required so much energy.
|
||||
@@ -0,0 +1 @@
|
||||
Welcome to test chamber 4. You're doing quite well. Once again, excellent work. As part of a required test protocol, we will not monitor the next test chamber. You will be entirely on your own. Good luck! As part of a required test protocol, our previous statement suggesting that we would not monitor this chamber was an outright fabrication. Good job! As part of a required test protocol, we will not monitor the next test protocol.
|
||||
@@ -0,0 +1 @@
|
||||
It's great. I mean, it's definitely comforting to go back to Australia when I come from there. So, you know, I get to see my parents, I get to see my friends and hang out. And I know the city really well because this was my fourth movie that I did in...
|
||||
Reference in New Issue
Block a user