diff --git a/persistent-memory.md b/persistent-memory.md index 49bb90f..c3f2157 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -111,7 +111,7 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-08-10 — long session; the dots.tts TTS-engine + LFM2.5 arcs LANDED (full detail per arc in Recent decisions). Genuinely in-flight is thin:_ -- **dots.tts is the new fleet RP-TTS engine — LIVE + consumer-confirmed.** v3 on irv-ml1:8198 (curly-sanitize + sentence-chunking + clause-break→period pause mapping), ratatoskr migrated + verified clean end-to-end; chatterbox-fast RECLAIMED (:8197 stopped, revive `docker start chatterbox-fast`). Canonical voice corpus in `voices/`. ⚠ WATCH: deployed container GPU creeps ~6→14GB/8h (cache accumulation; a redeploy resets it); the **pinned Dockerfile is the only rebuild path** until upstream fixes the phantom `gradio==6.17.0` constraint. Full arc → Recent decisions. +- **dots.tts is the new fleet RP-TTS engine — LIVE + consumer-confirmed.** v3 on irv-ml1:8198 (curly-sanitize + sentence-chunking + clause-break→period pause mapping), ratatoskr migrated + verified clean end-to-end; chatterbox-fast RECLAIMED (:8197 stopped, revive `docker start chatterbox-fast`). ⚠ WATCH: deployed container GPU creeps ~6→14GB/8h (cache accumulation; a redeploy resets it); the **pinned Dockerfile is the only rebuild path** until upstream fixes the phantom `gradio==6.17.0` constraint. Full arc → Recent decisions. **⤳ TTS DEV EXTRACTED to the dedicated `tts-stack` repo (`~/development/tts-stack`, 2026-08-11) — eshpfi `stacks/dots-tts/` is now a POINTER stub, `voices/` MOVED out. Do all TTS work in tts-stack; this session STOOD DOWN TTS dev.** - **LFM2.5-2.6b bake-off alias for brokkr — DELIVERED, awaiting his bench.** `vllm-lfm25` ana-ml2:8021 + LiteLLM `lfm2.5-2.6b` (deepseek_r1 reasoning-parser, vendor sampling); PRODUCTION-CLEARED (operator ASCII sanitize (fixes "Donut's"->"donut ess" on -# typographic apostrophes) + server-side sentence-chunking (long turns no longer -# truncate at dots' ~40s single-generate cap). -DOTS_TAG=v2 - -# ── network ────────────────────────────────────────────────────────── -DOTS_BIND=0.0.0.0 -DOTS_PORT=8198 - -# ── GPU ────────────────────────────────────────────────────────────── -# 0 = 3090 in Docker (PCI order), co-resident with chatterbox-fast. soar needs -# ~6GB; the 3090 has headroom with Zonos parked down. -DOTS_GPU_DEVICES=0 - -# ── model / inference ──────────────────────────────────────────────── -DOTS_MODEL=dots-studio/dots.tts-soar -DOTS_DEFAULT_VOICE=donut -DOTS_NUM_STEPS=10 # 10 = full quality @ RTF ~0.22; lower = faster/rougher -DOTS_GUIDANCE_SCALE=1.2 -DOTS_CHUNK_MAX_CHARS=280 # max chars per generate() chunk (dots caps ~40s/~500 patches) - -# ── host mounts ────────────────────────────────────────────────────── -# HF cache holding the downloaded soar snapshot (~5GB). Reuse the burn-in cache. -DOTS_HFCACHE_DIR=/home/lkraven/dots-tts/hf_cache -# dots-derived voice references (derive.py dots -> derived/dots/.{wav,txt}). -# Burn-in points at the corpus output directly; for a durable deploy, copy the -# derived set to /opt/docker/conf/dots-tts/voices and point here. -DOTS_VOICES_HOST_DIR=/home/lkraven/voice-corpus/derived/dots diff --git a/stacks/dots-tts/Dockerfile b/stacks/dots-tts/Dockerfile deleted file mode 100644 index a74c1ff..0000000 --- a/stacks/dots-tts/Dockerfile +++ /dev/null @@ -1,37 +0,0 @@ -# dots.tts OpenAI-compatible TTS server (thin FastAPI over DotsTtsRuntime). -# GPU access is via `runtime: nvidia` at run time (torch ships its own CUDA -# runtime; no CUDA toolkit / nvcc needed to build — the model uses no custom -# compiled kernels, confirmed on the irv-ml1 venv). -FROM python:3.11-slim - -RUN apt-get update && apt-get install -y --no-install-recommends \ - libsndfile1 ffmpeg git curl ca-certificates \ - && rm -rf /var/lib/apt/lists/* - -RUN pip install --no-cache-dir uv - -WORKDIR /app - -# Pin the proven-working version set (captured from the running v1 image). -# NOT using upstream constraints/recommended.txt: as of 2026-08-10 it pins -# gradio==6.17.0, which does not exist on PyPI and makes a fresh resolve -# unsatisfiable (upstream regression). dots.tts 0.2.1 pulls a working gradio -# (6.17.3) on its own; torch/numpy/soundfile pinned to the v1-image versions. -RUN uv pip install --system \ - dots.tts==0.2.1 torch==2.8.0 torchaudio==2.8.0 numpy==2.2.6 soundfile==0.13.1 \ - fastapi "uvicorn[standard]" - -# C compiler for the RUNTIME (not build): optimize=True drives torch.compile / -# inductor / triton, which JIT-compile kernels via gcc on model load. Without it -# the runtime dies with "Failed to find C compiler". Placed after the pip layer -# so it doesn't invalidate the expensive torch install cache. -RUN apt-get update && apt-get install -y --no-install-recommends build-essential \ - && rm -rf /var/lib/apt/lists/* - -COPY app.py /app/app.py - -# Persist the inductor compile cache on the mounted (rw) HF cache so kernel -# JIT doesn't re-run on every container restart (~70s warmup otherwise). -ENV HF_HOME=/hf_cache DOTS_PORT=8198 CC=gcc CXX=g++ TORCHINDUCTOR_CACHE_DIR=/hf_cache/inductor -EXPOSE 8198 -CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8198"] diff --git a/stacks/dots-tts/README.md b/stacks/dots-tts/README.md index 2289497..9ec05c3 100644 --- a/stacks/dots-tts/README.md +++ b/stacks/dots-tts/README.md @@ -1,64 +1,30 @@ -# dots-tts +# dots-tts — moved to its own repository -OpenAI-compatible zero-shot voice-clone TTS over **dots.tts** (rednote-hilab) — -2B continuous-AR, native **48kHz**, `optimize=True` CUDA graphs → **RTF ~0.22** on -the irv-ml1 3090. Thin FastAPI wrapper around `DotsTtsRuntime` (chosen over SGLang -Omni: Omni's batching is MeanFlow-only and unneeded for a single consumer; the raw -runtime already streams at the same RTF and is ~100 lines we control). +The dots.tts serving stack now lives in the dedicated **tts-stack** repo: -- **Host:** irv-ml1, port **8198** (chatterbox-fast is :8197 — they co-reside on the 3090) -- **Model:** `dots-studio/dots.tts-soar`, bf16, num_steps=10 -- **Voices:** every `.wav` (+ `.txt` transcript) in the mounted voices dir, - sourced from the [`voices/`](../../voices/) canonical corpus via `derive.py dots`. +> **`~/development/tts-stack`** → `stacks/dots-tts/` +> (gitea `vh/tts-stack` on gitea.phasefinal.com once pushed) -## API +Extracted from this workspace on 2026-08-11 so a dedicated agent can drive all TTS +tuning / development. Like chatterbox-fast, dots-tts is **authored software with a +test suite**, so it follows the sister-repo pattern rather than staying a thin +compose wrapper here. The new repo owns the code, Dockerfile, tests, the voice +corpus (formerly `voices/` here), the deploy runbook, and the running TTS +knowledge base. -``` -GET /health -> {status, model, sample_rate, voices[]} -GET /v1/voices -> {voices[]} -POST /v1/audio/speech -> audio - body: {input, voice, response_format?("wav"|"pcm"), stream?} -``` +## Deployed service -`stream:true` returns a WAV stream (placeholder-header + PCM frames, 48kHz mono -s16le) — the same shape the Zonos/chatterbox consumers already handle. Non-stream -returns a complete WAV (or raw PCM with `response_format:"pcm"`). +Live on **irv-ml1:8198** (`local/dots-tts:v3`, OpenAI `/v1/audio/speech`). The +host stack dir is `/opt/docker/compose/dots-tts/`. Deploy + rollback runbook and +all engine knowledge live in tts-stack (`docs/infrastructure.md`, `KNOWLEDGE.md`). -```bash -curl -X POST http://10.100.79.3:8198/v1/audio/speech \ - -H 'Content-Type: application/json' \ - -d '{"input":"Well, look who finally showed up.","voice":"glados"}' \ - --output out.wav -``` +## Voice corpus -## Deploy +The canonical voice corpus (was `eshpfi-management/voices/`) moved to +`tts-stack/voices/`. -Reference sets come from the canonical corpus, not this stack — derive then point -the mount at them: +## Other TTS engines -```bash -# 1. produce dots refs from the corpus (on a box with the whisper venv): -python voices/derive.py dots # -> voices/derived/dots/*.{wav,txt} - -# 2. build + run on irv-ml1 (cp .env.example .env first; adjust mounts): -scripts/deploy-stack.sh irv-ml1 dots-tts # or, on the host: -docker compose build && docker compose up -d -``` - -The model (~5GB) is **not** baked — it's read from the mounted `HF_HOME` -(`DOTS_HFCACHE_DIR`). First boot downloads it there if absent. - -## Voice cloning gotcha - -dots.tts clones from `(reference wav + its transcript)` and **leaks reference -audio into the output** if the transcript is inaccurate or ends mid-clause. The -`voices/` corpus + `derive.py` handle this (sentence-bounded trim + accurate -transcript); don't hand this server a raw reference wav without a matching `.txt`. - -## Notes - -- **GPU:** `NVIDIA_VISIBLE_DEVICES=0` = the 3090 in Docker (PCI order). `optimize=True` - is **incompatible with `PYTORCH_CUDA_ALLOC_CONF=expandable_segments`** (CUDA-graph - capture error) — don't set it. -- **Variants:** `dots.tts-mf` (MeanFlow, faster) is a drop-in via `DOTS_MODEL`; soar - is the quality pick and single-consumer doesn't need mf's batching. +The experimental TTS compose wrappers evaluated along the way (cosyvoice, dia, +kokoro, vibevoice, zonos, …) remain under `stacks/` here as reference; their +verdicts are catalogued in `tts-stack/KNOWLEDGE.md`. diff --git a/stacks/dots-tts/app.py b/stacks/dots-tts/app.py deleted file mode 100644 index ff9dbe7..0000000 --- a/stacks/dots-tts/app.py +++ /dev/null @@ -1,215 +0,0 @@ -"""OpenAI-compatible /v1/audio/speech server over dots.tts (rednote-hilab). - -Thin wrapper around DotsTtsRuntime — chosen over SGLang Omni because Omni's edge -(continuous batching) is MeanFlow-only and unneeded for a single-consumer surface, -while the raw runtime with optimize=True already streams at RTF ~0.22 on our 3090. - -Voice registry: every .wav (+ optional .txt transcript) under -DOTS_VOICES_DIR becomes a callable voice. dots.tts REQUIRES an accurate, -sentence-bounded transcript to clone cleanly (see the voices/ corpus) — the .txt -is that transcript; without it the model leaks reference audio into the output. -""" -import io -import os -import glob -import re -import struct -import threading -import wave - -import numpy as np -from fastapi import FastAPI, HTTPException -from fastapi.responses import Response, StreamingResponse -from pydantic import BaseModel - -from dots_tts.runtime import DotsTtsRuntime - -MODEL = os.environ.get("DOTS_MODEL", "dots-studio/dots.tts-soar") -VOICES_DIR = os.environ.get("DOTS_VOICES_DIR", "/voices") -DEFAULT_VOICE = os.environ.get("DOTS_DEFAULT_VOICE", "donut") -NUM_STEPS = int(os.environ.get("DOTS_NUM_STEPS", "10")) -GUIDANCE = float(os.environ.get("DOTS_GUIDANCE_SCALE", "1.2")) -SAMPLE_RATE = 48000 # dots.tts fixed native output - -# ratatoskr's LLM emits typographic (curly) punctuation, and dots' tokenizer -# mispronounces curly apostrophes ("Donut's" -> "donut ess"). Fold curly -> ASCII -# before synthesis. normalize_text stays ON (operator call — keeps number/date -# expansion); the sanitize just removes the curly trigger the model chokes on. -CURLY_MAP = str.maketrans({ - "’": "'", "‘": "'", "“": '"', "”": '"', - "–": "-", "…": "...", " ": " ", -}) -# dots caps a single generate() at ~500 audio patches (~40s). Long turns (RP -# monologues) truncate without chunking, so split into <=CHUNK_MAX_CHARS pieces -# on sentence (then clause) boundaries and stitch. A short input is one chunk = -# unchanged behavior. -CHUNK_MAX_CHARS = int(os.environ.get("DOTS_CHUNK_MAX_CHARS", "280")) - -app = FastAPI(title="dots.tts") -_rt = None -_voices: dict = {} -# One DotsTtsRuntime, and it is NOT safe to call concurrently (CUDA-graph capture -# + shared state). uvicorn runs sync endpoints in a threadpool, so we must -# serialize generation ourselves: requests queue and run one at a time. This is -# the deliberate trade for the thin-wrapper design — no vLLM-style continuous -# batching. If concurrency demand appears, swap the backend to SGLang Omni + the -# mf variant behind this same API (see README). -_gen_lock = threading.Lock() - - -def _load_voices() -> dict: - reg = {} - for wav in sorted(glob.glob(os.path.join(VOICES_DIR, "*.wav"))): - name = os.path.splitext(os.path.basename(wav))[0] - txt = os.path.splitext(wav)[0] + ".txt" - reg[name] = { - "wav": wav, - "text": open(txt).read().strip() if os.path.exists(txt) else "", - } - return reg - - -@app.on_event("startup") -def _startup(): - global _rt, _voices - _voices = _load_voices() - _rt = DotsTtsRuntime.from_pretrained(MODEL, precision="bfloat16", optimize=True) - - -@app.get("/health") -def health(): - return { - "status": "ok" if _rt is not None else "loading", - "model": MODEL, - "sample_rate": SAMPLE_RATE, - "voices": sorted(_voices), - } - - -@app.get("/v1/voices") -def list_voices(): - return {"voices": sorted(_voices)} - - -class SpeechRequest(BaseModel): - input: str - voice: str = DEFAULT_VOICE - model: str | None = None # accepted, ignored (single served model) - response_format: str = "wav" # wav | pcm - stream: bool = False - - -def _sanitize(text: str) -> str: - """Fold curly punctuation to ASCII, then map the clause breaks dots runs flat - on (semicolon, clause colon, em-dash — each measured ~+0.03s vs no pause) to a - period, which dots honors as a believable ~0.3s pause (ellipsis, at ~+0.43s, - read as too much). Number contexts are guarded: times (3:45) and ratios (2:1) - keep their colon, and en-dash ranges (folded to hyphen in CURLY_MAP) never - become "10.20".""" - text = text.translate(CURLY_MAP) - text = re.sub(r"\s*;\s*", ". ", text) # semicolon -> period - text = re.sub(r"(? list: - """Pack sentences into <=max_chars chunks (sub-splitting an over-long sentence - on commas) so each generate() stays under dots' ~40s cap. One chunk for short - input.""" - text = text.strip() - if len(text) <= max_chars: - return [text] - sentences = re.split(r"(?<=[.!?])\s+", text) - chunks, cur = [], "" - for s in sentences: - s = s.strip() - if not s: - continue - pieces = [s] - if len(s) > max_chars: # rare: a single sentence over the cap - pieces = [p.strip() for p in re.split(r"(?<=,)\s+", s) if p.strip()] - for p in pieces: - if not cur: - cur = p - elif len(cur) + 1 + len(p) <= max_chars: - cur = cur + " " + p - else: - chunks.append(cur) - cur = p - if cur: - chunks.append(cur) - return chunks or [text] - - -def _to_pcm16(audio: np.ndarray) -> bytes: - return np.round(np.clip(audio, -1.0, 1.0) * 32767.0).astype(" bytes: - buf = io.BytesIO() - w = wave.open(buf, "wb") - w.setnchannels(1) - w.setsampwidth(2) - w.setframerate(SAMPLE_RATE) - w.writeframes(pcm) - w.close() - return buf.getvalue() - - -def _streaming_wav_header() -> bytes: - """WAV header with placeholder (max) sizes — lets a client start playing the - stream before the total length is known (the pattern the Zonos/chatterbox - consumers already expect).""" - return ( - b"RIFF" + struct.pack(" derived/dots/.{wav,txt}). Read-only. - - ${DOTS_VOICES_HOST_DIR:-/home/lkraven/voice-corpus/derived/dots}:/voices:ro - healthcheck: - # /health returns {"status":"ok",...} only once the model has loaded. - test: ["CMD-SHELL", "python3 -c \"import urllib.request,sys; b=urllib.request.urlopen('http://127.0.0.1:8198/health',timeout=5).read(); sys.exit(0 if b'\\\"status\\\":\\\"ok\\\"' in b.replace(b' ',b'') else 1)\""] - interval: 30s - timeout: 10s - retries: 3 - # Model load + optimize=True CUDA-graph warmup measured ~70s; generous. - start_period: 180s - labels: - - homepage.group=AI - Speech (TTS) - - homepage.name=dots.tts - - homepage.icon=mdi-account-voice - - homepage.description=Continuous-AR 48kHz zero-shot voice clone (irv-ml1) - - homepage.href=http://10.100.79.3:${DOTS_PORT:-8198} diff --git a/stacks/dots-tts/test_sanitize.py b/stacks/dots-tts/test_sanitize.py deleted file mode 100644 index 9d4325a..0000000 --- a/stacks/dots-tts/test_sanitize.py +++ /dev/null @@ -1,73 +0,0 @@ -"""Unit tests for _sanitize — the text pre-fold dots.tts sees before synth. - -app.py imports dots_tts.runtime at module load (heavy, GPU-only), so we stub it -before import; _sanitize itself is pure and needs no model. - -Behavior under test (v3 clause-pause mapping): - * dots runs flat on ; : and em-dash (measured ~+0.03s vs none); a period gives - a believable ~0.3s clause pause. So map those clause breaks -> period. - * Guard number contexts: times (3:45) and ratios (2:1) keep their colon; - en-dash ranges (10-20) must NOT become "10.20". En-dash -> hyphen (as v2). - * Curly-apostrophe fix (the v2 reason this map exists) stays intact. - * A genuine ellipsis keeps its strong pause (-> "..."). -""" -import sys -import types - -# Stub the GPU-only runtime import so app.py loads on a CPU test box. -_stub = types.ModuleType("dots_tts.runtime") -_stub.DotsTtsRuntime = object # type: ignore[attr-defined] -sys.modules.setdefault("dots_tts", types.ModuleType("dots_tts")) -sys.modules["dots_tts.runtime"] = _stub - -import app # noqa: E402 - -s = app._sanitize - - -def test_semicolon_becomes_period(): - assert s("I waited; you left") == "I waited. you left" - - -def test_clause_colon_becomes_period(): - assert s("the truth: nobody knew") == "the truth. nobody knew" - - -def test_time_colon_preserved(): - # 3:45 must not become 3.45 ("three point four five") - assert "3:45" in s("meet me at 3:45 sharp") - - -def test_ratio_colon_preserved(): - assert "2:1" in s("the odds were 2:1 against") - - -def test_em_dash_becomes_period_spaced(): - assert s("you came — how touching") == "you came. how touching" - - -def test_em_dash_becomes_period_unspaced(): - assert s("you came—how touching") == "you came. how touching" - - -def test_en_dash_range_preserved_not_period(): - out = s("wait 10–20 minutes") - assert "10.20" not in out # the corruption we're guarding against - assert "10-20" in out # en-dash folds to hyphen (v2 behavior) - - -def test_curly_apostrophe_folds(): - # the original v2 bug: curly ' made "Donut's" -> "donut ess" - assert s("Donut’s treat") == "Donut's treat" - - -def test_curly_quotes_fold(): - assert s("“hi” there") == '"hi" there' - - -def test_ellipsis_keeps_strong_pause(): - assert s("wait… now") == "wait... now" - - -def test_no_doubled_spaces_introduced(): - assert " " not in s("a ; b : c — d") diff --git a/voices/.gitignore b/voices/.gitignore deleted file mode 100644 index 7085938..0000000 --- a/voices/.gitignore +++ /dev/null @@ -1,2 +0,0 @@ -# Per-engine reference sets are build outputs — regenerate with derive.py. -derived/ diff --git a/voices/README.md b/voices/README.md deleted file mode 100644 index 05a65b9..0000000 --- a/voices/README.md +++ /dev/null @@ -1,78 +0,0 @@ -# Canonical voice corpus - -Engine-agnostic source of truth for cloned voice identities. Each voice is -stored **once** as a canonical source clip + an accurate transcript; per-engine -reference sets (dots.tts, chatterbox, zonos, …) are **derived** from it on -demand by [`derive.py`](derive.py). Adding a new TTS engine is "add a profile -to [`engines.yaml`](engines.yaml) and re-derive" — not "re-hunt every voice." - -## Why this exists - -TTS engines disagree on what a reference clip must be: - -| Engine | SR | Transcript? | Reference shape | -|---|---|---|---| -| **dots.tts** | 48kHz | **required** | ~≤10s, **sentence-bounded**, accurate transcript | -| chatterbox-fast | 24kHz | no | any length, audio-only | -| Zonos2 | 44.1kHz | no | any length, audio-only + emotion dials | - -Keeping one canonical source per voice + a derivation step means a voice cloned -for Zonos a year ago can be re-optimized for whatever engine comes next without -re-sourcing the audio. - -## The dots.tts sensitivity finding (load-bearing) - -dots.tts conditions each generation on (reference audio **+ its transcript**) and -**regurgitates reference content into the output** when the transcript is wrong -**or ends mid-clause**. Symptoms seen during the 2026-08 burn-in: a mismatched -transcript collapsed output to 0.16s; an over-long reference with a repetitive -transcript prefixed the output with reference lines; a transcript trimmed -mid-clause ("…we will") leaked a stray "we'll" into the output. The reliable -recipe — encoded in `derive.py` for the `dots` profile — is **trim to a clean -~≤10s clip ending on a sentence boundary (. ! ?) with an accurate transcript of -exactly that clip.** - -## Layout - -``` -voices/ - manifest.yaml # voice registry: canonical path, transcript, SR, provenance - engines.yaml # per-engine reference requirements - derive.py # canonical -> derived//.{wav,txt} - canonical/.wav # source clip, best available SR (git-tracked, small + curated) - transcripts/.txt # full accurate transcript of the canonical source - derived/ # per-engine reference sets (GITIGNORED — regenerable) - dots/.{wav,txt} - chatterbox/.wav -``` - -## Usage - -```bash -# derive dots-ready references for every voice (needs faster-whisper for the trim): -python derive.py dots - -# just two voices: -python derive.py dots donut glados - -# a no-transcript engine (copies canonical; resample = follow-up, see engines.yaml): -python derive.py chatterbox -``` - -`derived/` is gitignored — treat it as a build output. Deploy a derived set to a -live engine by copying `derived//` into that stack's refs dir -(e.g. dots' voices mount, chatterbox `/worktank/chatterbox/reference_audio/`). - -## Adding a voice - -1. Drop the best available source clip in `canonical/.wav` (highest SR, - cleanest, ~10–30s is plenty). -2. Add its row to `manifest.yaml` (SR, duration, provenance). -3. `python derive.py dots ` — writes the transcript + dots reference and, - if you wire it, a verify pass. - -## Provenance discipline - -Record where each source came from in `manifest.yaml`. Unknown origin is fine to -start (`origin unrecorded`) but should be filled in when known — a canonical -corpus is only as trustworthy as its provenance. diff --git a/voices/canonical/donut.wav b/voices/canonical/donut.wav deleted file mode 100644 index aea646d..0000000 Binary files a/voices/canonical/donut.wav and /dev/null differ diff --git a/voices/canonical/emmie.wav b/voices/canonical/emmie.wav deleted file mode 100644 index a3ce463..0000000 Binary files a/voices/canonical/emmie.wav and /dev/null differ diff --git a/voices/canonical/glados.wav b/voices/canonical/glados.wav deleted file mode 100644 index e33f3e4..0000000 Binary files a/voices/canonical/glados.wav and /dev/null differ diff --git a/voices/canonical/miranda.wav b/voices/canonical/miranda.wav deleted file mode 100644 index a438faf..0000000 Binary files a/voices/canonical/miranda.wav and /dev/null differ diff --git a/voices/derive.py b/voices/derive.py deleted file mode 100644 index 1247aa5..0000000 --- a/voices/derive.py +++ /dev/null @@ -1,125 +0,0 @@ -#!/usr/bin/env python3 -"""Derive per-engine reference sets from the canonical voice corpus. - -Reads manifest.yaml + engines.yaml and, for a chosen engine, writes -derived//.wav (plus .txt when the engine needs a -transcript). - -Usage: - python derive.py [voice ...] # default: every voice in manifest - -Deps: pyyaml, soundfile. faster-whisper is imported lazily, only when an engine -sets ref_sentence_bounded (dots) — it picks a clean sentence-boundary trim and -its exact transcript. Run under a venv that has these (on irv-ml1 the dots + -whisper venvs already do). - -Known follow-up: `resample: true` engines (chatterbox, zonos) currently COPY the -canonical clip at its source SR rather than resampling — a proper resample step -(soundfile + a resampler) is a TODO. dots sets resample:false (it resamples -internally at load), so the dots path is complete. -""" -import sys -import wave -import pathlib -import shutil -import yaml - -ROOT = pathlib.Path(__file__).parent - - -def load(): - manifest = yaml.safe_load((ROOT / "manifest.yaml").read_text())["voices"] - engines = yaml.safe_load((ROOT / "engines.yaml").read_text())["engines"] - return manifest, engines - - -DANGLING = {"and", "but", "so", "or", "the", "a", "an", "that", "to", "my", - "because", "with", "of", "for", "as", "i", "we", "it", "is"} - - -def sentence_bounded_trim(src, target_s, model, min_s=6.0): - """Return (end_seconds, transcript) for a clip ending on a real sentence - boundary. - - Accumulates whisper segments and takes the FIRST point past `min_s` where the - running transcript ends in . ! ? — searching up to target_s+4 so a run-on - conversational source (no boundary early) still lands on a real sentence end - rather than a dangling clause. Only if the source has no boundary at all in - that window does it fall back to a best-effort trim with the trailing dangling - conjunction/article stripped — a partial-clause tail is exactly what dots.tts - regurgitates into its output. - """ - target_s = float(target_s) if target_s else 10.0 - hard_max = target_s + 4.0 - segs = list(model.transcribe(src, beam_size=5)[0]) - acc, best_end, best_txt = [], None, None - for s in segs: - if s.end > hard_max: - break - acc.append(s) - txt = " ".join(x.text.strip() for x in acc).strip() - if txt.endswith((".", "!", "?")): - best_end, best_txt = s.end, txt - if s.end >= min_s: - break - if best_end is not None: - return best_end, best_txt or "" - # no sentence boundary in-window — best effort, strip the dangling tail - end = acc[-1].end if acc else 0.0 - words = " ".join(x.text.strip() for x in acc).strip().rstrip(",").split() - while words and words[-1].lower().strip(",.") in DANGLING: - words.pop() - return end, " ".join(words) - - -def trim_wav(src, dst, end_s): - w = wave.open(str(src)) - sr = w.getframerate() - frames = w.readframes(int(end_s * sr)) - w.close() - o = wave.open(str(dst), "w") - o.setnchannels(1) - o.setsampwidth(2) - o.setframerate(sr) - o.writeframes(frames) - o.close() - - -def main(): - if len(sys.argv) < 2: - sys.exit("usage: derive.py [voice ...]") - engine = sys.argv[1] - manifest, engines = load() - if engine not in engines: - sys.exit(f"unknown engine '{engine}'; have {list(engines)}") - prof = engines[engine] - names = sys.argv[2:] or list(manifest) - - outdir = ROOT / "derived" / engine - outdir.mkdir(parents=True, exist_ok=True) - - model = None - if prof.get("ref_sentence_bounded"): - from faster_whisper import WhisperModel - model = WhisperModel("base.en", device="cpu", compute_type="int8") - - for v in names: - vc = manifest[v] - src = ROOT / vc["canonical"] - dst_wav = outdir / f"{v}.wav" - if prof.get("ref_sentence_bounded"): - end, txt = sentence_bounded_trim(str(src), prof.get("ref_max_seconds") or 10, model) - trim_wav(src, dst_wav, end) - if prof.get("needs_transcript"): - (outdir / f"{v}.txt").write_text(txt + "\n") - print(f"{engine}/{v}: {end:.1f}s sentence-bounded | {txt}") - else: - # TODO: resample to prof['sample_rate'] when resample:true - shutil.copy(src, dst_wav) - if prof.get("needs_transcript"): - (outdir / f"{v}.txt").write_text((ROOT / vc["transcript"]).read_text()) - print(f"{engine}/{v}: copied canonical ({vc.get('source_sr')}Hz) -> {dst_wav.name}") - - -if __name__ == "__main__": - main() diff --git a/voices/engines.yaml b/voices/engines.yaml deleted file mode 100644 index 5fe7d14..0000000 --- a/voices/engines.yaml +++ /dev/null @@ -1,40 +0,0 @@ -# Per-engine reference requirements. derive.py reads this to turn a canonical -# source + transcript into an engine-ready reference set under derived//. -# -# Fields: -# sample_rate native SR the engine wants -# resample true = derive.py should resample to sample_rate -# (NOTE: resample is a follow-up — see the resample TODO -# in derive.py; dots resamples internally so it's false there) -# needs_transcript engine requires a per-reference transcript file -# ref_max_seconds cap on derived reference length (null = uncapped) -# ref_sentence_bounded transcript/clip must end on a sentence boundary (. ! ?) -# — set for engines that leak reference content otherwise - -engines: - dots: - description: "dots.tts (rednote-hilab) — continuous-AR 48kHz zero-shot clone" - sample_rate: 48000 - resample: false # runtime auto-resamples at load; keep source SR - needs_transcript: true # REQUIRED and must be accurate + sentence-bounded - ref_max_seconds: 10 - ref_sentence_bounded: true - notes: > - Transcript accuracy AND sentence-boundary are load-bearing: a mismatched or - mid-clause transcript makes dots regurgitate reference audio into the output. - - chatterbox: - description: "chatterbox-fast (Turbo) — streaming 24kHz clone" - sample_rate: 24000 - resample: true - needs_transcript: false # audio-only clone; server globs its refs dir live - ref_max_seconds: null - ref_sentence_bounded: false - - zonos: - description: "Zonos2 — expressive 44.1kHz clone + emotion dials" - sample_rate: 44100 - resample: true - needs_transcript: false - ref_max_seconds: null - ref_sentence_bounded: false diff --git a/voices/manifest.yaml b/voices/manifest.yaml deleted file mode 100644 index 68f217a..0000000 --- a/voices/manifest.yaml +++ /dev/null @@ -1,35 +0,0 @@ -# Canonical voice corpus registry. One row per voice; the canonical clip + its -# full transcript are the source of truth, engine-agnostic. derive.py reads this -# together with engines.yaml to produce per-engine reference sets. - -voices: - donut: - canonical: canonical/donut.wav - transcript: transcripts/donut.txt - source_sr: 44100 - duration_s: 16.3 - character: "sassy fairy-charm kid" - provenance: "cloned from the 65-frost Booth bundle (2026-08)" - - glados: - canonical: canonical/glados.wav - transcript: transcripts/glados.txt - source_sr: 16000 - duration_s: 25.0 - character: "GLaDOS — flat, deliberate, menacing-cheerful" - provenance: "Portal GLaDOS lines" - warning: "LOW-SR source (16kHz) — upgrade the canonical clip if a cleaner GLaDOS source surfaces" - - emmie: - canonical: canonical/emmie.wav - transcript: transcripts/emmie.txt - source_sr: 24000 - duration_s: 19.3 - provenance: "Zonos clone added 2026-07-17; origin unrecorded" - - miranda: - canonical: canonical/miranda.wav - transcript: transcripts/miranda.txt - source_sr: 24000 - duration_s: 16.3 - provenance: "Zonos clone added 2026-07-17; origin unrecorded" diff --git a/voices/transcripts/donut.txt b/voices/transcripts/donut.txt deleted file mode 100644 index ebbd1c0..0000000 --- a/voices/transcripts/donut.txt +++ /dev/null @@ -1 +0,0 @@ -This is just not acceptable, Carl. I like my butterfly charm. It makes it so fairies like me, and it is pretty. It's part of my fit. I don't want to take it off. I don't see why I can't just wear two charms at the same time. Stupid angel of the caucus spaniel had like four or five tags. \ No newline at end of file diff --git a/voices/transcripts/emmie.txt b/voices/transcripts/emmie.txt deleted file mode 100644 index ba2dd76..0000000 --- a/voices/transcripts/emmie.txt +++ /dev/null @@ -1 +0,0 @@ -I think I mentioned but I read your book because my my dear friend Nupa told me that I should and every now and again I would see you come up. I don't know. I take my job seriously I guess and so interviews to me felt a lot like chess and it required so much energy. \ No newline at end of file diff --git a/voices/transcripts/glados.txt b/voices/transcripts/glados.txt deleted file mode 100644 index 657b36a..0000000 --- a/voices/transcripts/glados.txt +++ /dev/null @@ -1 +0,0 @@ -Welcome to test chamber 4. You're doing quite well. Once again, excellent work. As part of a required test protocol, we will not monitor the next test chamber. You will be entirely on your own. Good luck! As part of a required test protocol, our previous statement suggesting that we would not monitor this chamber was an outright fabrication. Good job! As part of a required test protocol, we will not monitor the next test protocol. \ No newline at end of file diff --git a/voices/transcripts/miranda.txt b/voices/transcripts/miranda.txt deleted file mode 100644 index b98604e..0000000 --- a/voices/transcripts/miranda.txt +++ /dev/null @@ -1 +0,0 @@ -It's great. I mean, it's definitely comforting to go back to Australia when I come from there. So, you know, I get to see my parents, I get to see my friends and hang out. And I know the city really well because this was my fourth movie that I did in... \ No newline at end of file