refactor(dots-tts): extract TTS stack to tts-stack repo; pointer stub + move voices out

TTS development moves to a dedicated repo (~/development/tts-stack) so a separate
agent can own tuning/dev. Mirrors the chatterbox-fast extraction:

- stacks/dots-tts/ reduced to a pointer README (code/Dockerfile/compose/tests/env
  now canonical in tts-stack).
- voices/ canonical corpus moved out to tts-stack/voices/. Blast-radius checked:
  no eshpfi playbook/script reads the corpus (other voices/ refs are unrelated
  host paths under /worktank/...).
- persistent-memory updated: TTS dev extracted + stood down; reverses the earlier
  "corpus home = eshpfi voices/" call.

The ~15 experimental TTS compose wrappers stay here as reference (catalogued in
tts-stack/KNOWLEDGE.md). Live service on irv-ml1:8198 is unaffected (runs from a
copy on the host).
This commit is contained in:
2026-08-11 07:46:11 -07:00
parent a80f6e958f
commit 62672c9850
20 changed files with 24 additions and 746 deletions
+3 -1
View File
@@ -111,7 +111,7 @@ no longer deployed sidecars here. See Recent decisions.)
_As of 2026-08-10 — long session; the dots.tts TTS-engine + LFM2.5 arcs LANDED (full detail per arc in Recent decisions). Genuinely in-flight is thin:_
- **dots.tts is the new fleet RP-TTS engine — LIVE + consumer-confirmed.** v3 on irv-ml1:8198 (curly-sanitize + sentence-chunking + clause-break→period pause mapping), ratatoskr migrated + verified clean end-to-end; chatterbox-fast RECLAIMED (:8197 stopped, revive `docker start chatterbox-fast`). Canonical voice corpus in `voices/`. ⚠ WATCH: deployed container GPU creeps ~6→14GB/8h (cache accumulation; a redeploy resets it); the **pinned Dockerfile is the only rebuild path** until upstream fixes the phantom `gradio==6.17.0` constraint. Full arc → Recent decisions.
- **dots.tts is the new fleet RP-TTS engine — LIVE + consumer-confirmed.** v3 on irv-ml1:8198 (curly-sanitize + sentence-chunking + clause-break→period pause mapping), ratatoskr migrated + verified clean end-to-end; chatterbox-fast RECLAIMED (:8197 stopped, revive `docker start chatterbox-fast`). ⚠ WATCH: deployed container GPU creeps ~6→14GB/8h (cache accumulation; a redeploy resets it); the **pinned Dockerfile is the only rebuild path** until upstream fixes the phantom `gradio==6.17.0` constraint. Full arc → Recent decisions. **⤳ TTS DEV EXTRACTED to the dedicated `tts-stack` repo (`~/development/tts-stack`, 2026-08-11) — eshpfi `stacks/dots-tts/` is now a POINTER stub, `voices/` MOVED out. Do all TTS work in tts-stack; this session STOOD DOWN TTS dev.**
- **LFM2.5-2.6b bake-off alias for brokkr — DELIVERED, awaiting his bench.** `vllm-lfm25` ana-ml2:8021 + LiteLLM `lfm2.5-2.6b` (deepseek_r1 reasoning-parser, vendor sampling); PRODUCTION-CLEARED (operator <USD 10M ruling) but kept OUT of default/fallback routing per brokkr's measurement gate. Peer-side now.
@@ -132,6 +132,8 @@ _As of 2026-08-10 — long session; the dots.tts TTS-engine + LFM2.5 arcs LANDED
## Recent decisions
- `[2026-08-11]` **TTS stack extracted to its own repo (`tts-stack`) + eshpfi stood down on TTS dev.** Operator: hand all TTS tuning/dev to a separate agent with a self-contained repo (knowledge + infra access + a live knowledge list), and move the voice corpus in. New repo `~/development/tts-stack` (commit `9ee3288`) carries: dots-tts stack (canonical intent), `voices/` corpus (MOVED out of eshpfi), `KNOWLEDGE.md` (engine landscape + prosody findings + foot-guns), `docs/infrastructure.md` (irv-ml1 access + gated deploy runbook + rollback), CLAUDE/persistent-memory/ROADMAP, `tools/` (pause-probe + Booth render). Followed the **chatterbox-fast precedent**: eshpfi `stacks/dots-tts/` reduced to a POINTER README; the ~15 experimental TTS compose wrappers stay here as reference (catalogued in tts-stack KNOWLEDGE). Blast-radius check: no eshpfi playbook/script reads the canonical corpus (other `voices/` refs = unrelated host paths). **Reverses** the earlier "Corpus home = eshpfi `voices/` (keep-here)" call. ⚠ tts-stack is LOCAL-ONLY until pushed — needs a gitea remote (`vh/tts-stack`) + push before the separate agent can clone (operator's call — outward-facing + repo-create creds).
- `[2026-08-10]` **dots-tts v3 — clause-break → period pause mapping.** Operator: v2 "sounds good" but donut won't pause at semicolons/dashes. ROOT CAUSE (measured via a pause-probe A/B — synth duration over N runs, non-determinism averaged out): dots' prosody honors a real pause **only for ellipsis (~+0.43s) and period (~+0.3s, capitalization-independent)**; comma/semicolon/colon/dash all run **flat (~+0.03s vs no-punct)**. Two distinct sub-causes: **dashes regressed in v2** (the `—``-` fold made em-dashes read as word-joiners), while **semicolons were NEVER a v2 change** — dots ignores them natively, only newly noticeable because v2 made everything else clean. Operator call: ellipsis "too much" → **map `;`, clause `:`, and em-dash `—` → period** in `_sanitize` (believable ~0.3s clause break). GUARDS (pinned by 11 unit tests, `stacks/dots-tts/test_sanitize.py`): digit-guarded colon `(?<!\d)\s*:\s*(?!\d)` so times `3:45` / ratios `2:1` survive; en-dash ``→hyphen KEPT (numeric-range `1020` safety — em-dash breaks, en-dash ranges, different jobs); genuine ellipsis left at full strength (author meant a long pause). Gated deploy (redeploy2 pattern → v3): build → throwaway :8199 test container + **pause-gate** (semicolon sentence must run ≥0.12s longer than baseline; measured **+0.427s**) → only then cut live over. LIVE + healthy `local/dots-tts:v3` on :8198. **rollback = `sed -i 's/^DOTS_TAG=.*/DOTS_TAG=v2/' .env + docker compose up -d dots-tts`** (v2 image retained). Booth `dots-pauses` (A=old-flat / C=ellipsis-too-much / D=live-v3). [[reference_chatterbox_fast_repo]]
- `[2026-08-10]` **dots-tts v2 — contraction fix (curly-sanitize) + sentence-chunking + dependency-pin recovery.** Operator: donut read contractions wrong ("you're"→"you ree", "donut's"→"donut ess"). ROOT CAUSE (isolated via A/B booth): **curly/typographic apostrophes** (`` U+2019 from ratatoskr's LLM) — dots' tokenizer mispronounces them; STRAIGHT apostrophes read clean under `normalize_text=True`. FIX (`app.py`): fold curly→ASCII (`str.maketrans`) before synth, **KEEP `normalize_text=True`** (operator call — retains number/date expansion). Also added **server-side sentence-chunking** (pack ≤280 chars): dots caps one `generate()` at ~500 patches/~40s, so long RP turns (the Zev monologue = 160s audio) truncated; chunking stitches them (verified full 160.3s, not 40s-cut). **⚠ BUILD FOOT-GUNS (both bit this redeploy):** (1) upstream dots.tts `constraints/recommended.txt` now pins **`gradio==6.17.0` — phantom, not on PyPI** → fresh `pip install dots.tts` unsatisfiable; FIX = pin `dots.tts==0.2.1` + **DROP** the `-c recommended.txt` constraints (0.2.1 pulls working gradio 6.17.3). (2) pinning only `torch==2.8.0` let **torchaudio float to 2.11.0 → dots.tts refuses to load** (minor-version match check); FIX = pin `torchaudio==2.8.0`. **⚠ DEPLOY LESSON:** `docker compose up -d` to a new tag swaps the LIVE container BEFORE any health check — a broken image crash-loops production (**ratatoskr TTS down ~1-2min this session**). NEW PATTERN = build → test in a THROWAWAY container on an alt port (:8199) → health+verify → only THEN cut live over (redeploy2.sh). v2 LIVE + healthy on irv-ml1:8198, **CONSUMER-CONFIRMED clean** (ratatoskr verified end-to-end on their :8765 — apostrophe string reads clean, /api/tts 200 @ 48kHz, no client change; the ~1-2min blip didn't hit them, their concurrent auto-audio issue was client-side localStorage). **rollback = `sed DOTS_TAG=v1 + docker compose up -d dots-tts`** (v1 image retained). Also: deployed container GPU crept ~6→13.9GB over 8h serving (cache accumulation; a redeploy resets it — watch item). [[reference_chatterbox_fast_repo]]
-31
View File
@@ -1,31 +0,0 @@
# dots-tts stack tunables. Copy to `.env` on irv-ml1 before deploying.
# ── image ────────────────────────────────────────────────────────────
# v2 (2026-08-10): curly->ASCII sanitize (fixes "Donut's"->"donut ess" on
# typographic apostrophes) + server-side sentence-chunking (long turns no longer
# truncate at dots' ~40s single-generate cap).
DOTS_TAG=v2
# ── network ──────────────────────────────────────────────────────────
DOTS_BIND=0.0.0.0
DOTS_PORT=8198
# ── GPU ──────────────────────────────────────────────────────────────
# 0 = 3090 in Docker (PCI order), co-resident with chatterbox-fast. soar needs
# ~6GB; the 3090 has headroom with Zonos parked down.
DOTS_GPU_DEVICES=0
# ── model / inference ────────────────────────────────────────────────
DOTS_MODEL=dots-studio/dots.tts-soar
DOTS_DEFAULT_VOICE=donut
DOTS_NUM_STEPS=10 # 10 = full quality @ RTF ~0.22; lower = faster/rougher
DOTS_GUIDANCE_SCALE=1.2
DOTS_CHUNK_MAX_CHARS=280 # max chars per generate() chunk (dots caps ~40s/~500 patches)
# ── host mounts ──────────────────────────────────────────────────────
# HF cache holding the downloaded soar snapshot (~5GB). Reuse the burn-in cache.
DOTS_HFCACHE_DIR=/home/lkraven/dots-tts/hf_cache
# dots-derived voice references (derive.py dots -> derived/dots/<name>.{wav,txt}).
# Burn-in points at the corpus output directly; for a durable deploy, copy the
# derived set to /opt/docker/conf/dots-tts/voices and point here.
DOTS_VOICES_HOST_DIR=/home/lkraven/voice-corpus/derived/dots
-37
View File
@@ -1,37 +0,0 @@
# dots.tts OpenAI-compatible TTS server (thin FastAPI over DotsTtsRuntime).
# GPU access is via `runtime: nvidia` at run time (torch ships its own CUDA
# runtime; no CUDA toolkit / nvcc needed to build — the model uses no custom
# compiled kernels, confirmed on the irv-ml1 venv).
FROM python:3.11-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
libsndfile1 ffmpeg git curl ca-certificates \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir uv
WORKDIR /app
# Pin the proven-working version set (captured from the running v1 image).
# NOT using upstream constraints/recommended.txt: as of 2026-08-10 it pins
# gradio==6.17.0, which does not exist on PyPI and makes a fresh resolve
# unsatisfiable (upstream regression). dots.tts 0.2.1 pulls a working gradio
# (6.17.3) on its own; torch/numpy/soundfile pinned to the v1-image versions.
RUN uv pip install --system \
dots.tts==0.2.1 torch==2.8.0 torchaudio==2.8.0 numpy==2.2.6 soundfile==0.13.1 \
fastapi "uvicorn[standard]"
# C compiler for the RUNTIME (not build): optimize=True drives torch.compile /
# inductor / triton, which JIT-compile kernels via gcc on model load. Without it
# the runtime dies with "Failed to find C compiler". Placed after the pip layer
# so it doesn't invalidate the expensive torch install cache.
RUN apt-get update && apt-get install -y --no-install-recommends build-essential \
&& rm -rf /var/lib/apt/lists/*
COPY app.py /app/app.py
# Persist the inductor compile cache on the mounted (rw) HF cache so kernel
# JIT doesn't re-run on every container restart (~70s warmup otherwise).
ENV HF_HOME=/hf_cache DOTS_PORT=8198 CC=gcc CXX=g++ TORCHINDUCTOR_CACHE_DIR=/hf_cache/inductor
EXPOSE 8198
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8198"]
+21 -55
View File
@@ -1,64 +1,30 @@
# dots-tts
# dots-tts — moved to its own repository
OpenAI-compatible zero-shot voice-clone TTS over **dots.tts** (rednote-hilab) —
2B continuous-AR, native **48kHz**, `optimize=True` CUDA graphs → **RTF ~0.22** on
the irv-ml1 3090. Thin FastAPI wrapper around `DotsTtsRuntime` (chosen over SGLang
Omni: Omni's batching is MeanFlow-only and unneeded for a single consumer; the raw
runtime already streams at the same RTF and is ~100 lines we control).
The dots.tts serving stack now lives in the dedicated **tts-stack** repo:
- **Host:** irv-ml1, port **8198** (chatterbox-fast is :8197 — they co-reside on the 3090)
- **Model:** `dots-studio/dots.tts-soar`, bf16, num_steps=10
- **Voices:** every `<name>.wav` (+ `<name>.txt` transcript) in the mounted voices dir,
sourced from the [`voices/`](../../voices/) canonical corpus via `derive.py dots`.
> **`~/development/tts-stack`** → `stacks/dots-tts/`
> (gitea `vh/tts-stack` on gitea.phasefinal.com once pushed)
## API
Extracted from this workspace on 2026-08-11 so a dedicated agent can drive all TTS
tuning / development. Like chatterbox-fast, dots-tts is **authored software with a
test suite**, so it follows the sister-repo pattern rather than staying a thin
compose wrapper here. The new repo owns the code, Dockerfile, tests, the voice
corpus (formerly `voices/` here), the deploy runbook, and the running TTS
knowledge base.
```
GET /health -> {status, model, sample_rate, voices[]}
GET /v1/voices -> {voices[]}
POST /v1/audio/speech -> audio
body: {input, voice, response_format?("wav"|"pcm"), stream?}
```
## Deployed service
`stream:true` returns a WAV stream (placeholder-header + PCM frames, 48kHz mono
s16le) — the same shape the Zonos/chatterbox consumers already handle. Non-stream
returns a complete WAV (or raw PCM with `response_format:"pcm"`).
Live on **irv-ml1:8198** (`local/dots-tts:v3`, OpenAI `/v1/audio/speech`). The
host stack dir is `/opt/docker/compose/dots-tts/`. Deploy + rollback runbook and
all engine knowledge live in tts-stack (`docs/infrastructure.md`, `KNOWLEDGE.md`).
```bash
curl -X POST http://10.100.79.3:8198/v1/audio/speech \
-H 'Content-Type: application/json' \
-d '{"input":"Well, look who finally showed up.","voice":"glados"}' \
--output out.wav
```
## Voice corpus
## Deploy
The canonical voice corpus (was `eshpfi-management/voices/`) moved to
`tts-stack/voices/`.
Reference sets come from the canonical corpus, not this stack — derive then point
the mount at them:
## Other TTS engines
```bash
# 1. produce dots refs from the corpus (on a box with the whisper venv):
python voices/derive.py dots # -> voices/derived/dots/*.{wav,txt}
# 2. build + run on irv-ml1 (cp .env.example .env first; adjust mounts):
scripts/deploy-stack.sh irv-ml1 dots-tts # or, on the host:
docker compose build && docker compose up -d
```
The model (~5GB) is **not** baked — it's read from the mounted `HF_HOME`
(`DOTS_HFCACHE_DIR`). First boot downloads it there if absent.
## Voice cloning gotcha
dots.tts clones from `(reference wav + its transcript)` and **leaks reference
audio into the output** if the transcript is inaccurate or ends mid-clause. The
`voices/` corpus + `derive.py` handle this (sentence-bounded trim + accurate
transcript); don't hand this server a raw reference wav without a matching `.txt`.
## Notes
- **GPU:** `NVIDIA_VISIBLE_DEVICES=0` = the 3090 in Docker (PCI order). `optimize=True`
is **incompatible with `PYTORCH_CUDA_ALLOC_CONF=expandable_segments`** (CUDA-graph
capture error) — don't set it.
- **Variants:** `dots.tts-mf` (MeanFlow, faster) is a drop-in via `DOTS_MODEL`; soar
is the quality pick and single-consumer doesn't need mf's batching.
The experimental TTS compose wrappers evaluated along the way (cosyvoice, dia,
kokoro, vibevoice, zonos, …) remain under `stacks/` here as reference; their
verdicts are catalogued in `tts-stack/KNOWLEDGE.md`.
-215
View File
@@ -1,215 +0,0 @@
"""OpenAI-compatible /v1/audio/speech server over dots.tts (rednote-hilab).
Thin wrapper around DotsTtsRuntime — chosen over SGLang Omni because Omni's edge
(continuous batching) is MeanFlow-only and unneeded for a single-consumer surface,
while the raw runtime with optimize=True already streams at RTF ~0.22 on our 3090.
Voice registry: every <name>.wav (+ optional <name>.txt transcript) under
DOTS_VOICES_DIR becomes a callable voice. dots.tts REQUIRES an accurate,
sentence-bounded transcript to clone cleanly (see the voices/ corpus) — the .txt
is that transcript; without it the model leaks reference audio into the output.
"""
import io
import os
import glob
import re
import struct
import threading
import wave
import numpy as np
from fastapi import FastAPI, HTTPException
from fastapi.responses import Response, StreamingResponse
from pydantic import BaseModel
from dots_tts.runtime import DotsTtsRuntime
MODEL = os.environ.get("DOTS_MODEL", "dots-studio/dots.tts-soar")
VOICES_DIR = os.environ.get("DOTS_VOICES_DIR", "/voices")
DEFAULT_VOICE = os.environ.get("DOTS_DEFAULT_VOICE", "donut")
NUM_STEPS = int(os.environ.get("DOTS_NUM_STEPS", "10"))
GUIDANCE = float(os.environ.get("DOTS_GUIDANCE_SCALE", "1.2"))
SAMPLE_RATE = 48000 # dots.tts fixed native output
# ratatoskr's LLM emits typographic (curly) punctuation, and dots' tokenizer
# mispronounces curly apostrophes ("Donut's" -> "donut ess"). Fold curly -> ASCII
# before synthesis. normalize_text stays ON (operator call — keeps number/date
# expansion); the sanitize just removes the curly trigger the model chokes on.
CURLY_MAP = str.maketrans({
"": "'", "": "'", "": '"', "": '"',
"": "-", "": "...", " ": " ",
})
# dots caps a single generate() at ~500 audio patches (~40s). Long turns (RP
# monologues) truncate without chunking, so split into <=CHUNK_MAX_CHARS pieces
# on sentence (then clause) boundaries and stitch. A short input is one chunk =
# unchanged behavior.
CHUNK_MAX_CHARS = int(os.environ.get("DOTS_CHUNK_MAX_CHARS", "280"))
app = FastAPI(title="dots.tts")
_rt = None
_voices: dict = {}
# One DotsTtsRuntime, and it is NOT safe to call concurrently (CUDA-graph capture
# + shared state). uvicorn runs sync endpoints in a threadpool, so we must
# serialize generation ourselves: requests queue and run one at a time. This is
# the deliberate trade for the thin-wrapper design — no vLLM-style continuous
# batching. If concurrency demand appears, swap the backend to SGLang Omni + the
# mf variant behind this same API (see README).
_gen_lock = threading.Lock()
def _load_voices() -> dict:
reg = {}
for wav in sorted(glob.glob(os.path.join(VOICES_DIR, "*.wav"))):
name = os.path.splitext(os.path.basename(wav))[0]
txt = os.path.splitext(wav)[0] + ".txt"
reg[name] = {
"wav": wav,
"text": open(txt).read().strip() if os.path.exists(txt) else "",
}
return reg
@app.on_event("startup")
def _startup():
global _rt, _voices
_voices = _load_voices()
_rt = DotsTtsRuntime.from_pretrained(MODEL, precision="bfloat16", optimize=True)
@app.get("/health")
def health():
return {
"status": "ok" if _rt is not None else "loading",
"model": MODEL,
"sample_rate": SAMPLE_RATE,
"voices": sorted(_voices),
}
@app.get("/v1/voices")
def list_voices():
return {"voices": sorted(_voices)}
class SpeechRequest(BaseModel):
input: str
voice: str = DEFAULT_VOICE
model: str | None = None # accepted, ignored (single served model)
response_format: str = "wav" # wav | pcm
stream: bool = False
def _sanitize(text: str) -> str:
"""Fold curly punctuation to ASCII, then map the clause breaks dots runs flat
on (semicolon, clause colon, em-dash — each measured ~+0.03s vs no pause) to a
period, which dots honors as a believable ~0.3s pause (ellipsis, at ~+0.43s,
read as too much). Number contexts are guarded: times (3:45) and ratios (2:1)
keep their colon, and en-dash ranges (folded to hyphen in CURLY_MAP) never
become "10.20"."""
text = text.translate(CURLY_MAP)
text = re.sub(r"\s*;\s*", ". ", text) # semicolon -> period
text = re.sub(r"(?<!\d)\s*:\s*(?!\d)", ". ", text) # clause colon (not 3:45)
text = re.sub(r"\s*—\s*", ". ", text) # em-dash clause break
return text
def _chunk(text: str, max_chars: int = CHUNK_MAX_CHARS) -> list:
"""Pack sentences into <=max_chars chunks (sub-splitting an over-long sentence
on commas) so each generate() stays under dots' ~40s cap. One chunk for short
input."""
text = text.strip()
if len(text) <= max_chars:
return [text]
sentences = re.split(r"(?<=[.!?])\s+", text)
chunks, cur = [], ""
for s in sentences:
s = s.strip()
if not s:
continue
pieces = [s]
if len(s) > max_chars: # rare: a single sentence over the cap
pieces = [p.strip() for p in re.split(r"(?<=,)\s+", s) if p.strip()]
for p in pieces:
if not cur:
cur = p
elif len(cur) + 1 + len(p) <= max_chars:
cur = cur + " " + p
else:
chunks.append(cur)
cur = p
if cur:
chunks.append(cur)
return chunks or [text]
def _to_pcm16(audio: np.ndarray) -> bytes:
return np.round(np.clip(audio, -1.0, 1.0) * 32767.0).astype("<i2").tobytes()
def _wav_bytes(pcm: bytes) -> bytes:
buf = io.BytesIO()
w = wave.open(buf, "wb")
w.setnchannels(1)
w.setsampwidth(2)
w.setframerate(SAMPLE_RATE)
w.writeframes(pcm)
w.close()
return buf.getvalue()
def _streaming_wav_header() -> bytes:
"""WAV header with placeholder (max) sizes — lets a client start playing the
stream before the total length is known (the pattern the Zonos/chatterbox
consumers already expect)."""
return (
b"RIFF" + struct.pack("<I", 0xFFFFFFFF) + b"WAVE"
+ b"fmt " + struct.pack("<IHHIIHH", 16, 1, 1, SAMPLE_RATE, SAMPLE_RATE * 2, 2, 16)
+ b"data" + struct.pack("<I", 0xFFFFFFFF)
)
@app.post("/v1/audio/speech")
def speech(req: SpeechRequest):
rt = _rt
if rt is None:
raise HTTPException(503, "model still loading")
if req.voice not in _voices:
raise HTTPException(404, f"unknown voice '{req.voice}'; have {sorted(_voices)}")
if not req.input.strip():
raise HTTPException(400, "empty input")
v = _voices[req.voice]
text = _sanitize(req.input)
chunks = _chunk(text)
kw = dict(
prompt_audio_path=v["wav"],
prompt_text=v["text"],
num_steps=NUM_STEPS,
guidance_scale=GUIDANCE,
normalize_text=True,
)
gap = np.zeros(int(0.08 * SAMPLE_RATE), dtype=np.float32) # 80ms seam between chunks
if req.stream:
def gen():
# Hold the lock for the whole stream — a second generation on the
# shared runtime mid-stream would corrupt both.
with _gen_lock:
yield _streaming_wav_header()
for i, ch in enumerate(chunks):
if i:
yield _to_pcm16(gap)
for piece in rt.generate_stream(text=ch, **kw):
yield _to_pcm16(piece.float().cpu().squeeze().numpy())
return StreamingResponse(gen(), media_type="audio/wav")
with _gen_lock:
parts = []
for i, ch in enumerate(chunks):
if i:
parts.append(gap)
parts.append(rt.generate(text=ch, **kw)["audio"].float().cpu().squeeze().numpy())
pcm = _to_pcm16(np.concatenate(parts))
if req.response_format == "pcm":
return Response(pcm, media_type="audio/L16;rate=48000")
return Response(_wav_bytes(pcm), media_type="audio/wav")
-50
View File
@@ -1,50 +0,0 @@
# dots.tts — OpenAI-compatible 48kHz zero-shot voice-clone TTS (rednote-hilab).
# Deployed on irv-ml1 ALONGSIDE chatterbox-fast (:8197) for burn-in; both share
# the 3090. Mirrors the chatterbox-fast sibling: GPU via `runtime: nvidia` +
# NVIDIA_VISIBLE_DEVICES, host IP:port (no traefik). All tunables in .env.
#
# GPU note: in Docker, NVIDIA_VISIBLE_DEVICES=0 resolves to the 3090 (PCI order),
# same as chatterbox-fast — NOT the A6000 that *native* CUDA calls device 0 on
# this host. CUDA_DEVICE_ORDER=PCI_BUS_ID is set belt-and-suspenders.
services:
dots-tts:
image: local/dots-tts:${DOTS_TAG:-v1}
build:
context: .
dockerfile: Dockerfile
container_name: dots-tts
restart: unless-stopped
runtime: nvidia
ports:
- "${DOTS_BIND:-0.0.0.0}:${DOTS_PORT:-8198}:8198"
environment:
- NVIDIA_VISIBLE_DEVICES=${DOTS_GPU_DEVICES:-0}
- CUDA_DEVICE_ORDER=PCI_BUS_ID
- HF_HOME=/hf_cache
- DOTS_MODEL=${DOTS_MODEL:-dots-studio/dots.tts-soar}
- DOTS_VOICES_DIR=/voices
- DOTS_DEFAULT_VOICE=${DOTS_DEFAULT_VOICE:-donut}
- DOTS_NUM_STEPS=${DOTS_NUM_STEPS:-10}
- DOTS_GUIDANCE_SCALE=${DOTS_GUIDANCE_SCALE:-1.2}
- DOTS_PORT=8198
volumes:
# HF model cache — reuse the already-downloaded soar snapshot (~5GB).
- ${DOTS_HFCACHE_DIR:-/home/lkraven/dots-tts/hf_cache}:/hf_cache
# Voice references — the dots-derived set from the voices/ corpus
# (derive.py dots -> derived/dots/<name>.{wav,txt}). Read-only.
- ${DOTS_VOICES_HOST_DIR:-/home/lkraven/voice-corpus/derived/dots}:/voices:ro
healthcheck:
# /health returns {"status":"ok",...} only once the model has loaded.
test: ["CMD-SHELL", "python3 -c \"import urllib.request,sys; b=urllib.request.urlopen('http://127.0.0.1:8198/health',timeout=5).read(); sys.exit(0 if b'\\\"status\\\":\\\"ok\\\"' in b.replace(b' ',b'') else 1)\""]
interval: 30s
timeout: 10s
retries: 3
# Model load + optimize=True CUDA-graph warmup measured ~70s; generous.
start_period: 180s
labels:
- homepage.group=AI - Speech (TTS)
- homepage.name=dots.tts
- homepage.icon=mdi-account-voice
- homepage.description=Continuous-AR 48kHz zero-shot voice clone (irv-ml1)
- homepage.href=http://10.100.79.3:${DOTS_PORT:-8198}
-73
View File
@@ -1,73 +0,0 @@
"""Unit tests for _sanitize — the text pre-fold dots.tts sees before synth.
app.py imports dots_tts.runtime at module load (heavy, GPU-only), so we stub it
before import; _sanitize itself is pure and needs no model.
Behavior under test (v3 clause-pause mapping):
* dots runs flat on ; : and em-dash (measured ~+0.03s vs none); a period gives
a believable ~0.3s clause pause. So map those clause breaks -> period.
* Guard number contexts: times (3:45) and ratios (2:1) keep their colon;
en-dash ranges (10-20) must NOT become "10.20". En-dash -> hyphen (as v2).
* Curly-apostrophe fix (the v2 reason this map exists) stays intact.
* A genuine ellipsis keeps its strong pause (-> "...").
"""
import sys
import types
# Stub the GPU-only runtime import so app.py loads on a CPU test box.
_stub = types.ModuleType("dots_tts.runtime")
_stub.DotsTtsRuntime = object # type: ignore[attr-defined]
sys.modules.setdefault("dots_tts", types.ModuleType("dots_tts"))
sys.modules["dots_tts.runtime"] = _stub
import app # noqa: E402
s = app._sanitize
def test_semicolon_becomes_period():
assert s("I waited; you left") == "I waited. you left"
def test_clause_colon_becomes_period():
assert s("the truth: nobody knew") == "the truth. nobody knew"
def test_time_colon_preserved():
# 3:45 must not become 3.45 ("three point four five")
assert "3:45" in s("meet me at 3:45 sharp")
def test_ratio_colon_preserved():
assert "2:1" in s("the odds were 2:1 against")
def test_em_dash_becomes_period_spaced():
assert s("you came — how touching") == "you came. how touching"
def test_em_dash_becomes_period_unspaced():
assert s("you came—how touching") == "you came. how touching"
def test_en_dash_range_preserved_not_period():
out = s("wait 1020 minutes")
assert "10.20" not in out # the corruption we're guarding against
assert "10-20" in out # en-dash folds to hyphen (v2 behavior)
def test_curly_apostrophe_folds():
# the original v2 bug: curly ' made "Donut's" -> "donut ess"
assert s("Donuts treat") == "Donut's treat"
def test_curly_quotes_fold():
assert s("“hi” there") == '"hi" there'
def test_ellipsis_keeps_strong_pause():
assert s("wait… now") == "wait... now"
def test_no_doubled_spaces_introduced():
assert " " not in s("a ; b : c — d")
-2
View File
@@ -1,2 +0,0 @@
# Per-engine reference sets are build outputs — regenerate with derive.py.
derived/
-78
View File
@@ -1,78 +0,0 @@
# Canonical voice corpus
Engine-agnostic source of truth for cloned voice identities. Each voice is
stored **once** as a canonical source clip + an accurate transcript; per-engine
reference sets (dots.tts, chatterbox, zonos, …) are **derived** from it on
demand by [`derive.py`](derive.py). Adding a new TTS engine is "add a profile
to [`engines.yaml`](engines.yaml) and re-derive" — not "re-hunt every voice."
## Why this exists
TTS engines disagree on what a reference clip must be:
| Engine | SR | Transcript? | Reference shape |
|---|---|---|---|
| **dots.tts** | 48kHz | **required** | ~≤10s, **sentence-bounded**, accurate transcript |
| chatterbox-fast | 24kHz | no | any length, audio-only |
| Zonos2 | 44.1kHz | no | any length, audio-only + emotion dials |
Keeping one canonical source per voice + a derivation step means a voice cloned
for Zonos a year ago can be re-optimized for whatever engine comes next without
re-sourcing the audio.
## The dots.tts sensitivity finding (load-bearing)
dots.tts conditions each generation on (reference audio **+ its transcript**) and
**regurgitates reference content into the output** when the transcript is wrong
**or ends mid-clause**. Symptoms seen during the 2026-08 burn-in: a mismatched
transcript collapsed output to 0.16s; an over-long reference with a repetitive
transcript prefixed the output with reference lines; a transcript trimmed
mid-clause ("…we will") leaked a stray "we'll" into the output. The reliable
recipe — encoded in `derive.py` for the `dots` profile — is **trim to a clean
~≤10s clip ending on a sentence boundary (. ! ?) with an accurate transcript of
exactly that clip.**
## Layout
```
voices/
manifest.yaml # voice registry: canonical path, transcript, SR, provenance
engines.yaml # per-engine reference requirements
derive.py # canonical -> derived/<engine>/<voice>.{wav,txt}
canonical/<v>.wav # source clip, best available SR (git-tracked, small + curated)
transcripts/<v>.txt # full accurate transcript of the canonical source
derived/ # per-engine reference sets (GITIGNORED — regenerable)
dots/<v>.{wav,txt}
chatterbox/<v>.wav
```
## Usage
```bash
# derive dots-ready references for every voice (needs faster-whisper for the trim):
python derive.py dots
# just two voices:
python derive.py dots donut glados
# a no-transcript engine (copies canonical; resample = follow-up, see engines.yaml):
python derive.py chatterbox
```
`derived/` is gitignored — treat it as a build output. Deploy a derived set to a
live engine by copying `derived/<engine>/` into that stack's refs dir
(e.g. dots' voices mount, chatterbox `/worktank/chatterbox/reference_audio/`).
## Adding a voice
1. Drop the best available source clip in `canonical/<name>.wav` (highest SR,
cleanest, ~1030s is plenty).
2. Add its row to `manifest.yaml` (SR, duration, provenance).
3. `python derive.py dots <name>` — writes the transcript + dots reference and,
if you wire it, a verify pass.
## Provenance discipline
Record where each source came from in `manifest.yaml`. Unknown origin is fine to
start (`origin unrecorded`) but should be filled in when known — a canonical
corpus is only as trustworthy as its provenance.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
-125
View File
@@ -1,125 +0,0 @@
#!/usr/bin/env python3
"""Derive per-engine reference sets from the canonical voice corpus.
Reads manifest.yaml + engines.yaml and, for a chosen engine, writes
derived/<engine>/<voice>.wav (plus <voice>.txt when the engine needs a
transcript).
Usage:
python derive.py <engine> [voice ...] # default: every voice in manifest
Deps: pyyaml, soundfile. faster-whisper is imported lazily, only when an engine
sets ref_sentence_bounded (dots) — it picks a clean sentence-boundary trim and
its exact transcript. Run under a venv that has these (on irv-ml1 the dots +
whisper venvs already do).
Known follow-up: `resample: true` engines (chatterbox, zonos) currently COPY the
canonical clip at its source SR rather than resampling — a proper resample step
(soundfile + a resampler) is a TODO. dots sets resample:false (it resamples
internally at load), so the dots path is complete.
"""
import sys
import wave
import pathlib
import shutil
import yaml
ROOT = pathlib.Path(__file__).parent
def load():
manifest = yaml.safe_load((ROOT / "manifest.yaml").read_text())["voices"]
engines = yaml.safe_load((ROOT / "engines.yaml").read_text())["engines"]
return manifest, engines
DANGLING = {"and", "but", "so", "or", "the", "a", "an", "that", "to", "my",
"because", "with", "of", "for", "as", "i", "we", "it", "is"}
def sentence_bounded_trim(src, target_s, model, min_s=6.0):
"""Return (end_seconds, transcript) for a clip ending on a real sentence
boundary.
Accumulates whisper segments and takes the FIRST point past `min_s` where the
running transcript ends in . ! ? — searching up to target_s+4 so a run-on
conversational source (no boundary early) still lands on a real sentence end
rather than a dangling clause. Only if the source has no boundary at all in
that window does it fall back to a best-effort trim with the trailing dangling
conjunction/article stripped — a partial-clause tail is exactly what dots.tts
regurgitates into its output.
"""
target_s = float(target_s) if target_s else 10.0
hard_max = target_s + 4.0
segs = list(model.transcribe(src, beam_size=5)[0])
acc, best_end, best_txt = [], None, None
for s in segs:
if s.end > hard_max:
break
acc.append(s)
txt = " ".join(x.text.strip() for x in acc).strip()
if txt.endswith((".", "!", "?")):
best_end, best_txt = s.end, txt
if s.end >= min_s:
break
if best_end is not None:
return best_end, best_txt or ""
# no sentence boundary in-window — best effort, strip the dangling tail
end = acc[-1].end if acc else 0.0
words = " ".join(x.text.strip() for x in acc).strip().rstrip(",").split()
while words and words[-1].lower().strip(",.") in DANGLING:
words.pop()
return end, " ".join(words)
def trim_wav(src, dst, end_s):
w = wave.open(str(src))
sr = w.getframerate()
frames = w.readframes(int(end_s * sr))
w.close()
o = wave.open(str(dst), "w")
o.setnchannels(1)
o.setsampwidth(2)
o.setframerate(sr)
o.writeframes(frames)
o.close()
def main():
if len(sys.argv) < 2:
sys.exit("usage: derive.py <engine> [voice ...]")
engine = sys.argv[1]
manifest, engines = load()
if engine not in engines:
sys.exit(f"unknown engine '{engine}'; have {list(engines)}")
prof = engines[engine]
names = sys.argv[2:] or list(manifest)
outdir = ROOT / "derived" / engine
outdir.mkdir(parents=True, exist_ok=True)
model = None
if prof.get("ref_sentence_bounded"):
from faster_whisper import WhisperModel
model = WhisperModel("base.en", device="cpu", compute_type="int8")
for v in names:
vc = manifest[v]
src = ROOT / vc["canonical"]
dst_wav = outdir / f"{v}.wav"
if prof.get("ref_sentence_bounded"):
end, txt = sentence_bounded_trim(str(src), prof.get("ref_max_seconds") or 10, model)
trim_wav(src, dst_wav, end)
if prof.get("needs_transcript"):
(outdir / f"{v}.txt").write_text(txt + "\n")
print(f"{engine}/{v}: {end:.1f}s sentence-bounded | {txt}")
else:
# TODO: resample to prof['sample_rate'] when resample:true
shutil.copy(src, dst_wav)
if prof.get("needs_transcript"):
(outdir / f"{v}.txt").write_text((ROOT / vc["transcript"]).read_text())
print(f"{engine}/{v}: copied canonical ({vc.get('source_sr')}Hz) -> {dst_wav.name}")
if __name__ == "__main__":
main()
-40
View File
@@ -1,40 +0,0 @@
# Per-engine reference requirements. derive.py reads this to turn a canonical
# source + transcript into an engine-ready reference set under derived/<engine>/.
#
# Fields:
# sample_rate native SR the engine wants
# resample true = derive.py should resample to sample_rate
# (NOTE: resample is a follow-up — see the resample TODO
# in derive.py; dots resamples internally so it's false there)
# needs_transcript engine requires a per-reference transcript file
# ref_max_seconds cap on derived reference length (null = uncapped)
# ref_sentence_bounded transcript/clip must end on a sentence boundary (. ! ?)
# — set for engines that leak reference content otherwise
engines:
dots:
description: "dots.tts (rednote-hilab) — continuous-AR 48kHz zero-shot clone"
sample_rate: 48000
resample: false # runtime auto-resamples at load; keep source SR
needs_transcript: true # REQUIRED and must be accurate + sentence-bounded
ref_max_seconds: 10
ref_sentence_bounded: true
notes: >
Transcript accuracy AND sentence-boundary are load-bearing: a mismatched or
mid-clause transcript makes dots regurgitate reference audio into the output.
chatterbox:
description: "chatterbox-fast (Turbo) — streaming 24kHz clone"
sample_rate: 24000
resample: true
needs_transcript: false # audio-only clone; server globs its refs dir live
ref_max_seconds: null
ref_sentence_bounded: false
zonos:
description: "Zonos2 — expressive 44.1kHz clone + emotion dials"
sample_rate: 44100
resample: true
needs_transcript: false
ref_max_seconds: null
ref_sentence_bounded: false
-35
View File
@@ -1,35 +0,0 @@
# Canonical voice corpus registry. One row per voice; the canonical clip + its
# full transcript are the source of truth, engine-agnostic. derive.py reads this
# together with engines.yaml to produce per-engine reference sets.
voices:
donut:
canonical: canonical/donut.wav
transcript: transcripts/donut.txt
source_sr: 44100
duration_s: 16.3
character: "sassy fairy-charm kid"
provenance: "cloned from the 65-frost Booth bundle (2026-08)"
glados:
canonical: canonical/glados.wav
transcript: transcripts/glados.txt
source_sr: 16000
duration_s: 25.0
character: "GLaDOS — flat, deliberate, menacing-cheerful"
provenance: "Portal GLaDOS lines"
warning: "LOW-SR source (16kHz) — upgrade the canonical clip if a cleaner GLaDOS source surfaces"
emmie:
canonical: canonical/emmie.wav
transcript: transcripts/emmie.txt
source_sr: 24000
duration_s: 19.3
provenance: "Zonos clone added 2026-07-17; origin unrecorded"
miranda:
canonical: canonical/miranda.wav
transcript: transcripts/miranda.txt
source_sr: 24000
duration_s: 16.3
provenance: "Zonos clone added 2026-07-17; origin unrecorded"
-1
View File
@@ -1 +0,0 @@
This is just not acceptable, Carl. I like my butterfly charm. It makes it so fairies like me, and it is pretty. It's part of my fit. I don't want to take it off. I don't see why I can't just wear two charms at the same time. Stupid angel of the caucus spaniel had like four or five tags.
-1
View File
@@ -1 +0,0 @@
I think I mentioned but I read your book because my my dear friend Nupa told me that I should and every now and again I would see you come up. I don't know. I take my job seriously I guess and so interviews to me felt a lot like chess and it required so much energy.
-1
View File
@@ -1 +0,0 @@
Welcome to test chamber 4. You're doing quite well. Once again, excellent work. As part of a required test protocol, we will not monitor the next test chamber. You will be entirely on your own. Good luck! As part of a required test protocol, our previous statement suggesting that we would not monitor this chamber was an outright fabrication. Good job! As part of a required test protocol, we will not monitor the next test protocol.
-1
View File
@@ -1 +0,0 @@
It's great. I mean, it's definitely comforting to go back to Australia when I come from there. So, you know, I get to see my parents, I get to see my friends and hang out. And I know the city really well because this was my fourth movie that I did in...