From c8acf6044934c1690b289dc35261cd47817ff78b Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 10 Aug 2026 01:07:37 -0700 Subject: [PATCH] feat(dots-tts): ship OpenAI-compatible dots.tts TTS stack on irv-ml1:8198 Thin FastAPI wrapper over DotsTtsRuntime (soar, optimize=True, RTF ~0.22), serialized single-consumer; OpenAI /v1/audio/speech (stream + non-stream), voices from the voices/ corpus derived set. Live + healthy alongside chatterbox-fast on the 3090; nothing repointed. Dockerfile needs build-essential (torch.compile/inductor JITs via gcc at runtime) + persisted inductor cache. Remaining Phase-2: ratatoskr client cutover. --- persistent-memory.md | 4 +- stacks/dots-tts/.env.example | 27 +++++++ stacks/dots-tts/Dockerfile | 34 ++++++++ stacks/dots-tts/README.md | 64 +++++++++++++++ stacks/dots-tts/app.py | 147 +++++++++++++++++++++++++++++++++++ stacks/dots-tts/compose.yaml | 50 ++++++++++++ 6 files changed, 325 insertions(+), 1 deletion(-) create mode 100644 stacks/dots-tts/.env.example create mode 100644 stacks/dots-tts/Dockerfile create mode 100644 stacks/dots-tts/README.md create mode 100644 stacks/dots-tts/app.py create mode 100644 stacks/dots-tts/compose.yaml diff --git a/persistent-memory.md b/persistent-memory.md index 41704b7..898c419 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -119,6 +119,8 @@ _As of 2026-08-08 — long session; all major arcs LANDED (full detail per arc i - **EVIDENCE HOLD (partial):** WT #394 index-row half LIFTED + swept (kb 8,230→2,876); the **FILE half STILL STANDS** — do NOT delete the on-disk generation dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`). Also un-actioned worldtree-side: the `reconcile --repair` rc=1 git-staging bug (phantom `docs/specs/saga-taxonomy-redesign-plan.md`) — flagged, their code. +- **OPEN LOOPS (2026-08-10):** (1) **dots-tts ratatoskr cutover** — draft the ask to repoint ratatoskr's RP TTS to irv-ml1:8198 `/v1/audio/speech` (Phase-2 tail; dots container LIVE, see Recent decisions). (2) **LFM2.5-2.6b bake-off alias for brokkr-smithy-dev** — arch confirmed (Lfm2ForCausalLM, vLLM 0.24.0 on ana-ml2 supports it, no bump), replied w/ alias `lfm2.5-2.6b` + plan (non-prod, vendor sampling temp0.1/topk50/reppen1.1, reasoning-channel content-non-empty smoke, LFM Open License = eval-only pending operator ruling); **stand-up QUEUED** (thread `01KZNA4MKPDR…`, brokkr said no-rush). + - **OPEN FOLLOW-UPS:** chatterbox-fast **deployed flat-build-context vs package-repo divergence** (reconcile so a repo rebuild matches deploy — repo commit `6bc7bf0` has cap+norm_loudness in package layout; deployed is flat cap-only). Standing/parked: CI-flip runner-auth research, gitea `REQUIRE_SIGNIN_VIEW=false`, #363 research-wing ingest (no deadline), zonos-gateway CI-wire. **althing monitor** ARMED (handle `infra-ops`, task `b1uat4njl`; herald up). ⚠️ Re-arm ONLY after a real FIRE (` completed rc0`), NEVER after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, **NEVER chained with `&`/`&&`** (orphans it to init — bit me this session; reclaim with `althing-cli stop-monitor` then re-arm cleanly). @@ -128,7 +130,7 @@ _As of 2026-08-08 — long session; all major arcs LANDED (full detail per arc i ## Recent decisions -- `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/.wav` + `transcripts/.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **PENDING: operator A/B ear-verdict → Phase 2** (containerize SGLang Omni serving dots.tts-soar on the 3090 alongside chatterbox; ratatoskr client cutover to `/v1/audio/speech`); wrapper-vs-Omni serving-layer choice deferred to Phase 2. **OPEN operator call:** corpus home = eshpfi `voices/` (my rec) vs spin-out `vh/voice-corpus`. [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]] +- `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/.wav` + `transcripts/.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]] - `[2026-08-08]` **worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev.** worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared `~/snapshots/worldtree-400-fiction-decomp` (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). **LEFT INTACT:** `rex393-fiction-index`/`rex393-fiction-snapshot` (separate operator KEEP word, unchanged) + `r42-gate-*`. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (`01KZJ9GMCC…`). diff --git a/stacks/dots-tts/.env.example b/stacks/dots-tts/.env.example new file mode 100644 index 0000000..8216c5a --- /dev/null +++ b/stacks/dots-tts/.env.example @@ -0,0 +1,27 @@ +# dots-tts stack tunables. Copy to `.env` on irv-ml1 before deploying. + +# ── image ──────────────────────────────────────────────────────────── +DOTS_TAG=v1 + +# ── network ────────────────────────────────────────────────────────── +DOTS_BIND=0.0.0.0 +DOTS_PORT=8198 + +# ── GPU ────────────────────────────────────────────────────────────── +# 0 = 3090 in Docker (PCI order), co-resident with chatterbox-fast. soar needs +# ~6GB; the 3090 has headroom with Zonos parked down. +DOTS_GPU_DEVICES=0 + +# ── model / inference ──────────────────────────────────────────────── +DOTS_MODEL=dots-studio/dots.tts-soar +DOTS_DEFAULT_VOICE=donut +DOTS_NUM_STEPS=10 # 10 = full quality @ RTF ~0.22; lower = faster/rougher +DOTS_GUIDANCE_SCALE=1.2 + +# ── host mounts ────────────────────────────────────────────────────── +# HF cache holding the downloaded soar snapshot (~5GB). Reuse the burn-in cache. +DOTS_HFCACHE_DIR=/home/lkraven/dots-tts/hf_cache +# dots-derived voice references (derive.py dots -> derived/dots/.{wav,txt}). +# Burn-in points at the corpus output directly; for a durable deploy, copy the +# derived set to /opt/docker/conf/dots-tts/voices and point here. +DOTS_VOICES_HOST_DIR=/home/lkraven/voice-corpus/derived/dots diff --git a/stacks/dots-tts/Dockerfile b/stacks/dots-tts/Dockerfile new file mode 100644 index 0000000..adfbb43 --- /dev/null +++ b/stacks/dots-tts/Dockerfile @@ -0,0 +1,34 @@ +# dots.tts OpenAI-compatible TTS server (thin FastAPI over DotsTtsRuntime). +# GPU access is via `runtime: nvidia` at run time (torch ships its own CUDA +# runtime; no CUDA toolkit / nvcc needed to build — the model uses no custom +# compiled kernels, confirmed on the irv-ml1 venv). +FROM python:3.11-slim + +RUN apt-get update && apt-get install -y --no-install-recommends \ + libsndfile1 ffmpeg git curl ca-certificates \ + && rm -rf /var/lib/apt/lists/* + +RUN pip install --no-cache-dir uv + +WORKDIR /app + +# Pin torch + deps to dots.tts upstream recommended constraints (the same set the +# irv-ml1 venv installed against). ADD caches on the URL contents. +ADD https://raw.githubusercontent.com/rednote-hilab/dots.tts/main/constraints/recommended.txt /tmp/rec.txt +RUN uv pip install --system -c /tmp/rec.txt \ + dots.tts soundfile fastapi "uvicorn[standard]" + +# C compiler for the RUNTIME (not build): optimize=True drives torch.compile / +# inductor / triton, which JIT-compile kernels via gcc on model load. Without it +# the runtime dies with "Failed to find C compiler". Placed after the pip layer +# so it doesn't invalidate the expensive torch install cache. +RUN apt-get update && apt-get install -y --no-install-recommends build-essential \ + && rm -rf /var/lib/apt/lists/* + +COPY app.py /app/app.py + +# Persist the inductor compile cache on the mounted (rw) HF cache so kernel +# JIT doesn't re-run on every container restart (~70s warmup otherwise). +ENV HF_HOME=/hf_cache DOTS_PORT=8198 CC=gcc CXX=g++ TORCHINDUCTOR_CACHE_DIR=/hf_cache/inductor +EXPOSE 8198 +CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8198"] diff --git a/stacks/dots-tts/README.md b/stacks/dots-tts/README.md new file mode 100644 index 0000000..2289497 --- /dev/null +++ b/stacks/dots-tts/README.md @@ -0,0 +1,64 @@ +# dots-tts + +OpenAI-compatible zero-shot voice-clone TTS over **dots.tts** (rednote-hilab) — +2B continuous-AR, native **48kHz**, `optimize=True` CUDA graphs → **RTF ~0.22** on +the irv-ml1 3090. Thin FastAPI wrapper around `DotsTtsRuntime` (chosen over SGLang +Omni: Omni's batching is MeanFlow-only and unneeded for a single consumer; the raw +runtime already streams at the same RTF and is ~100 lines we control). + +- **Host:** irv-ml1, port **8198** (chatterbox-fast is :8197 — they co-reside on the 3090) +- **Model:** `dots-studio/dots.tts-soar`, bf16, num_steps=10 +- **Voices:** every `.wav` (+ `.txt` transcript) in the mounted voices dir, + sourced from the [`voices/`](../../voices/) canonical corpus via `derive.py dots`. + +## API + +``` +GET /health -> {status, model, sample_rate, voices[]} +GET /v1/voices -> {voices[]} +POST /v1/audio/speech -> audio + body: {input, voice, response_format?("wav"|"pcm"), stream?} +``` + +`stream:true` returns a WAV stream (placeholder-header + PCM frames, 48kHz mono +s16le) — the same shape the Zonos/chatterbox consumers already handle. Non-stream +returns a complete WAV (or raw PCM with `response_format:"pcm"`). + +```bash +curl -X POST http://10.100.79.3:8198/v1/audio/speech \ + -H 'Content-Type: application/json' \ + -d '{"input":"Well, look who finally showed up.","voice":"glados"}' \ + --output out.wav +``` + +## Deploy + +Reference sets come from the canonical corpus, not this stack — derive then point +the mount at them: + +```bash +# 1. produce dots refs from the corpus (on a box with the whisper venv): +python voices/derive.py dots # -> voices/derived/dots/*.{wav,txt} + +# 2. build + run on irv-ml1 (cp .env.example .env first; adjust mounts): +scripts/deploy-stack.sh irv-ml1 dots-tts # or, on the host: +docker compose build && docker compose up -d +``` + +The model (~5GB) is **not** baked — it's read from the mounted `HF_HOME` +(`DOTS_HFCACHE_DIR`). First boot downloads it there if absent. + +## Voice cloning gotcha + +dots.tts clones from `(reference wav + its transcript)` and **leaks reference +audio into the output** if the transcript is inaccurate or ends mid-clause. The +`voices/` corpus + `derive.py` handle this (sentence-bounded trim + accurate +transcript); don't hand this server a raw reference wav without a matching `.txt`. + +## Notes + +- **GPU:** `NVIDIA_VISIBLE_DEVICES=0` = the 3090 in Docker (PCI order). `optimize=True` + is **incompatible with `PYTORCH_CUDA_ALLOC_CONF=expandable_segments`** (CUDA-graph + capture error) — don't set it. +- **Variants:** `dots.tts-mf` (MeanFlow, faster) is a drop-in via `DOTS_MODEL`; soar + is the quality pick and single-consumer doesn't need mf's batching. diff --git a/stacks/dots-tts/app.py b/stacks/dots-tts/app.py new file mode 100644 index 0000000..392d0d2 --- /dev/null +++ b/stacks/dots-tts/app.py @@ -0,0 +1,147 @@ +"""OpenAI-compatible /v1/audio/speech server over dots.tts (rednote-hilab). + +Thin wrapper around DotsTtsRuntime — chosen over SGLang Omni because Omni's edge +(continuous batching) is MeanFlow-only and unneeded for a single-consumer surface, +while the raw runtime with optimize=True already streams at RTF ~0.22 on our 3090. + +Voice registry: every .wav (+ optional .txt transcript) under +DOTS_VOICES_DIR becomes a callable voice. dots.tts REQUIRES an accurate, +sentence-bounded transcript to clone cleanly (see the voices/ corpus) — the .txt +is that transcript; without it the model leaks reference audio into the output. +""" +import io +import os +import glob +import struct +import threading +import wave + +import numpy as np +from fastapi import FastAPI, HTTPException +from fastapi.responses import Response, StreamingResponse +from pydantic import BaseModel + +from dots_tts.runtime import DotsTtsRuntime + +MODEL = os.environ.get("DOTS_MODEL", "dots-studio/dots.tts-soar") +VOICES_DIR = os.environ.get("DOTS_VOICES_DIR", "/voices") +DEFAULT_VOICE = os.environ.get("DOTS_DEFAULT_VOICE", "donut") +NUM_STEPS = int(os.environ.get("DOTS_NUM_STEPS", "10")) +GUIDANCE = float(os.environ.get("DOTS_GUIDANCE_SCALE", "1.2")) +SAMPLE_RATE = 48000 # dots.tts fixed native output + +app = FastAPI(title="dots.tts") +_rt = None +_voices: dict = {} +# One DotsTtsRuntime, and it is NOT safe to call concurrently (CUDA-graph capture +# + shared state). uvicorn runs sync endpoints in a threadpool, so we must +# serialize generation ourselves: requests queue and run one at a time. This is +# the deliberate trade for the thin-wrapper design — no vLLM-style continuous +# batching. If concurrency demand appears, swap the backend to SGLang Omni + the +# mf variant behind this same API (see README). +_gen_lock = threading.Lock() + + +def _load_voices() -> dict: + reg = {} + for wav in sorted(glob.glob(os.path.join(VOICES_DIR, "*.wav"))): + name = os.path.splitext(os.path.basename(wav))[0] + txt = os.path.splitext(wav)[0] + ".txt" + reg[name] = { + "wav": wav, + "text": open(txt).read().strip() if os.path.exists(txt) else "", + } + return reg + + +@app.on_event("startup") +def _startup(): + global _rt, _voices + _voices = _load_voices() + _rt = DotsTtsRuntime.from_pretrained(MODEL, precision="bfloat16", optimize=True) + + +@app.get("/health") +def health(): + return { + "status": "ok" if _rt is not None else "loading", + "model": MODEL, + "sample_rate": SAMPLE_RATE, + "voices": sorted(_voices), + } + + +@app.get("/v1/voices") +def list_voices(): + return {"voices": sorted(_voices)} + + +class SpeechRequest(BaseModel): + input: str + voice: str = DEFAULT_VOICE + model: str | None = None # accepted, ignored (single served model) + response_format: str = "wav" # wav | pcm + stream: bool = False + + +def _to_pcm16(audio: np.ndarray) -> bytes: + return np.round(np.clip(audio, -1.0, 1.0) * 32767.0).astype(" bytes: + buf = io.BytesIO() + w = wave.open(buf, "wb") + w.setnchannels(1) + w.setsampwidth(2) + w.setframerate(SAMPLE_RATE) + w.writeframes(pcm) + w.close() + return buf.getvalue() + + +def _streaming_wav_header() -> bytes: + """WAV header with placeholder (max) sizes — lets a client start playing the + stream before the total length is known (the pattern the Zonos/chatterbox + consumers already expect).""" + return ( + b"RIFF" + struct.pack(" derived/dots/.{wav,txt}). Read-only. + - ${DOTS_VOICES_HOST_DIR:-/home/lkraven/voice-corpus/derived/dots}:/voices:ro + healthcheck: + # /health returns {"status":"ok",...} only once the model has loaded. + test: ["CMD-SHELL", "python3 -c \"import urllib.request,sys; b=urllib.request.urlopen('http://127.0.0.1:8198/health',timeout=5).read(); sys.exit(0 if b'\\\"status\\\":\\\"ok\\\"' in b.replace(b' ',b'') else 1)\""] + interval: 30s + timeout: 10s + retries: 3 + # Model load + optimize=True CUDA-graph warmup measured ~70s; generous. + start_period: 180s + labels: + - homepage.group=AI - Speech (TTS) + - homepage.name=dots.tts + - homepage.icon=mdi-account-voice + - homepage.description=Continuous-AR 48kHz zero-shot voice clone (irv-ml1) + - homepage.href=http://10.100.79.3:${DOTS_PORT:-8198}