From 04884742e23dd376ea46291d8f22cdd0897ad449 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 24 Apr 2026 00:28:15 -0700 Subject: [PATCH] cosyvoice: rewrite README smoke test + document gotchas MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The original smoke-test curl used voice="default" which doesn't exist — the neosun wrapper ships zero preset voices, and the built-in SFT speakers (中文男/女 etc.) are not surfaced. Calling /v1/audio/speech with any unregistered voice returns a 400 whose JSON body curl happily writes into the .wav (124-byte phantom). Replaced the smoke test with the full clone → synthesize flow and added a gotchas section covering: - No default voice; /v1/voices/create is mandatory - Reference audio ≤30s (frontend asserts; longer clips 500 at synthesis time, not at upload) - Providing an explicit transcript beats the auto-ASR fallback - voice_id is the handle, not name - Both cosyvoice-v3 and cosyvoice-v2 ship in the image Also documented streaming: available via /api/tts with stream=true (~150ms TTFB), NOT on /v1/audio/speech. Clarified field-name differences between the OpenAI-compat and native endpoints in a table. No WebSocket / SSE in this wrapper despite upstream support. --- stacks/cosyvoice/README.md | 133 +++++++++++++++++++++++++++++++------ 1 file changed, 113 insertions(+), 20 deletions(-) diff --git a/stacks/cosyvoice/README.md b/stacks/cosyvoice/README.md index 18b8426..6661dff 100644 --- a/stacks/cosyvoice/README.md +++ b/stacks/cosyvoice/README.md @@ -17,16 +17,9 @@ internally but host port moved to avoid ComfyUI's 8188) Emotive content was the deal-breaker for Kokoro. CosyVoice 3 extended the v2 instruction-following dataset from 1,500 → 5,000 hours specifically covering emotions, speed, tones, dialects, accents, and -role-playing. Streaming TTFB stays at ~150 ms. - -Two ways to request emotion / style: - -- **XML tags** — `That's my line!`, `…`, - `…`, `…`, `…`, - `…` -- **Instruction prompts** — `You are a helpful assistant. 请用尽可能快地语速说一句话。<|endofprompt|>` - gives finer control via natural-language directives in the - instruction channel +role-playing. Streaming is available on `/api/tts` at ~150 ms TTFB +(not on the OpenAI-compat `/v1/audio/speech` path — see API table +below). Language center of gravity is Mandarin + Cantonese (18+ Chinese dialects) and then 8 other languages (English, Japanese, Korean, @@ -36,12 +29,28 @@ your actual content. ## API endpoints -| Method + path | Purpose | -|---|---| -| `POST /v1/audio/speech` | OpenAI drop-in for TTS | -| `POST /v1/voices/create` | Clone a speaker from reference audio (auto-transcription via built-in ASR) | -| `GET /v1/voices` | List cloned voices by `voice_id` | -| `GET /health` | Health probe (used by docker healthcheck) | +| Method + path | Purpose | Streaming? | +|---|---|---| +| `POST /v1/voices/create` | Clone a speaker from reference audio | n/a | +| `GET /v1/voices` | List cloned voices | n/a | +| `GET /v1/voices/{voice_id}` | Inspect one voice | n/a | +| `DELETE /v1/voices/{voice_id}` | Remove a cloned voice | n/a | +| `GET /v1/models` | Lists `cosyvoice-v3` + `cosyvoice-v2` (both bundled) | n/a | +| `POST /v1/audio/speech` | OpenAI drop-in TTS | **no** — single response body | +| `POST /api/tts` | Native TTS with finer control | **yes** when `stream=true` | +| `POST /api/tts/async` | Job-based TTS; poll `/api/task/{task_id}` | no | +| `GET /health` | Health probe | n/a | +| Web UI at `/` | Gradio-style voice-cloning + inference UI | — | + +Notably **not** exposed: +- No `preset_voices` / `spk_id` — `/api/speakers` always returns `[]` + in this image. CosyVoice's built-in SFT speakers (中文男/女 etc.) are + not surfaced. Every synthesis needs a voice that came from + `/v1/voices/create`. +- No WebSocket / SSE endpoints — the upstream CosyVoice 3 model + supports bi-directional streaming at ~150 ms TTFB, but this wrapper + doesn't surface WS. Use `/api/tts` with `stream=true` for HTTP + chunked streaming (still ~150 ms TTFB). ## Path layout @@ -78,16 +87,100 @@ ssh irv-ml1 ' ' ``` -## Smoke test +## Gotchas (read before first use) + +- **No default voice ships with the image.** You must clone one via + `/v1/voices/create` before `/v1/audio/speech` can return audio. + Calling `/v1/audio/speech` with `voice="default"` (or any + unregistered id) returns HTTP 400; curl writing the error JSON into + a `.wav` file is what the 124-byte phantom output was in an early + smoke test. +- **Reference audio must be ≤ 30 seconds.** The frontend asserts + `speech.shape[1] / 16000 <= 30` — longer clips are accepted at + upload time but synthesis raises `AssertionError: do not support + extract speech token for audio longer than 30s`, producing a 500. + Trim with ffmpeg before `/v1/voices/create`: + `ffmpeg -i src.wav -ss 0 -t 25 -ac 1 -ar 16000 ref.wav` +- **Provide an accurate transcript with the clone.** The `text` field + of `/v1/voices/create` defaults to empty, in which case the wrapper + auto-transcribes via Fun-ASR-Nano. That often misses names / + technical terms. Supplying your own transcript (e.g., from the + Parakeet stack at `:8765`) produces better clones. +- **Voice id vs. name.** The response's `voice_id` is what you pass + to subsequent `/v1/audio/speech` calls. The `name` is purely for + humans — reusing a name with a new upload creates a **new** voice + id, it doesn't overwrite. +- **Model choice.** `cosyvoice-v3` is default and best; `cosyvoice-v2` + is available as a fallback if you have voices tuned against the + v2 model's idiosyncrasies. + +## Smoke test (full clone → synthesize) ```bash -# From the workstation over WG — OpenAI-compatible request +# 1. Trim reference audio to ≤30s (16 kHz mono is cleanest input) +ffmpeg -i glados.wav -ss 0 -t 25 -ac 1 -ar 16000 glados_25s.wav + +# 2. Transcribe the trimmed clip via Parakeet for an accurate text field +curl -s -F "file=@./glados_25s.wav" http://10.100.79.3:8765/transcribe \ + | tee glados_25s.txt + +# 3. Clone the voice +curl -X POST http://10.100.79.3:8190/v1/voices/create \ + -F "audio=@./glados_25s.wav" \ + -F "name=glados" \ + -F "text=" +# → {"voice_id": "", "name": "glados", ...} + +# 4. Synthesize via OpenAI-compat endpoint (single-shot) curl -X POST http://10.100.79.3:8190/v1/audio/speech \ -H 'Content-Type: application/json' \ - -d '{"model":"cosyvoice","voice":"default","input":"Hello there.","response_format":"wav"}' \ - -o /tmp/out.wav + -d '{ + "model": "cosyvoice-v3", + "voice": "", + "input": "Hello. I did not ask for visitors today.", + "instruct": "speak with a flat, menacing calm", + "response_format": "wav", + "speed": 1.0 + }' \ + -o out.wav + +# 5. Or synthesize via native endpoint with streaming — curl writes +# audio chunks to the file as they arrive (~150 ms TTFB) +curl -N -X POST http://10.100.79.3:8190/api/tts \ + -F "text=Hello. I did not ask for visitors today." \ + -F "voice=" \ + -F "mode=zero_shot" \ + -F "instruct_text=speak with a flat, menacing calm" \ + -F "speed=1.0" \ + -F "stream=true" \ + -o out_stream.wav ``` +Field naming differs between the two synthesis endpoints: + +| `/v1/audio/speech` (JSON) | `/api/tts` (multipart) | +|---|---| +| `input` | `text` | +| `instruct` | `instruct_text` | +| `voice` | `voice` (same) | +| `response_format` | — (always wav/pcm chunks) | +| — | `stream` (bool) | +| — | `mode` (`zero_shot` / `instruct` / `sft`) | +| — | `prompt_wav` + `prompt_text` (inline ref audio; skip /v1/voices/create) | + +## Emotion / style control + +Two syntaxes, combine freely: + +| Syntax | Where | Example | +|---|---|---| +| XML tags in the text | inline in `input` / `text` | `You're late.` | +| Natural-language directive | `instruct` / `instruct_text` | `"speak with a flat, menacing calm"` | + +Tag vocabulary includes ``, ``, ``, +``, ``, ``, ``, ``, plus +character/style tags like ``, ``. + ## Version bump ```bash