cosyvoice: rewrite README smoke test + document gotchas

The original smoke-test curl used voice="default" which doesn't exist — the neosun wrapper ships zero preset voices, and the built-in SFT speakers (中文男/女 etc.) are not surfaced. Calling /v1/audio/speech with any unregistered voice returns a 400 whose JSON body curl happily writes into the .wav (124-byte phantom). Replaced the smoke test with the full clone → synthesize flow and added a gotchas section covering: - No default voice; /v1/voices/create is mandatory - Reference audio ≤30s (frontend asserts; longer clips 500 at synthesis time, not at upload) - Providing an explicit transcript beats the auto-ASR fallback - voice_id is the handle, not name - Both cosyvoice-v3 and cosyvoice-v2 ship in the image Also documented streaming: available via /api/tts with stream=true (~150ms TTFB), NOT on /v1/audio/speech. Clarified field-name differences between the OpenAI-compat and native endpoints in a table. No WebSocket / SSE in this wrapper despite upstream support.
2026-04-24 00:28:15 -07:00
parent 01c5380059
commit 04884742e2
1 changed files with 113 additions and 20 deletions
@@ -17,16 +17,9 @@ internally but host port moved to avoid ComfyUI's 8188)
 Emotive content was the deal-breaker for Kokoro. CosyVoice 3 extended
 the v2 instruction-following dataset from 1,500 → 5,000 hours
 specifically covering emotions, speed, tones, dialects, accents, and
-role-playing. Streaming TTFB stays at ~150 ms.
-
-Two ways to request emotion / style:
-
- **XML tags** — `<angry>That's my line!</angry>`, `<sad>…</sad>`,
-  `<fast>…</fast>`, `<slow>…</slow>`, `<peppa>…</peppa>`,
-  `<robot>…</robot>`
- **Instruction prompts** — `You are a helpful assistant. 请用尽可能快地语速说一句话。<|endofprompt|>`
-  gives finer control via natural-language directives in the
-  instruction channel
+role-playing. Streaming is available on `/api/tts` at ~150 ms TTFB
+(not on the OpenAI-compat `/v1/audio/speech` path — see API table
+below).

 Language center of gravity is Mandarin + Cantonese (18+ Chinese
 dialects) and then 8 other languages (English, Japanese, Korean,
@@ -36,12 +29,28 @@ your actual content.

 ## API endpoints

-| Method + path | Purpose |
-|---|---|
-| `POST /v1/audio/speech` | OpenAI drop-in for TTS |
-| `POST /v1/voices/create` | Clone a speaker from reference audio (auto-transcription via built-in ASR) |
-| `GET /v1/voices` | List cloned voices by `voice_id` |
-| `GET /health` | Health probe (used by docker healthcheck) |
+| Method + path | Purpose | Streaming? |
+|---|---|---|
+| `POST /v1/voices/create` | Clone a speaker from reference audio | n/a |
+| `GET /v1/voices` | List cloned voices | n/a |
+| `GET /v1/voices/{voice_id}` | Inspect one voice | n/a |
+| `DELETE /v1/voices/{voice_id}` | Remove a cloned voice | n/a |
+| `GET /v1/models` | Lists `cosyvoice-v3` + `cosyvoice-v2` (both bundled) | n/a |
+| `POST /v1/audio/speech` | OpenAI drop-in TTS | **no** — single response body |
+| `POST /api/tts` | Native TTS with finer control | **yes** when `stream=true` |
+| `POST /api/tts/async` | Job-based TTS; poll `/api/task/{task_id}` | no |
+| `GET /health` | Health probe | n/a |
+| Web UI at `/` | Gradio-style voice-cloning + inference UI | — |
+
+Notably **not** exposed:
+- No `preset_voices` / `spk_id` — `/api/speakers` always returns `[]`
+  in this image. CosyVoice's built-in SFT speakers (中文男/女 etc.) are
+  not surfaced. Every synthesis needs a voice that came from
+  `/v1/voices/create`.
+- No WebSocket / SSE endpoints — the upstream CosyVoice 3 model
+  supports bi-directional streaming at ~150 ms TTFB, but this wrapper
+  doesn't surface WS. Use `/api/tts` with `stream=true` for HTTP
+  chunked streaming (still ~150 ms TTFB).

 ## Path layout

@@ -78,16 +87,100 @@ ssh irv-ml1 '
 '
 ```

-## Smoke test
+## Gotchas (read before first use)
+
+- **No default voice ships with the image.** You must clone one via
+  `/v1/voices/create` before `/v1/audio/speech` can return audio.
+  Calling `/v1/audio/speech` with `voice="default"` (or any
+  unregistered id) returns HTTP 400; curl writing the error JSON into
+  a `.wav` file is what the 124-byte phantom output was in an early
+  smoke test.
+- **Reference audio must be ≤ 30 seconds.** The frontend asserts
+  `speech.shape[1] / 16000 <= 30` — longer clips are accepted at
+  upload time but synthesis raises `AssertionError: do not support
+  extract speech token for audio longer than 30s`, producing a 500.
+  Trim with ffmpeg before `/v1/voices/create`:
+  `ffmpeg -i src.wav -ss 0 -t 25 -ac 1 -ar 16000 ref.wav`
+- **Provide an accurate transcript with the clone.** The `text` field
+  of `/v1/voices/create` defaults to empty, in which case the wrapper
+  auto-transcribes via Fun-ASR-Nano. That often misses names /
+  technical terms. Supplying your own transcript (e.g., from the
+  Parakeet stack at `:8765`) produces better clones.
+- **Voice id vs. name.** The response's `voice_id` is what you pass
+  to subsequent `/v1/audio/speech` calls. The `name` is purely for
+  humans — reusing a name with a new upload creates a **new** voice
+  id, it doesn't overwrite.
+- **Model choice.** `cosyvoice-v3` is default and best; `cosyvoice-v2`
+  is available as a fallback if you have voices tuned against the
+  v2 model's idiosyncrasies.
+
+## Smoke test (full clone → synthesize)

 ```bash
-# From the workstation over WG — OpenAI-compatible request
+# 1. Trim reference audio to ≤30s (16 kHz mono is cleanest input)
+ffmpeg -i glados.wav -ss 0 -t 25 -ac 1 -ar 16000 glados_25s.wav
+
+# 2. Transcribe the trimmed clip via Parakeet for an accurate text field
+curl -s -F "file=@./glados_25s.wav" http://10.100.79.3:8765/transcribe \
+  | tee glados_25s.txt
+
+# 3. Clone the voice
+curl -X POST http://10.100.79.3:8190/v1/voices/create \
+  -F "audio=@./glados_25s.wav" \
+  -F "name=glados" \
+  -F "text=<paste from glados_25s.txt>"
+# → {"voice_id": "<id>", "name": "glados", ...}
+
+# 4. Synthesize via OpenAI-compat endpoint (single-shot)
 curl -X POST http://10.100.79.3:8190/v1/audio/speech \
  -H 'Content-Type: application/json' \
-  -d '{"model":"cosyvoice","voice":"default","input":"<angry>Hello there.</angry>","response_format":"wav"}' \
-  -o /tmp/out.wav
+  -d '{
+    "model": "cosyvoice-v3",
+    "voice": "<voice_id>",
+    "input": "Hello. I did not ask for visitors today.",
+    "instruct": "speak with a flat, menacing calm",
+    "response_format": "wav",
+    "speed": 1.0
+  }' \
+  -o out.wav
+
+# 5. Or synthesize via native endpoint with streaming — curl writes
+#    audio chunks to the file as they arrive (~150 ms TTFB)
+curl -N -X POST http://10.100.79.3:8190/api/tts \
+  -F "text=Hello. I did not ask for visitors today." \
+  -F "voice=<voice_id>" \
+  -F "mode=zero_shot" \
+  -F "instruct_text=speak with a flat, menacing calm" \
+  -F "speed=1.0" \
+  -F "stream=true" \
+  -o out_stream.wav
 ```

+Field naming differs between the two synthesis endpoints:
+
+| `/v1/audio/speech` (JSON) | `/api/tts` (multipart) |
+|---|---|
+| `input` | `text` |
+| `instruct` | `instruct_text` |
+| `voice` | `voice` (same) |
+| `response_format` | — (always wav/pcm chunks) |
+| — | `stream` (bool) |
+| — | `mode` (`zero_shot` / `instruct` / `sft`) |
+| — | `prompt_wav` + `prompt_text` (inline ref audio; skip /v1/voices/create) |
+
+## Emotion / style control
+
+Two syntaxes, combine freely:
+
+| Syntax | Where | Example |
+|---|---|---|
+| XML tags in the text | inline in `input` / `text` | `<angry>You're late.</angry>` |
+| Natural-language directive | `instruct` / `instruct_text` | `"speak with a flat, menacing calm"` |
+
+Tag vocabulary includes `<angry>`, `<sad>`, `<surprised>`,
+`<fearful>`, `<fast>`, `<slow>`, `<whisper>`, `<shout>`, plus
+character/style tags like `<peppa>`, `<robot>`.
+
 ## Version bump

 ```bash