cosyvoice: warn that \instruct\ is Chinese-only; XML tags for EN

Verified empirically (gitea #7): the neosun wrapper's \`instruct\`
field routes through CosyVoice's inference_instruct2 which expects
a Chinese template. English directives produce Chinese-phonemed
speech of the directive itself — the model is stuck in Chinese
context regardless of the directive language.

XML inline tags (<angry>, <sad>, etc.) bypass that path entirely
and work cleanly for English emotion control.

README updated:
  - Added Gotcha section entry explaining the instruct trap with
    the specific failure mode and the byte-identical-response
    evidence
  - Smoke-test recipes (steps 4 + 5) reworked to use <angry>...</angry>
    in the input instead of an English instruct value
  - "Emotion / style control" table now marks instruct as Chinese-
    only with ⚠️ and XML tags as ✅
This commit is contained in:
vh
2026-04-24 16:45:19 -07:00
parent f7a8b668d3
commit 8f6c364f47
+34 -13
View File
@@ -113,6 +113,20 @@ ssh irv-ml1 '
- **Model choice.** `cosyvoice-v3` is default and best; `cosyvoice-v2`
is available as a fallback if you have voices tuned against the
v2 model's idiosyncrasies.
- **The `instruct` field is Chinese-context-only — DON'T use it
for English emotion control.** Verified empirically 2026-04-24:
any value in `instruct` (English, Chinese, or empty hints inside
it) routes through CosyVoice's `inference_instruct2` which expects
a directive in the upstream Chinese template
(`You are a helpful assistant. <directive>请用…<|endofprompt|>`).
English values like `instruct: "speak with cold contempt"` cause
the model to *vocalize the instruction text itself* — you'll hear
Chinese-flavored phonemes reading the English directive instead
of your `input`. Workaround: **use the XML inline tags** (`<angry>`,
`<sad>`, `<surprised>`, `<fearful>`, `<fast>`, `<slow>`,
`<whisper>`, `<shout>`, `<peppa>`, `<robot>`) for English emotion
control. They're parsed as control tokens, not content. See
upstream issue [reference TBD].
## Smoke test (full clone → synthesize)
@@ -131,29 +145,31 @@ curl -X POST http://10.100.79.3:8190/v1/voices/create \
-F "text=<paste from glados_25s.txt>"
# → {"voice_id": "<id>", "name": "glados", ...}
# 4. Synthesize via OpenAI-compat endpoint (single-shot)
# 4. Synthesize via OpenAI-compat endpoint (single-shot).
# For English emotion control, use the XML inline tag — NOT the
# `instruct` field (that's Chinese-context only; see Gotchas).
curl -X POST http://10.100.79.3:8190/v1/audio/speech \
-H 'Content-Type: application/json' \
-d '{
"model": "cosyvoice-v3",
"voice": "<voice_id>",
"input": "Hello. I did not ask for visitors today.",
"instruct": "speak with a flat, menacing calm",
"input": "<angry>I did not ask for visitors today.</angry>",
"response_format": "wav",
"speed": 1.0
}' \
-o out.wav
# 5. Or synthesize via native endpoint with streaming — curl writes
# audio chunks to the file as they arrive (~150 ms TTFB)
# audio chunks to the file as they arrive (~150 ms TTFB).
# `instruct_text` here has the same Chinese-only caveat as the
# OpenAI path's `instruct` — leave it empty for English work.
curl -N -X POST http://10.100.79.3:8190/api/tts \
-F "text=Hello. I did not ask for visitors today." \
-F "text=<angry>I did not ask for visitors today.</angry>" \
-F "voice=<voice_id>" \
-F "mode=zero_shot" \
-F "instruct_text=speak with a flat, menacing calm" \
-F "speed=1.0" \
-F "stream=true" \
-o out_stream.wav
-o out_stream.pcm
```
Field naming differs between the two synthesis endpoints:
@@ -170,16 +186,21 @@ Field naming differs between the two synthesis endpoints:
## Emotion / style control
Two syntaxes, combine freely:
**For English content, use XML inline tags. Do not use the `instruct`
field** — see the Gotcha above; `instruct` routes through CosyVoice's
Chinese-context inference path and English values get spoken aloud
instead of treated as direction.
| Syntax | Where | Example |
|---|---|---|
| XML tags in the text | inline in `input` / `text` | `<angry>You're late.</angry>` |
| Natural-language directive | `instruct` / `instruct_text` | `"speak with a flat, menacing calm"` |
| Syntax | Where | Example | English-safe? |
|---|---|---|---|
| XML tags in the text | inline in `input` / `text` | `<angry>You're late.</angry>` | ✅ |
| Natural-language directive | `instruct` / `instruct_text` | `"用愤怒的语气说"` (Chinese) | ⚠️ Chinese only |
Tag vocabulary includes `<angry>`, `<sad>`, `<surprised>`,
`<fearful>`, `<fast>`, `<slow>`, `<whisper>`, `<shout>`, plus
character/style tags like `<peppa>`, `<robot>`.
character/style tags like `<peppa>`, `<robot>`. Tags can be nested
or applied at sentence granularity (the model accepts paragraph-
length input with multiple tagged segments).
## Version bump