cosyvoice: warn that \instruct\ is Chinese-only; XML tags for EN
Verified empirically (gitea #7): the neosun wrapper's \`instruct\` field routes through CosyVoice's inference_instruct2 which expects a Chinese template. English directives produce Chinese-phonemed speech of the directive itself — the model is stuck in Chinese context regardless of the directive language. XML inline tags (<angry>, <sad>, etc.) bypass that path entirely and work cleanly for English emotion control. README updated: - Added Gotcha section entry explaining the instruct trap with the specific failure mode and the byte-identical-response evidence - Smoke-test recipes (steps 4 + 5) reworked to use <angry>...</angry> in the input instead of an English instruct value - "Emotion / style control" table now marks instruct as Chinese- only with ⚠️ and XML tags as ✅
This commit is contained in:
+34
-13
@@ -113,6 +113,20 @@ ssh irv-ml1 '
|
||||
- **Model choice.** `cosyvoice-v3` is default and best; `cosyvoice-v2`
|
||||
is available as a fallback if you have voices tuned against the
|
||||
v2 model's idiosyncrasies.
|
||||
- **The `instruct` field is Chinese-context-only — DON'T use it
|
||||
for English emotion control.** Verified empirically 2026-04-24:
|
||||
any value in `instruct` (English, Chinese, or empty hints inside
|
||||
it) routes through CosyVoice's `inference_instruct2` which expects
|
||||
a directive in the upstream Chinese template
|
||||
(`You are a helpful assistant. <directive>请用…<|endofprompt|>`).
|
||||
English values like `instruct: "speak with cold contempt"` cause
|
||||
the model to *vocalize the instruction text itself* — you'll hear
|
||||
Chinese-flavored phonemes reading the English directive instead
|
||||
of your `input`. Workaround: **use the XML inline tags** (`<angry>`,
|
||||
`<sad>`, `<surprised>`, `<fearful>`, `<fast>`, `<slow>`,
|
||||
`<whisper>`, `<shout>`, `<peppa>`, `<robot>`) for English emotion
|
||||
control. They're parsed as control tokens, not content. See
|
||||
upstream issue [reference TBD].
|
||||
|
||||
## Smoke test (full clone → synthesize)
|
||||
|
||||
@@ -131,29 +145,31 @@ curl -X POST http://10.100.79.3:8190/v1/voices/create \
|
||||
-F "text=<paste from glados_25s.txt>"
|
||||
# → {"voice_id": "<id>", "name": "glados", ...}
|
||||
|
||||
# 4. Synthesize via OpenAI-compat endpoint (single-shot)
|
||||
# 4. Synthesize via OpenAI-compat endpoint (single-shot).
|
||||
# For English emotion control, use the XML inline tag — NOT the
|
||||
# `instruct` field (that's Chinese-context only; see Gotchas).
|
||||
curl -X POST http://10.100.79.3:8190/v1/audio/speech \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"model": "cosyvoice-v3",
|
||||
"voice": "<voice_id>",
|
||||
"input": "Hello. I did not ask for visitors today.",
|
||||
"instruct": "speak with a flat, menacing calm",
|
||||
"input": "<angry>I did not ask for visitors today.</angry>",
|
||||
"response_format": "wav",
|
||||
"speed": 1.0
|
||||
}' \
|
||||
-o out.wav
|
||||
|
||||
# 5. Or synthesize via native endpoint with streaming — curl writes
|
||||
# audio chunks to the file as they arrive (~150 ms TTFB)
|
||||
# audio chunks to the file as they arrive (~150 ms TTFB).
|
||||
# `instruct_text` here has the same Chinese-only caveat as the
|
||||
# OpenAI path's `instruct` — leave it empty for English work.
|
||||
curl -N -X POST http://10.100.79.3:8190/api/tts \
|
||||
-F "text=Hello. I did not ask for visitors today." \
|
||||
-F "text=<angry>I did not ask for visitors today.</angry>" \
|
||||
-F "voice=<voice_id>" \
|
||||
-F "mode=zero_shot" \
|
||||
-F "instruct_text=speak with a flat, menacing calm" \
|
||||
-F "speed=1.0" \
|
||||
-F "stream=true" \
|
||||
-o out_stream.wav
|
||||
-o out_stream.pcm
|
||||
```
|
||||
|
||||
Field naming differs between the two synthesis endpoints:
|
||||
@@ -170,16 +186,21 @@ Field naming differs between the two synthesis endpoints:
|
||||
|
||||
## Emotion / style control
|
||||
|
||||
Two syntaxes, combine freely:
|
||||
**For English content, use XML inline tags. Do not use the `instruct`
|
||||
field** — see the Gotcha above; `instruct` routes through CosyVoice's
|
||||
Chinese-context inference path and English values get spoken aloud
|
||||
instead of treated as direction.
|
||||
|
||||
| Syntax | Where | Example |
|
||||
|---|---|---|
|
||||
| XML tags in the text | inline in `input` / `text` | `<angry>You're late.</angry>` |
|
||||
| Natural-language directive | `instruct` / `instruct_text` | `"speak with a flat, menacing calm"` |
|
||||
| Syntax | Where | Example | English-safe? |
|
||||
|---|---|---|---|
|
||||
| XML tags in the text | inline in `input` / `text` | `<angry>You're late.</angry>` | ✅ |
|
||||
| Natural-language directive | `instruct` / `instruct_text` | `"用愤怒的语气说"` (Chinese) | ⚠️ Chinese only |
|
||||
|
||||
Tag vocabulary includes `<angry>`, `<sad>`, `<surprised>`,
|
||||
`<fearful>`, `<fast>`, `<slow>`, `<whisper>`, `<shout>`, plus
|
||||
character/style tags like `<peppa>`, `<robot>`.
|
||||
character/style tags like `<peppa>`, `<robot>`. Tags can be nested
|
||||
or applied at sentence granularity (the model accepts paragraph-
|
||||
length input with multiple tagged segments).
|
||||
|
||||
## Version bump
|
||||
|
||||
|
||||
Reference in New Issue
Block a user