cosyvoice: instruct field is Chinese-context-only — English directives get vocalized #7

New Issue

2026-04-24T16:45:08-07:00

vh commented

2026-04-24 16:45:08 -07:00

Symptom

Calling POST /v1/audio/speech with English text in instruct produces audio that sounds like Chinese-flavored phonemes reading the English directive aloud, rather than English speech in the requested tone.

User-reported example, request:

{"model":"cosyvoice-v3","voice":"<id>","input":"You should leave now.","instruct":"speak with cold contempt","response_format":"wav"}

Audible result: a string of Chinese-sounding speech, not the English input.

Root cause (verified 2026-04-24)

The neosun/cosyvoice wrapper maps the instruct field to CosyVoice's inference_instruct2, which expects a Chinese-template directive of the form:

You are a helpful assistant. <directive>请用…<|endofprompt|>

(per FunAudioLLM/CosyVoice upstream docs; see the model card on HF).

When instruct is set, the wrapper concatenates the user value into that template position. English directives (or any non-Chinese-template text) get treated as content rather than direction — the model vocalizes them with Chinese phoneme tendencies.

Empirical bracket

Five variants tested, byte-for-byte response sizes against voice=9053c5a52f80, model=cosyvoice-v3:

Variant	`instruct` value	Response size	Behavior
A	(omitted)	92 KB	English speech, no emotion ✅
B	`""` (empty string)	27 KB	Truncated/garbled
C	`"speak with cold contempt"`	192,078 B	Chinese-flavored, reads instruction
D	`"用冷酷蔑视的语气说"`	192,078 B	Same byte count as C
G	`"Use English. Speak with cold contempt."`	192,078 B	Same byte count
I	`"用冷酷蔑视的语气朗读以下英文"`	192,078 B	Same byte count

C/D/G/I all producing identical 192,078-byte responses confirms the wrapper's instruct path is deterministic on string length and not sensitive to English-vs-Chinese semantics — the model is stuck in a Chinese-context branch regardless of input.

In contrast, XML inline tags work cleanly:

Variant	input	Size	Behavior
E	`<angry>You should leave now.</angry>`	110 KB	English speech, angry tone ✅
H	`<sad>You should leave now.</sad>`	184 KB	English speech, sad tone (longer due to slower cadence) ✅

Workaround

For English content: use XML inline tags exclusively. Do not pass instruct (or instruct_text on /api/tts) for English work. README updated.

Possible upstream paths

The neosun/cosyvoice wrapper could detect English directives and either reject them or build a different prompt template. File issue against neosun100/cosyvoice-docker.
The CosyVoice 3 model itself may need an English-context instruct format that this wrapper isn't using. Worth checking FunAudioLLM/CosyVoice for an inference_instruct_en or similar.
Live with XML-tag-only English emotion control — perfectly serviceable for the use cases we have today.

Likely outcome: keep XML-tag pattern, file upstream issue when motivated.

## Symptom Calling `POST /v1/audio/speech` with English text in `instruct` produces audio that sounds like Chinese-flavored phonemes reading the English directive aloud, rather than English speech in the requested tone. User-reported example, request: ```json {"model":"cosyvoice-v3","voice":"<id>","input":"You should leave now.","instruct":"speak with cold contempt","response_format":"wav"} ``` Audible result: a string of Chinese-sounding speech, not the English `input`. ## Root cause (verified 2026-04-24) The neosun/cosyvoice wrapper maps the `instruct` field to CosyVoice's `inference_instruct2`, which expects a Chinese-template directive of the form: ``` You are a helpful assistant. <directive>请用…<|endofprompt|> ``` (per FunAudioLLM/CosyVoice upstream docs; see the model card on HF). When `instruct` is set, the wrapper concatenates the user value into that template position. English directives (or any non-Chinese-template text) get treated as content rather than direction — the model vocalizes them with Chinese phoneme tendencies. ## Empirical bracket Five variants tested, byte-for-byte response sizes against `voice=9053c5a52f80`, model=cosyvoice-v3: | Variant | `instruct` value | Response size | Behavior | |---|---|---|---| | A | (omitted) | 92 KB | English speech, no emotion ✅ | | B | `""` (empty string) | 27 KB | Truncated/garbled | | C | `"speak with cold contempt"` | **192,078 B** | Chinese-flavored, reads instruction | | D | `"用冷酷蔑视的语气说"` | **192,078 B** | Same byte count as C | | G | `"Use English. Speak with cold contempt."` | **192,078 B** | Same byte count | | I | `"用冷酷蔑视的语气朗读以下英文"` | **192,078 B** | Same byte count | C/D/G/I all producing identical 192,078-byte responses confirms the wrapper's instruct path is deterministic on string length and not sensitive to English-vs-Chinese semantics — the model is stuck in a Chinese-context branch regardless of input. In contrast, XML inline tags work cleanly: | Variant | input | Size | Behavior | |---|---|---|---| | E | `<angry>You should leave now.</angry>` | 110 KB | English speech, angry tone ✅ | | H | `<sad>You should leave now.</sad>` | 184 KB | English speech, sad tone (longer due to slower cadence) ✅ | ## Workaround For English content: use XML inline tags exclusively. Do not pass `instruct` (or `instruct_text` on `/api/tts`) for English work. README updated. ## Possible upstream paths 1. The neosun/cosyvoice wrapper could detect English directives and either reject them or build a different prompt template. File issue against `neosun100/cosyvoice-docker`. 2. The CosyVoice 3 model itself may need an English-context instruct format that this wrapper isn't using. Worth checking FunAudioLLM/CosyVoice for an `inference_instruct_en` or similar. 3. Live with XML-tag-only English emotion control — perfectly serviceable for the use cases we have today. Likely outcome: keep XML-tag pattern, file upstream issue when motivated.

vh added the investigation label 2026-04-24 16:45:08 -07:00

vh referenced this issue from a commit

2026-04-24 16:45:21 -07:00

cosyvoice: warn that \`instruct\` is Chinese-only; XML tags for EN

vh commented

2026-04-24 17:32:54 -07:00

Web-search update

This is the same upstream bug as FunAudioLLM/CosyVoice#1790 ("Fun-CosyVoice3-0.5B-2512 generates Chinese speech instead of Russian regardless of input language") — closed 2026-04-14 with no posted fix. Users on that thread tried, all unsuccessfully:

inference_zero_shot with target-language reference + transcript
inference_cross_lingual
inference_instruct2 with explicit language directive
Language tags like <|ru|>
Different reference audios + transcripts

So the bug is in the model checkpoint (Fun-CosyVoice3-0.5B-2512), not the neosun wrapper specifically — the wrapper's instruct path makes it worse but the underlying Chinese bias affects every inference function.

Wrapper-side details we mapped

Reading /app/app.py inside the running container:

/v1/audio/speech with a custom voice → inference_zero_shot with prompt_text = f"<|endofprompt|>{custom_voice_transcript}" (a wrapper-side fix for upstream #967/#1704 audio-repetition).
/v1/audio/speech with instruct set → inference_instruct2 (the Chinese-template path documented in the README).
/api/tts with mode=cross_lingual → inference_cross_lingual directly, no <|endofprompt|> prefix.
/api/tts with mode=sft requires a registered spk_id; /api/speakers returns [] on this image, so no preset English speakers exist.

Workaround triage (empirically, voice = `9053c5a52f80`, GLaDOS English clone)

All three of these return 200 + valid 24kHz mono WAV:

Variant	Endpoint	Mode	Notes
X1	`/api/tts`	`cross_lingual`	Different code path; bypasses `<
X2	`/api/tts`	`zero_shot` + explicit `prompt_text`	Same call as `/v1/audio/speech` but with caller-controlled prompt
X3	`/api/tts`	`cross_lingual` + `model=cosyvoice-v2`	v2 model, different language-bias defaults

Listening required to know which (if any) actually produce English audio. User report pending.

Fallbacks if no variant works

Re-clone the voice with a longer, more diverse English reference (close to 30s, multiple sentences) to give the speaker embedding a stronger English anchor.
Accept Kokoro for English narration; keep CosyVoice 3 for Chinese/dialect.
Subscribe to upstream issue #1790 and revisit when a fixed checkpoint lands.

Tagging this issue stays open pending user-side audition of X1/X2/X3.

## Web-search update This is the same upstream bug as **FunAudioLLM/CosyVoice#1790** ("Fun-CosyVoice3-0.5B-2512 generates Chinese speech instead of Russian regardless of input language") — closed 2026-04-14 with no posted fix. Users on that thread tried, all unsuccessfully: - `inference_zero_shot` with target-language reference + transcript - `inference_cross_lingual` - `inference_instruct2` with explicit language directive - Language tags like `<|ru|>` - Different reference audios + transcripts So the bug is in the model checkpoint (Fun-CosyVoice3-0.5B-2512), not the neosun wrapper specifically — the wrapper's `instruct` path makes it worse but the underlying Chinese bias affects every inference function. ## Wrapper-side details we mapped Reading `/app/app.py` inside the running container: - `/v1/audio/speech` with a custom voice → `inference_zero_shot` with `prompt_text = f"<|endofprompt|>{custom_voice_transcript}"` (a wrapper-side fix for upstream #967/#1704 audio-repetition). - `/v1/audio/speech` with `instruct` set → `inference_instruct2` (the Chinese-template path documented in the README). - `/api/tts` with `mode=cross_lingual` → `inference_cross_lingual` directly, no `<|endofprompt|>` prefix. - `/api/tts` with `mode=sft` requires a registered `spk_id`; `/api/speakers` returns `[]` on this image, so no preset English speakers exist. ## Workaround triage (empirically, voice = `9053c5a52f80`, GLaDOS English clone) All three of these return 200 + valid 24kHz mono WAV: | Variant | Endpoint | Mode | Notes | |---|---|---|---| | X1 | `/api/tts` | `cross_lingual` | Different code path; bypasses `<|endofprompt|>` prefix | | X2 | `/api/tts` | `zero_shot` + explicit `prompt_text` | Same call as `/v1/audio/speech` but with caller-controlled prompt | | X3 | `/api/tts` | `cross_lingual` + `model=cosyvoice-v2` | v2 model, different language-bias defaults | Listening required to know which (if any) actually produce English audio. User report pending. ## Fallbacks if no variant works 1. Re-clone the voice with a longer, more diverse English reference (close to 30s, multiple sentences) to give the speaker embedding a stronger English anchor. 2. Accept Kokoro for English narration; keep CosyVoice 3 for Chinese/dialect. 3. Subscribe to upstream issue #1790 and revisit when a fixed checkpoint lands. Tagging this issue stays open pending user-side audition of X1/X2/X3.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: vh/esh-pfi-infrastructure#7

cosyvoice: instruct field is Chinese-context-only — English directives get vocalized #7