vh
54fef0e9d8
IndexTTS-2's tts.infer(stream_return=True) is a generator that yields
audio chunks per text segment as they finish, plus inter-segment
silence. Expose this via the existing POST /v1/audio/speech with a new
"stream": true field on the request body.
Wire-up:
- 44-byte WAV header emitted up front with placeholder data length
(0xFFFFFFFF) so chunks can be written before total samples are
known. Players that read until EOF (mpv, ffplay, aplay, sox,
browsers via <audio>) handle this fine.
- Each yielded chunk goes through _chunk_to_pcm_bytes(), which
handles torch tensors / numpy arrays in either int16 or float
(-1..1) form.
- 22050 Hz mono int16 — IndexTTS-2's hardcoded output shape.
Time-to-first-audio drops from full-file latency to ~one-segment
latency. Single-sentence inputs barely benefit; long passages /
multi-paragraph reads benefit a lot. Strict metadata parsers may
balk at the placeholder size — request without stream for a
closed-length WAV in that case.
INDEX_TTS_TAG bumped to v2 to force a rebuild.
181 lines
6.7 KiB
Markdown
181 lines
6.7 KiB
Markdown
# IndexTTS-2
|
|
|
|
Bilibili's emotion-controllable zero-shot TTS
|
|
([paper](https://arxiv.org/abs/2506.21619),
|
|
[code](https://github.com/index-tts/index-tts),
|
|
[weights](https://huggingface.co/IndexTeam/IndexTTS-2)) served behind
|
|
our own thin FastAPI wrapper.
|
|
|
|
## Why this stack exists alongside the other two TTS
|
|
|
|
| | CosyVoice 3 | Qwen3-TTS-1.7B-Base | **IndexTTS-2** |
|
|
|---|---|---|---|
|
|
| Voice cloning | ✅ | ✅ (`-Base` variant only) | ✅ |
|
|
| English quality | medium (Chinese-leaning) | high (English-first) | medium (better than CosyVoice) |
|
|
| Emotion control | `instruct` mode is Chinese-only | inline tags | **explicit: audio / 8-vector / text** |
|
|
| Duration control | implicit | implicit | **explicit token-count mode** |
|
|
| License | Apache 2.0 | Apache 2.0 | custom (Bilibili — free at our scale) |
|
|
| Wrapper | bare CosyVoice CLI | groxaxo upstream FastAPI | ours, in this dir |
|
|
|
|
The differentiator is **disentangled emotion**. IndexTTS-2 lets you
|
|
clone a voice's timbre from one reference and the emotion from a
|
|
different reference — or skip emotion-audio entirely and supply an
|
|
8-vector or a text description. Neither of the other two does this
|
|
cleanly in English.
|
|
|
|
## API
|
|
|
|
OpenAI-compat-ish:
|
|
|
|
```bash
|
|
# List available voices + emotions
|
|
curl http://10.100.79.3:8192/v1/voices
|
|
|
|
# Basic synthesis (uses speaker WAV's natural emotion)
|
|
curl -X POST http://10.100.79.3:8192/v1/audio/speech \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{
|
|
"input": "I have all the time in the world.",
|
|
"voice": "glados"
|
|
}' > glados.wav
|
|
|
|
# Same speaker, emotion taken from a separate reference WAV
|
|
curl -X POST http://10.100.79.3:8192/v1/audio/speech \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{
|
|
"input": "I have all the time in the world.",
|
|
"voice": "glados",
|
|
"emotion_voice": "menacing",
|
|
"emotion_alpha": 0.9
|
|
}' > glados-menacing.wav
|
|
|
|
# Same speaker, emotion as 8-vector
|
|
# Order: happy, angry, sad, afraid, disgusted, melancholic, surprised, calm
|
|
curl -X POST http://10.100.79.3:8192/v1/audio/speech \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{
|
|
"input": "I have all the time in the world.",
|
|
"voice": "glados",
|
|
"emotion_vector": [0, 0.7, 0, 0, 0.2, 0, 0, 0]
|
|
}' > glados-angry.wav
|
|
|
|
# Same speaker, emotion derived from text by bundled QwenEmotion model
|
|
curl -X POST http://10.100.79.3:8192/v1/audio/speech \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{
|
|
"input": "I have all the time in the world.",
|
|
"voice": "glados",
|
|
"emotion_text": "she said with quiet menace"
|
|
}' > glados-menacing.wav
|
|
|
|
# Health
|
|
curl http://10.100.79.3:8192/healthz
|
|
```
|
|
|
|
Output is always WAV (PCM_16, 22050 Hz mono — IndexTTS-2's native rate).
|
|
`response_format` other than `wav` is rejected.
|
|
|
|
### Streaming (since 0.2.0)
|
|
|
|
Add `"stream": true` to any request to stream the WAV as it generates.
|
|
IndexTTS-2 emits one chunk per text segment (~120 tokens) as soon as it
|
|
finishes synthesizing it; long inputs start playing while the rest is
|
|
still being generated.
|
|
|
|
```bash
|
|
# Pipe straight into a player. Time-to-first-audio drops from
|
|
# whole-file latency to ~one-segment latency.
|
|
curl -fsS -X POST http://10.100.79.3:8192/v1/audio/speech \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"input":"long passage of text...","voice":"glados","stream":true}' \
|
|
| mpv --no-cache -
|
|
|
|
# Or save while playing (tee).
|
|
curl -fsS -X POST http://10.100.79.3:8192/v1/audio/speech \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"input":"...","voice":"glados","stream":true}' \
|
|
| tee out.wav | mpv -
|
|
```
|
|
|
|
The streaming WAV uses a placeholder data-length in the header
|
|
(`0xFFFFFFFF`) so chunks can be written before the total is known.
|
|
Players that read until EOF (mpv, ffplay, aplay, sox, browsers via
|
|
`<audio>`) handle this fine. Strict parsers (some metadata extractors,
|
|
foobar2000 default settings) may complain about the size. If that
|
|
matters, request without `stream` and you get a normal closed-length
|
|
WAV.
|
|
|
|
## Voice library
|
|
|
|
Flat dirs on the host (bind-mounted; survives container recreates):
|
|
|
|
```
|
|
/worktank/index-tts/voices/<name>.wav # timbre references
|
|
/worktank/index-tts/emotions/<name>.wav # emotion references
|
|
```
|
|
|
|
Drop a new WAV in either dir and `/v1/voices` picks it up immediately —
|
|
no restart. Use clean reference clips, 5-30 s each, single speaker.
|
|
Cloned voices live under restic; cache (model weights) is excluded.
|
|
|
|
## Deploy
|
|
|
|
```bash
|
|
scripts/elway irv-ml1 --playbook playbooks/deploy-index-tts.yaml
|
|
```
|
|
|
|
First build: ~5-10 min for the docker image (CUDA torch + IndexTTS
|
|
deps), plus ~5-7 GB model download on first container start. Subsequent
|
|
starts: ~30 s warmup.
|
|
|
|
## Switching GPUs
|
|
|
|
irv-ml1 has an RTX 3090 (cuda:0) + RTX A6000 (cuda:1). Default is
|
|
auto-pick (cuda:0). To pin to the A6000 alongside Qwen3-TTS-on-3090:
|
|
|
|
```bash
|
|
ssh irv-ml1 '
|
|
cd /opt/docker/compose/index-tts
|
|
sed -i "s|^INDEX_TTS_DEVICE=.*|INDEX_TTS_DEVICE=cuda:1|" .env
|
|
docker compose up -d
|
|
'
|
|
```
|
|
|
|
## Gotchas
|
|
|
|
- **License** — IndexTeam/IndexTTS-2 ships under a custom Bilibili
|
|
license, not Apache/MIT. Free at our scale (the commercial tier kicks
|
|
in at 100M MAU / RMB 1B revenue). Restricts using outputs to train
|
|
other AI models. Read `INDEX_MODEL_LICENSE` in the HF repo before
|
|
using outputs anywhere external.
|
|
- **Sample rate** — 22050 Hz is hardcoded upstream. If you need 24 kHz
|
|
or 48 kHz, resample in the caller.
|
|
- **Emotion-source precedence** — if multiple emotion controls are
|
|
specified in one request, the first non-empty one wins in this order:
|
|
`emotion_voice` > `emotion_vector` > `emotion_text`. The others are
|
|
silently ignored.
|
|
- **Model download** — happens in the entrypoint on first start; the
|
|
config.yaml file in the cache dir is the gate. To force a re-download,
|
|
delete that file and recreate the container.
|
|
- **Bundled example WAVs are LFS pointers, not audio.** Upstream stores
|
|
`examples/emo_*.wav` and `examples/voice_*.wav` as Git LFS objects.
|
|
The image clones the repo without `git lfs pull` (the index-tts org
|
|
has exhausted GitHub's LFS bandwidth budget repeatedly, so doing it
|
|
in the Dockerfile aborts the build). If you want the IndexTTS-2
|
|
example clips as starter material, fetch them once via the media
|
|
CDN — that's a separate code path that doesn't count against the
|
|
LFS API budget:
|
|
|
|
```bash
|
|
ssh irv-ml1 '
|
|
cd /worktank/index-tts/emotions
|
|
curl -fsSL -o hate.wav https://media.githubusercontent.com/media/index-tts/index-tts/main/examples/emo_hate.wav
|
|
curl -fsSL -o sad.wav https://media.githubusercontent.com/media/index-tts/index-tts/main/examples/emo_sad.wav
|
|
'
|
|
```
|
|
- **HF cache pinning** — `infer_v2.py` pins `HF_HUB_CACHE` at import
|
|
time to `./checkpoints/hf_cache`. The wrapper sets this env var
|
|
before importing, so auxiliary HF assets (MaskGCT, campplus, BigVGAN,
|
|
w2v-bert) land alongside the IndexTTS-2 weights and are excluded
|
|
from restic together.
|