# IndexTTS-2

Bilibili's emotion-controllable zero-shot TTS
([paper](https://arxiv.org/abs/2506.21619),
[code](https://github.com/index-tts/index-tts),
[weights](https://huggingface.co/IndexTeam/IndexTTS-2)) served behind
our own thin FastAPI wrapper.

## Why this stack exists alongside the other two TTS

| | CosyVoice 3 | Qwen3-TTS-1.7B-Base | **IndexTTS-2** |
|---|---|---|---|
| Voice cloning | ✅ | ✅ (`-Base` variant only) | ✅ |
| English quality | medium (Chinese-leaning) | high (English-first) | medium (better than CosyVoice) |
| Emotion control | `instruct` mode is Chinese-only | inline tags | **explicit: audio / 8-vector / text** |
| Duration control | implicit | implicit | **explicit token-count mode** |
| License | Apache 2.0 | Apache 2.0 | custom (Bilibili — free at our scale) |
| Wrapper | bare CosyVoice CLI | groxaxo upstream FastAPI | ours, in this dir |

The differentiator is **disentangled emotion**. IndexTTS-2 lets you
clone a voice's timbre from one reference and the emotion from a
different reference — or skip emotion-audio entirely and supply an
8-vector or a text description. Neither of the other two does this
cleanly in English.

## API

OpenAI-compat-ish:

```bash
# List available voices + emotions
curl http://10.100.79.3:8192/v1/voices

# Basic synthesis (uses speaker WAV's natural emotion)
curl -X POST http://10.100.79.3:8192/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "input": "I have all the time in the world.",
    "voice": "glados"
  }' > glados.wav

# Same speaker, emotion taken from a separate reference WAV
curl -X POST http://10.100.79.3:8192/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "input": "I have all the time in the world.",
    "voice": "glados",
    "emotion_voice": "menacing",
    "emotion_alpha": 0.9
  }' > glados-menacing.wav

# Same speaker, emotion as 8-vector
# Order: happy, angry, sad, afraid, disgusted, melancholic, surprised, calm
curl -X POST http://10.100.79.3:8192/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "input": "I have all the time in the world.",
    "voice": "glados",
    "emotion_vector": [0, 0.7, 0, 0, 0.2, 0, 0, 0]
  }' > glados-angry.wav

# Same speaker, emotion derived from text by bundled QwenEmotion model
curl -X POST http://10.100.79.3:8192/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "input": "I have all the time in the world.",
    "voice": "glados",
    "emotion_text": "she said with quiet menace"
  }' > glados-menacing.wav

# Health
curl http://10.100.79.3:8192/healthz
```

Output is always WAV (PCM_16, 22050 Hz mono — IndexTTS-2's native rate).
`response_format` other than `wav` is rejected.

### Streaming (since 0.2.0)

Add `"stream": true` to any request to stream the WAV as it generates.
IndexTTS-2 emits one chunk per text segment (~120 tokens) as soon as it
finishes synthesizing it; long inputs start playing while the rest is
still being generated.

```bash
# Pipe straight into a player. Time-to-first-audio drops from
# whole-file latency to ~one-segment latency.
curl -fsS -X POST http://10.100.79.3:8192/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"input":"long passage of text...","voice":"glados","stream":true}' \
  | mpv --no-cache -

# Or save while playing (tee).
curl -fsS -X POST http://10.100.79.3:8192/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"input":"...","voice":"glados","stream":true}' \
  | tee out.wav | mpv -
```

The streaming WAV uses a placeholder data-length in the header
(`0xFFFFFFFF`) so chunks can be written before the total is known.
Players that read until EOF (mpv, ffplay, aplay, sox, browsers via
`<audio>`) handle this fine. Strict parsers (some metadata extractors,
foobar2000 default settings) may complain about the size. If that
matters, request without `stream` and you get a normal closed-length
WAV.

## Voice library

Flat dirs on the host (bind-mounted; survives container recreates):

```
/worktank/index-tts/voices/<name>.wav        # timbre references
/worktank/index-tts/emotions/<name>.wav      # emotion references
```

Drop a new WAV in either dir and `/v1/voices` picks it up immediately —
no restart. Use clean reference clips, 5-30 s each, single speaker.
Cloned voices live under restic; cache (model weights) is excluded.

## Deploy

```bash
scripts/elway irv-ml1 --playbook playbooks/deploy-index-tts.yaml
```

First build: ~5-10 min for the docker image (CUDA torch + IndexTTS
deps), plus ~5-7 GB model download on first container start. Subsequent
starts: ~30 s warmup.

## Switching GPUs

irv-ml1 has an RTX 3090 (cuda:0) + RTX A6000 (cuda:1). Default is
auto-pick (cuda:0). To pin to the A6000 alongside Qwen3-TTS-on-3090:

```bash
ssh irv-ml1 '
  cd /opt/docker/compose/index-tts
  sed -i "s|^INDEX_TTS_DEVICE=.*|INDEX_TTS_DEVICE=cuda:1|" .env
  docker compose up -d
'
```

## Gotchas

- **License** — IndexTeam/IndexTTS-2 ships under a custom Bilibili
  license, not Apache/MIT. Free at our scale (the commercial tier kicks
  in at 100M MAU / RMB 1B revenue). Restricts using outputs to train
  other AI models. Read `INDEX_MODEL_LICENSE` in the HF repo before
  using outputs anywhere external.
- **Sample rate** — 22050 Hz is hardcoded upstream. If you need 24 kHz
  or 48 kHz, resample in the caller.
- **Emotion-source precedence** — if multiple emotion controls are
  specified in one request, the first non-empty one wins in this order:
  `emotion_voice` > `emotion_vector` > `emotion_text`. The others are
  silently ignored.
- **Model download** — happens in the entrypoint on first start; the
  config.yaml file in the cache dir is the gate. To force a re-download,
  delete that file and recreate the container.
- **Bundled example WAVs are LFS pointers, not audio.** Upstream stores
  `examples/emo_*.wav` and `examples/voice_*.wav` as Git LFS objects.
  The image clones the repo without `git lfs pull` (the index-tts org
  has exhausted GitHub's LFS bandwidth budget repeatedly, so doing it
  in the Dockerfile aborts the build). If you want the IndexTTS-2
  example clips as starter material, fetch them once via the media
  CDN — that's a separate code path that doesn't count against the
  LFS API budget:

  ```bash
  ssh irv-ml1 '
    cd /worktank/index-tts/emotions
    curl -fsSL -o hate.wav https://media.githubusercontent.com/media/index-tts/index-tts/main/examples/emo_hate.wav
    curl -fsSL -o sad.wav  https://media.githubusercontent.com/media/index-tts/index-tts/main/examples/emo_sad.wav
  '
  ```
- **HF cache pinning** — `infer_v2.py` pins `HF_HUB_CACHE` at import
  time to `./checkpoints/hf_cache`. The wrapper sets this env var
  before importing, so auxiliary HF assets (MaskGCT, campplus, BigVGAN,
  w2v-bert) land alongside the IndexTTS-2 weights and are excluded
  from restic together.