Files

T

vh b75f020cc9 stacks/index-tts: own FastAPI wrapper for IndexTTS-2 + deploy playbook

Adds a third TTS to the irv-ml1 fleet. IndexTTS-2 is Bilibili's
emotion-controllable zero-shot TTS (paper 2506.21619). Distinguishing
capability vs the existing two: timbre and emotion are disentangled —
clone a voice's timbre from one reference and the emotion from a
different reference, OR set emotion via 8-vector, OR derive it from a
text description. Neither CosyVoice 3 nor Qwen3-TTS-1.7B-Base does
this cleanly in English.

Wrapper is owned end-to-end (~150 lines in app.py) — the only existing
FastAPI fork (csllpr/index-tts-fastapi) targets v1 and is a dormant
single-commit repo. Upstream IndexTTS-2 ships only a Gradio webui.

Layout follows the qwen3-tts pattern:
  stacks/index-tts/
    Dockerfile           — CUDA 12.8 base, IndexTTS pinned to a SHA
    app.py               — FastAPI: POST /v1/audio/speech + /v1/voices
    entrypoint.sh        — one-time HF snapshot_download of the weights
    compose.yaml         — env-driven, GPU pinning support, bind mounts
    .env.example         — port 8192, fp16, paths
    README.md            — API examples + comparison vs the other TTS
  playbooks/deploy-index-tts.yaml  — elway playbook for irv-ml1

Voice and emotion libraries are flat host dirs of WAVs, bind-mounted.
Drop a new <name>.wav and /v1/voices picks it up immediately.

License caveat: IndexTTS-2 weights ship under a custom Bilibili
license (free at our scale, not OSI-open). README documents it.

2026-04-25 10:38:14 -07:00

4.8 KiB

Raw Blame History

IndexTTS-2

Bilibili's emotion-controllable zero-shot TTS (paper, code, weights) served behind our own thin FastAPI wrapper.

Why this stack exists alongside the other two TTS

	CosyVoice 3	Qwen3-TTS-1.7B-Base	IndexTTS-2
Voice cloning	✅	✅ (`-Base` variant only)	✅
English quality	medium (Chinese-leaning)	high (English-first)	medium (better than CosyVoice)
Emotion control	`instruct` mode is Chinese-only	inline tags	explicit: audio / 8-vector / text
Duration control	implicit	implicit	explicit token-count mode
License	Apache 2.0	Apache 2.0	custom (Bilibili — free at our scale)
Wrapper	bare CosyVoice CLI	groxaxo upstream FastAPI	ours, in this dir

The differentiator is disentangled emotion. IndexTTS-2 lets you clone a voice's timbre from one reference and the emotion from a different reference — or skip emotion-audio entirely and supply an 8-vector or a text description. Neither of the other two does this cleanly in English.

API

OpenAI-compat-ish:

# List available voices + emotions
curl http://10.100.79.3:8192/v1/voices

# Basic synthesis (uses speaker WAV's natural emotion)
curl -X POST http://10.100.79.3:8192/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "input": "I have all the time in the world.",
    "voice": "glados"
  }' > glados.wav

# Same speaker, emotion taken from a separate reference WAV
curl -X POST http://10.100.79.3:8192/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "input": "I have all the time in the world.",
    "voice": "glados",
    "emotion_voice": "menacing",
    "emotion_alpha": 0.9
  }' > glados-menacing.wav

# Same speaker, emotion as 8-vector
# Order: happy, angry, sad, afraid, disgusted, melancholic, surprised, calm
curl -X POST http://10.100.79.3:8192/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "input": "I have all the time in the world.",
    "voice": "glados",
    "emotion_vector": [0, 0.7, 0, 0, 0.2, 0, 0, 0]
  }' > glados-angry.wav

# Same speaker, emotion derived from text by bundled QwenEmotion model
curl -X POST http://10.100.79.3:8192/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{
    "input": "I have all the time in the world.",
    "voice": "glados",
    "emotion_text": "she said with quiet menace"
  }' > glados-menacing.wav

# Health
curl http://10.100.79.3:8192/healthz

Output is always WAV (PCM_16, 22050 Hz — IndexTTS-2's native rate). response_format other than wav is rejected.

Voice library

Flat dirs on the host (bind-mounted; survives container recreates):

/worktank/index-tts/voices/<name>.wav        # timbre references
/worktank/index-tts/emotions/<name>.wav      # emotion references

Drop a new WAV in either dir and /v1/voices picks it up immediately — no restart. Use clean reference clips, 5-30 s each, single speaker. Cloned voices live under restic; cache (model weights) is excluded.

Deploy

scripts/elway irv-ml1 --playbook playbooks/deploy-index-tts.yaml

First build: ~5-10 min for the docker image (CUDA torch + IndexTTS deps), plus ~5-7 GB model download on first container start. Subsequent starts: ~30 s warmup.

Switching GPUs

irv-ml1 has an RTX 3090 (cuda:0) + RTX A6000 (cuda:1). Default is auto-pick (cuda:0). To pin to the A6000 alongside Qwen3-TTS-on-3090:

ssh irv-ml1 '
  cd /opt/docker/compose/index-tts
  sed -i "s|^INDEX_TTS_DEVICE=.*|INDEX_TTS_DEVICE=cuda:1|" .env
  docker compose up -d
'

Gotchas

License — IndexTeam/IndexTTS-2 ships under a custom Bilibili license, not Apache/MIT. Free at our scale (the commercial tier kicks in at 100M MAU / RMB 1B revenue). Restricts using outputs to train other AI models. Read INDEX_MODEL_LICENSE in the HF repo before using outputs anywhere external.
Sample rate — 22050 Hz is hardcoded upstream. If you need 24 kHz or 48 kHz, resample in the caller.
Emotion-source precedence — if multiple emotion controls are specified in one request, the first non-empty one wins in this order: emotion_voice > emotion_vector > emotion_text. The others are silently ignored.
Model download — happens in the entrypoint on first start; the config.yaml file in the cache dir is the gate. To force a re-download, delete that file and recreate the container.
HF cache pinning — infer_v2.py pins HF_HUB_CACHE at import time to ./checkpoints/hf_cache. The wrapper sets this env var before importing, so auxiliary HF assets (MaskGCT, campplus, BigVGAN, w2v-bert) land alongside the IndexTTS-2 weights and are excluded from restic together.

4.8 KiB Raw Blame History