esh-pfi-infrastructure/stacks/fish-s2/README.md

Fish Audio S2-Pro

fishaudio/s2-pro — the most expressive open-source TTS model as of 2026-04, served via the official fishaudio/fish-speech inference engine.

Why this stack exists

Three of the existing TTS already cover the basics — Kokoro for raw speed, Chatterbox for speed-with-cloning, IndexTTS-2 for precision emotion control. Fish Audio S2-Pro fills a different slot: dramatically richer paralinguistic control via natural-language tags (15,000+ vs Chatterbox Turbo's 9 fixed tags), with comparable latency (~150 ms streaming) and voice cloning.

Released March 9, 2026; we missed it during the original irv-ml1 build-out in early April.

	use case
Fish Audio S2-Pro	richest emotive / paralinguistic English TTS — 15k+ tags
Kokoro	low-latency English, fixed voice library
Chatterbox Turbo	low-latency English w/ cloning + 9 paralinguistic tags
IndexTTS-2	English voice cloning + emotion vector / text control
Qwen3-TTS-1.7B	English voice cloning (slow on official backend)
CosyVoice 3	multilingual (Chinese-leaning)
VibeVoice 1.5B	long-form / multi-speaker dialogue

Architecture

Dual-AR (Slow + Fast):

• Slow AR operates along the time axis, predicts the primary semantic codebook.
• Fast AR generates the remaining 9 residual codebooks per time step, reconstructing fine-grained acoustic detail.

Trained on 10M+ hours of audio across 80+ languages with reinforcement-learning alignment. Win rates per upstream:

benchmark	S2-Pro
EmergentTTS-Eval paralinguistics	91.61%
Blind A/B vs ElevenLabs Flash v2.5 (multilingual)	strong

Headline features

• 15,000+ paralinguistic / emotion tags via natural language:

  [laugh] [whispers] [super happy] [sigh] [excited] [heavy breathing]
  [angry] [sleepy] [crying] [surprise] ...
  

  Drop them inline in the input text. Different shape from IndexTTS-2's 8-vector emotion control — this is "say it like this" markup directly in the prompt, with a far larger vocabulary.
• Voice cloning from ~5-15 s reference WAV.
• Multi-speaker / multi-turn generation natively supported.
• 80+ languages (English-strong, not Chinese-leaning like CosyVoice).

API

Fish ships a custom API, NOT OpenAI-compatible. The wrapper has exactly one TTS endpoint (POST /v1/tts) plus liveness probes — no /v1/audio/speech, no /v1/audio/voices, no /v1/models. Voice cloning happens via the references field in the request body (pointing at files under /app/references).

# Minimal POST — text only, default voice.
curl -fsS -X POST http://10.100.79.3:8195/v1/tts \
  -H 'Content-Type: application/json' \
  -d '{"text":"Oh wow [super happy] I cannot believe it. [laugh] What a day."}' \
  > out.wav

# With voice cloning — point at a reference file (drop the .wav into
# /worktank/fish-s2/references/ on the host first).
curl -fsS -X POST http://10.100.79.3:8195/v1/tts \
  -H 'Content-Type: application/json' \
  -d '{"text":"...", "references":[{"audio":"/app/references/glados.wav","text":"transcript of the reference"}]}' \
  > out.wav

Other endpoints:

path	purpose
GET /v1/health	liveness probe (used by our Docker healthcheck)
GET /heartbeat	alternate liveness signal
GET /	Swagger Editor UI for the OpenAPI spec

The 200-line OpenAPI spec is rendered through Swagger Editor at the root path; there's no /openapi.json endpoint exposed directly.

Voice library

Drop reference WAV / MP3 / FLAC into /worktank/fish-s2/references/ on the host. The wrapper scans on request — no restart needed. Use clean ~5-15 s clips, single speaker, ideally with diverse intonation samples.

Deploy

scripts/elway irv-ml1 --playbook playbooks/deploy-fish-s2.yaml

First boot pulls the s2-pro checkpoint (~9 GB BF16) into the HF cache + warms torch.compile (adds ~60 s). Both are cached afterwards.

Hardware footprint

• VRAM: ~17 GB practical, 24 GB recommended. Pinned to GPU 1 (RTX A6000) by default — plenty of headroom for long contexts and large mmproj if a future checkpoint adds vision.
• Disk: ~11 GB for the s2-pro checkpoint (codec.pth 1.9 GB + 2 safetensors shards 9 GB + tokenizer/config).

Lessons learned during deploy (2026-04-27)

Took 5 iterations to land. Recording for next time:

1. dockerfile (lowercase, root) — doesn't exist. Fish doesn't ship a plain Dockerfile.
2. dockerfile.dev — exists at root, but it's a 2-line wrapper (FROM ghcr.io/fishaudio/fish-speech:${VERSION}) over a private GHCR base image. Anonymous pulls 403.
3. docker/Dockerfile — the real build path (referenced by upstream's compose.base.yml).
4. Multi-stage default builds the wrong target. Without target: server, docker builds the last stage which is webui (gradio only — no start_server.sh, container exits silently rc=0 because the API entrypoint is missing).
5. Fish doesn't auto-download checkpoints. start_server.sh validates /app/checkpoints/s2-pro/ exists and exits clean if not. The playbook now pre-pulls fishaudio/s2-pro (~11 GB) via a one-shot huggingface_hub.snapshot_download container before starting the service.
6. API is NOT OpenAI-compatible. Endpoint is POST /v1/tts, not /v1/audio/speech. No /v1/audio/voices or /v1/models. Voice cloning is via references field in the POST body.