# Fish Audio S2-Pro

[fishaudio/s2-pro](https://huggingface.co/fishaudio/s2-pro) — the most
expressive open-source TTS model as of 2026-04, served via the
official [fishaudio/fish-speech](https://github.com/fishaudio/fish-speech)
inference engine.

## Why this stack exists

Three of the existing TTS already cover the basics — Kokoro for raw
speed, Chatterbox for speed-with-cloning, IndexTTS-2 for precision
emotion control. Fish Audio S2-Pro fills a different slot:
**dramatically richer paralinguistic control via natural-language
tags** (15,000+ vs Chatterbox Turbo's 9 fixed tags), with comparable
latency (~150 ms streaming) and voice cloning.

Released March 9, 2026; we missed it during the original irv-ml1
build-out in early April.

| | use case |
|---|---|
| **Fish Audio S2-Pro** | richest emotive / paralinguistic English TTS — 15k+ tags |
| Kokoro | low-latency English, fixed voice library |
| Chatterbox Turbo | low-latency English w/ cloning + 9 paralinguistic tags |
| IndexTTS-2 | English voice cloning + emotion vector / text control |
| Qwen3-TTS-1.7B | English voice cloning (slow on official backend) |
| CosyVoice 3 | multilingual (Chinese-leaning) |
| VibeVoice 1.5B | long-form / multi-speaker dialogue |

## Architecture

Dual-AR (Slow + Fast):
- **Slow AR** operates along the time axis, predicts the primary
  semantic codebook.
- **Fast AR** generates the remaining 9 residual codebooks per time
  step, reconstructing fine-grained acoustic detail.

Trained on 10M+ hours of audio across 80+ languages with
reinforcement-learning alignment. Win rates per upstream:

| benchmark | S2-Pro |
|---|---|
| EmergentTTS-Eval paralinguistics | 91.61% |
| Blind A/B vs ElevenLabs Flash v2.5 (multilingual) | strong |

## Headline features

- **15,000+ paralinguistic / emotion tags** via natural language:
  ```
  [laugh] [whispers] [super happy] [sigh] [excited] [heavy breathing]
  [angry] [sleepy] [crying] [surprise] ...
  ```
  Drop them inline in the input text. Different shape from
  IndexTTS-2's 8-vector emotion control — this is "say it like this"
  markup directly in the prompt, with a far larger vocabulary.
- **Voice cloning** from ~5-15 s reference WAV.
- **Multi-speaker / multi-turn** generation natively supported.
- **80+ languages** (English-strong, not Chinese-leaning like
  CosyVoice).

## API

**Fish ships a custom API, NOT OpenAI-compatible.** The wrapper has
exactly one TTS endpoint (`POST /v1/tts`) plus liveness probes — no
`/v1/audio/speech`, no `/v1/audio/voices`, no `/v1/models`. Voice
cloning happens via the `references` field in the request body
(pointing at files under `/app/references`).

```bash
# Minimal POST — text only, default voice.
curl -fsS -X POST http://10.100.79.3:8195/v1/tts \
  -H 'Content-Type: application/json' \
  -d '{"text":"Oh wow [super happy] I cannot believe it. [laugh] What a day."}' \
  > out.wav

# With voice cloning — point at a reference file (drop the .wav into
# /worktank/fish-s2/references/ on the host first).
curl -fsS -X POST http://10.100.79.3:8195/v1/tts \
  -H 'Content-Type: application/json' \
  -d '{"text":"...", "references":[{"audio":"/app/references/glados.wav","text":"transcript of the reference"}]}' \
  > out.wav
```

Other endpoints:

| path | purpose |
|---|---|
| `GET /v1/health` | liveness probe (used by our Docker healthcheck) |
| `GET /heartbeat` | alternate liveness signal |
| `GET /` | Swagger Editor UI for the OpenAPI spec |

The 200-line OpenAPI spec is rendered through Swagger Editor at the
root path; there's no `/openapi.json` endpoint exposed directly.

## Voice library

Named voices are selected via the `reference_id` field, which Fish
resolves against the **directory** `references/<name>/` — NOT a flat
`references/<name>.wav`. Stage each voice as:

```
/worktank/fish-s2/references/<name>/<name>.wav   # clean ~5-15 s, single speaker
/worktank/fish-s2/references/<name>/<name>.lab   # transcript of that clip
```

A flat `references/<name>.wav` (or an empty `references/<name>/` dir)
is **silently ignored** — `reference_id` resolves to nothing and Fish
falls back to its default speaker, so every voice sounds identical.
This was the 2026-06-01 "no accent" root cause; the deploy playbook now
carries a normalize-layout step + an A/B smoke gate (`reference_id`
MUST change the output) to keep it from regressing. The wrapper scans
on request — no restart needed after adding a voice.

## Deploy

```bash
scripts/elway irv-ml1 --playbook playbooks/deploy-fish-s2.yaml
```

First boot pulls the s2-pro checkpoint (~9 GB BF16) into the HF
cache + warms torch.compile (adds ~60 s). Both are cached afterwards.

## Hardware footprint

- **VRAM**: ~17 GB practical, 24 GB recommended. Pinned to GPU 1
  (RTX A6000) by default — plenty of headroom for long contexts and
  large mmproj if a future checkpoint adds vision.
- **Disk**: ~11 GB for the s2-pro checkpoint
  (codec.pth 1.9 GB + 2 safetensors shards 9 GB + tokenizer/config).

## Lessons learned during deploy (2026-04-27)

Took 5 iterations to land. Recording for next time:

1. **`dockerfile`** (lowercase, root) — doesn't exist. Fish doesn't
   ship a plain Dockerfile.
2. **`dockerfile.dev`** — exists at root, but it's a 2-line wrapper
   (`FROM ghcr.io/fishaudio/fish-speech:${VERSION}`) over a private
   GHCR base image. Anonymous pulls 403.
3. **`docker/Dockerfile`** — the real build path (referenced by
   upstream's `compose.base.yml`).
4. **Multi-stage default builds the wrong target.** Without
   `target: server`, docker builds the last stage which is `webui`
   (gradio only — no `start_server.sh`, container exits silently
   rc=0 because the API entrypoint is missing).
5. **Fish doesn't auto-download checkpoints.** `start_server.sh`
   validates `/app/checkpoints/s2-pro/` exists and exits clean if
   not. The playbook now pre-pulls `fishaudio/s2-pro` (~11 GB) via a
   one-shot `huggingface_hub.snapshot_download` container before
   starting the service.
6. **API is NOT OpenAI-compatible.** Endpoint is `POST /v1/tts`, not
   `/v1/audio/speech`. No `/v1/audio/voices` or `/v1/models`. Voice
   cloning is via `references` field in the POST body.