Files

T

vh 68f3cd05fe voxtral: mount patched stage YAML to dodge hardcoded 0.8 GPU util; fish-s2: --half + streaming wins

Voxtral final fix (8th iteration):
* The bundled voxtral_tts.yaml hardcodes gpu_memory_utilization: 0.8
  on the language_model stage — overrides the CLI flag. Mounted a
  patched copy (0.4) at /etc/voxtral/voxtral_tts.yaml and pointed
  --stage-configs-path there.
* With Kyutai stopped to free 5 GB on the 3090, both stages fit
  (target 9.4 + 2.4 GB ≈ 11.8 GB; 17 GB free post-kyutai-stop).
* Voxtral now healthy on GPU 0 — bench: 1.9-2.7 s TTFB, real WAV.

Fish s2-pro optimization (per-request sweep, no model swap):
* `streaming: true` in request body drops TTFB from 7.7 s → 0.026 s
  (300×). Total time goes up ~1 s (chunked HTTP overhead) but
  perceived latency = TTFB. Use stream:true for any interactive use.
* `latency: "balanced"` actually slower than default — bad name; skip.
* `use_memory_cache: "on"` no measurable benefit.
* `chunk_length: 100` (default 200) no TTFB benefit non-streaming.
* Server-side `--half` (fp16 inference) added via compose `command`
  override — passes through start_server.sh's $@ unchanged into
  api_server.py. Should reduce total time too. Validation pending
  the post-restart bench.

Kyutai stopped to free GPU 0 budget — the bench numbers earlier
(3.4 s avg) were unimpressive vs Voxtral's 2.3 s in the same
multilingual slot. Kept the stack files for future re-deploy if
needed; just the running container is gone.

2026-04-28 00:35:55 -07:00

.env.example

voxtral + kyutai-tts: fix wrong image tag / wrong endpoint paths; fish-s2: env-selectable model variant

2026-04-27 23:54:59 -07:00

compose.yaml

voxtral: mount patched stage YAML to dodge hardcoded 0.8 GPU util; fish-s2: --half + streaming wins

2026-04-28 00:35:55 -07:00

README.md

fish-s2: docs + verify reflect actual API (POST /v1/tts, not OpenAI-compat)

2026-04-27 23:28:02 -07:00

README.md

Fish Audio S2-Pro

fishaudio/s2-pro — the most expressive open-source TTS model as of 2026-04, served via the official fishaudio/fish-speech inference engine.

Why this stack exists

Three of the existing TTS already cover the basics — Kokoro for raw speed, Chatterbox for speed-with-cloning, IndexTTS-2 for precision emotion control. Fish Audio S2-Pro fills a different slot: dramatically richer paralinguistic control via natural-language tags (15,000+ vs Chatterbox Turbo's 9 fixed tags), with comparable latency (~150 ms streaming) and voice cloning.

Released March 9, 2026; we missed it during the original irv-ml1 build-out in early April.

	use case
Fish Audio S2-Pro	richest emotive / paralinguistic English TTS — 15k+ tags
Kokoro	low-latency English, fixed voice library
Chatterbox Turbo	low-latency English w/ cloning + 9 paralinguistic tags
IndexTTS-2	English voice cloning + emotion vector / text control
Qwen3-TTS-1.7B	English voice cloning (slow on official backend)
CosyVoice 3	multilingual (Chinese-leaning)
VibeVoice 1.5B	long-form / multi-speaker dialogue

Architecture

Dual-AR (Slow + Fast):

Slow AR operates along the time axis, predicts the primary semantic codebook.
Fast AR generates the remaining 9 residual codebooks per time step, reconstructing fine-grained acoustic detail.

Trained on 10M+ hours of audio across 80+ languages with reinforcement-learning alignment. Win rates per upstream:

benchmark	S2-Pro
EmergentTTS-Eval paralinguistics	91.61%
Blind A/B vs ElevenLabs Flash v2.5 (multilingual)	strong

Headline features

15,000+ paralinguistic / emotion tags via natural language:
```
[laugh] [whispers] [super happy] [sigh] [excited] [heavy breathing]
[angry] [sleepy] [crying] [surprise] ...
```
Drop them inline in the input text. Different shape from IndexTTS-2's 8-vector emotion control — this is "say it like this" markup directly in the prompt, with a far larger vocabulary.
Voice cloning from ~5-15 s reference WAV.
Multi-speaker / multi-turn generation natively supported.
80+ languages (English-strong, not Chinese-leaning like CosyVoice).

API

Fish ships a custom API, NOT OpenAI-compatible. The wrapper has exactly one TTS endpoint (POST /v1/tts) plus liveness probes — no /v1/audio/speech, no /v1/audio/voices, no /v1/models. Voice cloning happens via the references field in the request body (pointing at files under /app/references).

# Minimal POST — text only, default voice.
curl -fsS -X POST http://10.100.79.3:8195/v1/tts \
  -H 'Content-Type: application/json' \
  -d '{"text":"Oh wow [super happy] I cannot believe it. [laugh] What a day."}' \
  > out.wav

# With voice cloning — point at a reference file (drop the .wav into
# /worktank/fish-s2/references/ on the host first).
curl -fsS -X POST http://10.100.79.3:8195/v1/tts \
  -H 'Content-Type: application/json' \
  -d '{"text":"...", "references":[{"audio":"/app/references/glados.wav","text":"transcript of the reference"}]}' \
  > out.wav

Other endpoints:

path	purpose
`GET /v1/health`	liveness probe (used by our Docker healthcheck)
`GET /heartbeat`	alternate liveness signal
`GET /`	Swagger Editor UI for the OpenAPI spec

The 200-line OpenAPI spec is rendered through Swagger Editor at the root path; there's no /openapi.json endpoint exposed directly.

Voice library

Drop reference WAV / MP3 / FLAC into /worktank/fish-s2/references/ on the host. The wrapper scans on request — no restart needed. Use clean ~5-15 s clips, single speaker, ideally with diverse intonation samples.

Deploy

scripts/elway irv-ml1 --playbook playbooks/deploy-fish-s2.yaml

First boot pulls the s2-pro checkpoint (~9 GB BF16) into the HF cache + warms torch.compile (adds ~60 s). Both are cached afterwards.

Hardware footprint

VRAM: ~17 GB practical, 24 GB recommended. Pinned to GPU 1 (RTX A6000) by default — plenty of headroom for long contexts and large mmproj if a future checkpoint adds vision.
Disk: ~11 GB for the s2-pro checkpoint (codec.pth 1.9 GB + 2 safetensors shards 9 GB + tokenizer/config).

Lessons learned during deploy (2026-04-27)

Took 5 iterations to land. Recording for next time:

dockerfile (lowercase, root) — doesn't exist. Fish doesn't ship a plain Dockerfile.
dockerfile.dev — exists at root, but it's a 2-line wrapper (FROM ghcr.io/fishaudio/fish-speech:${VERSION}) over a private GHCR base image. Anonymous pulls 403.
docker/Dockerfile — the real build path (referenced by upstream's compose.base.yml).
Multi-stage default builds the wrong target. Without target: server, docker builds the last stage which is webui (gradio only — no start_server.sh, container exits silently rc=0 because the API entrypoint is missing).
Fish doesn't auto-download checkpoints. start_server.sh validates /app/checkpoints/s2-pro/ exists and exits clean if not. The playbook now pre-pulls fishaudio/s2-pro (~11 GB) via a one-shot huggingface_hub.snapshot_download container before starting the service.
API is NOT OpenAI-compatible. Endpoint is POST /v1/tts, not /v1/audio/speech. No /v1/audio/voices or /v1/models. Voice cloning is via references field in the POST body.