vh
68f3cd05fe
Voxtral final fix (8th iteration): * The bundled voxtral_tts.yaml hardcodes gpu_memory_utilization: 0.8 on the language_model stage — overrides the CLI flag. Mounted a patched copy (0.4) at /etc/voxtral/voxtral_tts.yaml and pointed --stage-configs-path there. * With Kyutai stopped to free 5 GB on the 3090, both stages fit (target 9.4 + 2.4 GB ≈ 11.8 GB; 17 GB free post-kyutai-stop). * Voxtral now healthy on GPU 0 — bench: 1.9-2.7 s TTFB, real WAV. Fish s2-pro optimization (per-request sweep, no model swap): * `streaming: true` in request body drops TTFB from 7.7 s → 0.026 s (300×). Total time goes up ~1 s (chunked HTTP overhead) but perceived latency = TTFB. Use stream:true for any interactive use. * `latency: "balanced"` actually slower than default — bad name; skip. * `use_memory_cache: "on"` no measurable benefit. * `chunk_length: 100` (default 200) no TTFB benefit non-streaming. * Server-side `--half` (fp16 inference) added via compose `command` override — passes through start_server.sh's $@ unchanged into api_server.py. Should reduce total time too. Validation pending the post-restart bench. Kyutai stopped to free GPU 0 budget — the bench numbers earlier (3.4 s avg) were unimpressive vs Voxtral's 2.3 s in the same multilingual slot. Kept the stack files for future re-deploy if needed; just the running container is gone.
Voxtral TTS
mistralai/Voxtral-4B-TTS-2603 — Mistral AI's 4B open-weight streaming TTS, served via the vLLM-Omni production serving stack (Mistral co-developed). Released March 28, 2026.
⚠️ License
CC BY-NC. Personal use, research, and internal tooling are fine. Don't ship Voxtral output in any commercial product without re-licensing from Mistral. The other TTS in this fleet (Kokoro, Chatterbox, Fish S2-Pro, IndexTTS-2, Qwen3-TTS, CosyVoice) are all open-licensed and clean for commercial work.
Why this stack exists
Multilingual streaming with serious speed:
| use case | |
|---|---|
| Voxtral | multilingual EN/FR/DE/ES/IT/PT/NL/HI streaming, 70 ms model latency |
| Kokoro | low-latency English, fixed voice library |
| Chatterbox Turbo | low-latency English w/ cloning + 9 paralinguistic tags |
| Fish Audio S2-Pro | richest paralinguistic English (15k+ tags) |
| IndexTTS-2 | English voice cloning + emotion vector / text control |
| Qwen3-TTS-1.7B | English voice cloning (slow on official backend) |
| CosyVoice 3 | multilingual (Chinese-leaning) |
| VibeVoice 1.5B | long-form / multi-speaker dialogue |
Voxtral fills the multilingual + low-latency + cloning slot that's been weak in the fleet (CosyVoice is multilingual but slow on English; nothing else is multilingual at all).
Headline numbers
- 70 ms model latency for a typical 10 s sample (500-char input)
- 9.7× realtime factor
- 68.4% blind A/B win rate vs ElevenLabs Flash v2.5 in voice cloning evaluations
- 8 languages: EN, FR, DE, ES, IT, PT, NL, HI
API
vLLM-Omni serves an OpenAI-compatible API at
http://10.100.79.3:8197/v1:
# Single-shot synthesis.
curl -fsS -X POST http://10.100.79.3:8197/v1/audio/speech \
-H 'Content-Type: application/json' \
-d '{"model":"mistralai/Voxtral-4B-TTS-2603","input":"Hello there.","voice":"alloy","response_format":"wav"}' \
> out.wav
# Streaming.
curl -fsS -X POST http://10.100.79.3:8197/v1/audio/speech \
-H 'Content-Type: application/json' \
-d '{"model":"mistralai/Voxtral-4B-TTS-2603","input":"long passage…","voice":"alloy","stream":true}' \
| mpv --no-cache -
# vLLM-Omni standard endpoints.
curl http://10.100.79.3:8197/v1/models # confirms model loaded
curl http://10.100.79.3:8197/v1/audio/voices # built-in + cloned voices
Deploy
scripts/elway irv-ml1 --playbook playbooks/deploy-voxtral.yaml
First boot pulls Voxtral-4B (~8 GB BF16) into the HF cache + warms vLLM. Both are cached afterwards.
Hardware footprint
- VRAM: ~16 GB practical (8 GB weights + KV + activation). Pinned to GPU 1 (RTX A6000) by default — comfortable headroom. The 3090's 24 GB CAN fit but it's tight for long streaming sessions.
- Disk: ~8 GB for the Voxtral checkpoint + HF cache.
Voice library
Drop reference WAV / FLAC into /worktank/voxtral/voices/ on the
host. The wrapper scans on request — no restart needed.