vh
68f3cd05fe
Voxtral final fix (8th iteration): * The bundled voxtral_tts.yaml hardcodes gpu_memory_utilization: 0.8 on the language_model stage — overrides the CLI flag. Mounted a patched copy (0.4) at /etc/voxtral/voxtral_tts.yaml and pointed --stage-configs-path there. * With Kyutai stopped to free 5 GB on the 3090, both stages fit (target 9.4 + 2.4 GB ≈ 11.8 GB; 17 GB free post-kyutai-stop). * Voxtral now healthy on GPU 0 — bench: 1.9-2.7 s TTFB, real WAV. Fish s2-pro optimization (per-request sweep, no model swap): * `streaming: true` in request body drops TTFB from 7.7 s → 0.026 s (300×). Total time goes up ~1 s (chunked HTTP overhead) but perceived latency = TTFB. Use stream:true for any interactive use. * `latency: "balanced"` actually slower than default — bad name; skip. * `use_memory_cache: "on"` no measurable benefit. * `chunk_length: 100` (default 200) no TTFB benefit non-streaming. * Server-side `--half` (fp16 inference) added via compose `command` override — passes through start_server.sh's $@ unchanged into api_server.py. Should reduce total time too. Validation pending the post-restart bench. Kyutai stopped to free GPU 0 budget — the bench numbers earlier (3.4 s avg) were unimpressive vs Voxtral's 2.3 s in the same multilingual slot. Kept the stack files for future re-deploy if needed; just the running container is gone.
87 lines
4.1 KiB
YAML
87 lines
4.1 KiB
YAML
# Voxtral TTS — Mistral AI's 4B open-weight streaming TTS, served via
|
||
# vLLM-Omni (the production serving stack Mistral co-developed for
|
||
# Voxtral). Released March 28, 2026.
|
||
#
|
||
# Why this stack alongside the existing TTS:
|
||
# * 70 ms model latency, 9.7× realtime — fastest non-Kokoro option.
|
||
# * Multilingual-first (EN strong, plus FR, DE, ES, IT, PT, NL, HI).
|
||
# Different from CosyVoice's Chinese-leaning balance.
|
||
# * 68.4% blind A/B win rate vs ElevenLabs Flash v2.5 in cloning.
|
||
# * vLLM-Omni serving = continuous batching + paged attention — the
|
||
# same mechanism that gave qwen3.6 its speed on llama-swap.
|
||
#
|
||
# LICENSE: CC BY-NC. Personal / research use only. Don't ship Voxtral
|
||
# output in any commercial product without re-licensing from Mistral.
|
||
#
|
||
# All tunables live in .env — edit that, not this file.
|
||
|
||
services:
|
||
voxtral:
|
||
# vLLM-Omni image — Mistral's official partnership for Voxtral
|
||
# serving. Version-pinned via .env.
|
||
image: vllm/vllm-omni:${VOXTRAL_VLLM_TAG}
|
||
container_name: voxtral
|
||
restart: unless-stopped
|
||
runtime: nvidia
|
||
ports:
|
||
- "${VOXTRAL_BIND:-0.0.0.0}:${VOXTRAL_PORT}:8000"
|
||
environment:
|
||
- NVIDIA_VISIBLE_DEVICES=${VOXTRAL_GPU_DEVICES:-1}
|
||
- HF_HOME=/root/.cache/huggingface
|
||
# HF_TOKEN required to dodge 429 rate limits on the model
|
||
# download (HF aggressively throttles unauthenticated IPs that
|
||
# pull large repos repeatedly). Set in .env — see .env.example.
|
||
- HF_TOKEN=${VOXTRAL_HF_TOKEN}
|
||
# vLLM serving args — see https://docs.vllm.ai for full list.
|
||
# We override the default model via cmd args below.
|
||
volumes:
|
||
- ${VOXTRAL_CACHE_DIR}:/root/.cache/huggingface
|
||
- ${VOXTRAL_VOICES_DIR}:/voices:ro
|
||
# Patched stage config — bundled YAML hardcodes
|
||
# gpu_memory_utilization: 0.8 on the language_model stage which
|
||
# OOMs anywhere we have other models resident on the same GPU.
|
||
# Mount our own copy at a custom path; the --stage-configs-path
|
||
# flag below points at it.
|
||
- /opt/docker/conf/voxtral/voxtral_tts.yaml:/etc/voxtral/voxtral_tts.yaml:ro
|
||
# vllm/vllm-omni image has no default ENTRYPOINT or CMD — the
|
||
# container init expected --model=... as argv[0]. Set entrypoint
|
||
# to `vllm serve` (the standard CLI) and pass model as positional
|
||
# + tuning flags via command.
|
||
# Voxtral TTS is a STAGE-BASED pipeline (language_model →
|
||
# acoustic_transformer → audio output), not a flat
|
||
# MistralForCausalLM. vllm-omni's `--omni` mode + a stage config
|
||
# YAML drives this. The standard `vllm serve` errors with
|
||
# "no module named 'acoustic_transformer'" because it tries to
|
||
# load Voxtral as a vanilla Mistral causal LM.
|
||
#
|
||
# Pattern lifted from /workspace/vllm-omni/examples/online_serving/
|
||
# qwen3_tts/run_server.sh (closest analog example in the image).
|
||
# Stage config path is relative to WORKDIR=/workspace/vllm-omni.
|
||
entrypoint: ["vllm-omni", "serve"]
|
||
command:
|
||
- "${VOXTRAL_MODEL:-mistralai/Voxtral-4B-TTS-2603}"
|
||
- "--stage-configs-path=/etc/voxtral/voxtral_tts.yaml"
|
||
- "--host=0.0.0.0"
|
||
- "--port=8000"
|
||
- "--gpu-memory-utilization=${VOXTRAL_GPU_UTIL:-0.45}"
|
||
- "--trust-remote-code"
|
||
- "--omni"
|
||
healthcheck:
|
||
# vLLM-Omni exposes /health for liveness + /v1/models for readiness.
|
||
# /health 200 means the server's listening; /v1/models 200 means
|
||
# the model is loaded and request-ready. Use readiness as the
|
||
# healthy signal so we don't mark it "healthy" before it can
|
||
# accept synthesis requests.
|
||
test: ["CMD-SHELL", "python3 -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/v1/models', timeout=5).status==200 else 1)\""]
|
||
interval: 30s
|
||
timeout: 10s
|
||
retries: 3
|
||
# First boot pulls Voxtral-4B (~8 GB BF16) + warms vLLM. Generous.
|
||
start_period: 600s
|
||
labels:
|
||
- homepage.group=AI Systems
|
||
- homepage.name=Voxtral TTS
|
||
- homepage.icon=mdi-translate
|
||
- homepage.description=Mistral 4B multilingual streaming TTS — 70 ms latency, voice cloning (irv-ml1)
|
||
- homepage.href=http://10.100.79.3:${VOXTRAL_PORT}
|