vh
fe01f73d84
Fourth attempt finally found the right invocation. Voxtral is a
two-stage TTS pipeline (language_model → acoustic_transformer →
audio output), not a flat MistralForCausalLM. Standard `vllm serve`
errored with "no module named 'acoustic_transformer'" because it
loads the model as a vanilla Mistral causal LM.
Pattern from /workspace/vllm-omni/examples/online_serving/
qwen3_tts/run_server.sh (closest in-image analog):
vllm-omni serve <MODEL> \
--stage-configs-path vllm_omni/model_executor/stage_configs/voxtral_tts.yaml \
--host 0.0.0.0 --port 8000 \
--gpu-memory-utilization 0.45 \
--trust-remote-code --omni
Key differences from previous attempt:
* `vllm-omni` binary, not `vllm`
* `--omni` flag activates multi-stage pipeline
* `--stage-configs-path` points at the bundled YAML that maps
stages to GPU + scheduler + worker classes
* Dropped --load-format/--tokenizer-mode/--config-format=mistral
flags — the stage config handles tokenizer_mode internally
* --trust-remote-code is required for the acoustic_transformer
custom code path
Default .env.example now: GPU 0 (3090) with util 0.45 (~10.6 GB
target on 24 GB GPU). The A6000 is fully booked by Fish s2-pro.
81 lines
3.8 KiB
YAML
81 lines
3.8 KiB
YAML
# Voxtral TTS — Mistral AI's 4B open-weight streaming TTS, served via
|
||
# vLLM-Omni (the production serving stack Mistral co-developed for
|
||
# Voxtral). Released March 28, 2026.
|
||
#
|
||
# Why this stack alongside the existing TTS:
|
||
# * 70 ms model latency, 9.7× realtime — fastest non-Kokoro option.
|
||
# * Multilingual-first (EN strong, plus FR, DE, ES, IT, PT, NL, HI).
|
||
# Different from CosyVoice's Chinese-leaning balance.
|
||
# * 68.4% blind A/B win rate vs ElevenLabs Flash v2.5 in cloning.
|
||
# * vLLM-Omni serving = continuous batching + paged attention — the
|
||
# same mechanism that gave qwen3.6 its speed on llama-swap.
|
||
#
|
||
# LICENSE: CC BY-NC. Personal / research use only. Don't ship Voxtral
|
||
# output in any commercial product without re-licensing from Mistral.
|
||
#
|
||
# All tunables live in .env — edit that, not this file.
|
||
|
||
services:
|
||
voxtral:
|
||
# vLLM-Omni image — Mistral's official partnership for Voxtral
|
||
# serving. Version-pinned via .env.
|
||
image: vllm/vllm-omni:${VOXTRAL_VLLM_TAG}
|
||
container_name: voxtral
|
||
restart: unless-stopped
|
||
runtime: nvidia
|
||
ports:
|
||
- "${VOXTRAL_BIND:-0.0.0.0}:${VOXTRAL_PORT}:8000"
|
||
environment:
|
||
- NVIDIA_VISIBLE_DEVICES=${VOXTRAL_GPU_DEVICES:-1}
|
||
- HF_HOME=/root/.cache/huggingface
|
||
# HF_TOKEN required to dodge 429 rate limits on the model
|
||
# download (HF aggressively throttles unauthenticated IPs that
|
||
# pull large repos repeatedly). Set in .env — see .env.example.
|
||
- HF_TOKEN=${VOXTRAL_HF_TOKEN}
|
||
# vLLM serving args — see https://docs.vllm.ai for full list.
|
||
# We override the default model via cmd args below.
|
||
volumes:
|
||
- ${VOXTRAL_CACHE_DIR}:/root/.cache/huggingface
|
||
- ${VOXTRAL_VOICES_DIR}:/voices:ro
|
||
# vllm/vllm-omni image has no default ENTRYPOINT or CMD — the
|
||
# container init expected --model=... as argv[0]. Set entrypoint
|
||
# to `vllm serve` (the standard CLI) and pass model as positional
|
||
# + tuning flags via command.
|
||
# Voxtral TTS is a STAGE-BASED pipeline (language_model →
|
||
# acoustic_transformer → audio output), not a flat
|
||
# MistralForCausalLM. vllm-omni's `--omni` mode + a stage config
|
||
# YAML drives this. The standard `vllm serve` errors with
|
||
# "no module named 'acoustic_transformer'" because it tries to
|
||
# load Voxtral as a vanilla Mistral causal LM.
|
||
#
|
||
# Pattern lifted from /workspace/vllm-omni/examples/online_serving/
|
||
# qwen3_tts/run_server.sh (closest analog example in the image).
|
||
# Stage config path is relative to WORKDIR=/workspace/vllm-omni.
|
||
entrypoint: ["vllm-omni", "serve"]
|
||
command:
|
||
- "${VOXTRAL_MODEL:-mistralai/Voxtral-4B-TTS-2603}"
|
||
- "--stage-configs-path=vllm_omni/model_executor/stage_configs/voxtral_tts.yaml"
|
||
- "--host=0.0.0.0"
|
||
- "--port=8000"
|
||
- "--gpu-memory-utilization=${VOXTRAL_GPU_UTIL:-0.45}"
|
||
- "--trust-remote-code"
|
||
- "--omni"
|
||
healthcheck:
|
||
# vLLM-Omni exposes /health for liveness + /v1/models for readiness.
|
||
# /health 200 means the server's listening; /v1/models 200 means
|
||
# the model is loaded and request-ready. Use readiness as the
|
||
# healthy signal so we don't mark it "healthy" before it can
|
||
# accept synthesis requests.
|
||
test: ["CMD-SHELL", "python3 -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/v1/models', timeout=5).status==200 else 1)\""]
|
||
interval: 30s
|
||
timeout: 10s
|
||
retries: 3
|
||
# First boot pulls Voxtral-4B (~8 GB BF16) + warms vLLM. Generous.
|
||
start_period: 600s
|
||
labels:
|
||
- homepage.group=AI Systems
|
||
- homepage.name=Voxtral TTS
|
||
- homepage.icon=mdi-translate
|
||
- homepage.description=Mistral 4B multilingual streaming TTS — 70 ms latency, voice cloning (irv-ml1)
|
||
- homepage.href=http://10.100.79.3:${VOXTRAL_PORT}
|