vh
fe01f73d84
Fourth attempt finally found the right invocation. Voxtral is a
two-stage TTS pipeline (language_model → acoustic_transformer →
audio output), not a flat MistralForCausalLM. Standard `vllm serve`
errored with "no module named 'acoustic_transformer'" because it
loads the model as a vanilla Mistral causal LM.
Pattern from /workspace/vllm-omni/examples/online_serving/
qwen3_tts/run_server.sh (closest in-image analog):
vllm-omni serve <MODEL> \
--stage-configs-path vllm_omni/model_executor/stage_configs/voxtral_tts.yaml \
--host 0.0.0.0 --port 8000 \
--gpu-memory-utilization 0.45 \
--trust-remote-code --omni
Key differences from previous attempt:
* `vllm-omni` binary, not `vllm`
* `--omni` flag activates multi-stage pipeline
* `--stage-configs-path` points at the bundled YAML that maps
stages to GPU + scheduler + worker classes
* Dropped --load-format/--tokenizer-mode/--config-format=mistral
flags — the stage config handles tokenizer_mode internally
* --trust-remote-code is required for the acoustic_transformer
custom code path
Default .env.example now: GPU 0 (3090) with util 0.45 (~10.6 GB
target on 24 GB GPU). The A6000 is fully booked by Fish s2-pro.
51 lines
2.8 KiB
Bash
51 lines
2.8 KiB
Bash
# Voxtral TTS stack tunables. Copy to `.env` on irv-ml1 before
|
|
# deploying.
|
|
|
|
# ── image pin ────────────────────────────────────────────────────────
|
|
# vLLM-Omni image tag (Mistral's partner serving stack for Voxtral).
|
|
# Pin a specific version — vllm/vllm-omni does NOT publish `latest`;
|
|
# `:latest` 404s with "manifest unknown". v0.18.0 was released
|
|
# 2026-03-29, one day after the Voxtral 4B TTS release, and is the
|
|
# first vLLM-Omni cut with Voxtral support.
|
|
VOXTRAL_VLLM_TAG=v0.18.0
|
|
|
|
# Voxtral model on Hugging Face. The 4B variant is the only released
|
|
# checkpoint as of 2026-04. Default BF16 weights are ~8 GB.
|
|
VOXTRAL_MODEL=mistralai/Voxtral-4B-TTS-2603
|
|
|
|
# ── network ──────────────────────────────────────────────────────────
|
|
# Host port (container listens on 8000 internally).
|
|
VOXTRAL_PORT=8197
|
|
VOXTRAL_BIND=0.0.0.0
|
|
|
|
# ── runtime / GPU ────────────────────────────────────────────────────
|
|
# GPU pinning. "0" = RTX 3090 (24 GB), "1" = RTX A6000 (48 GB).
|
|
# Pinned to GPU 0 (3090) — the A6000 is fully booked by Fish S2-Pro
|
|
# (~17 GB) + Qwen3-TTS / IndexTTS-2 / VibeVoice slots. Voxtral 4B BF16
|
|
# needs ~10-12 GB practical (model + small KV); the 3090's 24 GB is
|
|
# enough alongside Kokoro (~1 GB) + Chatterbox (~3 GB) + Kyutai (~6 GB).
|
|
VOXTRAL_GPU_DEVICES=0
|
|
|
|
# vLLM GPU memory utilization fraction (0.0-1.0). On the 3090 alongside
|
|
# Kokoro/Chatterbox/Kyutai (~10 GB used), 0.5 = ~12 GB target gives
|
|
# Voxtral enough room for weights + KV. Bump to 0.85 if Voxtral ever
|
|
# moves to a dedicated GPU.
|
|
VOXTRAL_GPU_UTIL=0.5
|
|
|
|
# ── HuggingFace auth ─────────────────────────────────────────────────
|
|
# HF_TOKEN — required to dodge 429 rate limits on Voxtral download.
|
|
# HF aggressively throttles unauthenticated IPs that pull large repos.
|
|
# Generate one at https://huggingface.co/settings/tokens (a read-only
|
|
# token is sufficient). Without this, the first model download fails
|
|
# mid-stream and vLLM aborts engine init.
|
|
VOXTRAL_HF_TOKEN=
|
|
|
|
# ── persistent storage on the host ───────────────────────────────────
|
|
# HF cache — first start pulls the Voxtral checkpoint (~8 GB) into
|
|
# this dir. Persistent across container recreates.
|
|
VOXTRAL_CACHE_DIR=/worktank/voxtral/hf_cache
|
|
|
|
# Reference voices for cloning. Read-only mount inside the container.
|
|
# Drop ~5-15 s WAV / FLAC clips here.
|
|
VOXTRAL_VOICES_DIR=/worktank/voxtral/voices
|