vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

voxtral: switch to vllm-omni serve --omni with stage config — Voxtral is a multi-stage pipeline

Browse Source 
Fourth attempt finally found the right invocation. Voxtral is a
two-stage TTS pipeline (language_model → acoustic_transformer →
audio output), not a flat MistralForCausalLM. Standard `vllm serve`
errored with "no module named 'acoustic_transformer'" because it
loads the model as a vanilla Mistral causal LM.

Pattern from /workspace/vllm-omni/examples/online_serving/
qwen3_tts/run_server.sh (closest in-image analog):

  vllm-omni serve <MODEL> \
    --stage-configs-path vllm_omni/model_executor/stage_configs/voxtral_tts.yaml \
    --host 0.0.0.0 --port 8000 \
    --gpu-memory-utilization 0.45 \
    --trust-remote-code --omni

Key differences from previous attempt:
  * `vllm-omni` binary, not `vllm`
  * `--omni` flag activates multi-stage pipeline
  * `--stage-configs-path` points at the bundled YAML that maps
    stages to GPU + scheduler + worker classes
  * Dropped --load-format/--tokenizer-mode/--config-format=mistral
    flags — the stage config handles tokenizer_mode internally
  * --trust-remote-code is required for the acoustic_transformer
    custom code path

Default .env.example now: GPU 0 (3090) with util 0.45 (~10.6 GB
target on 24 GB GPU). The A6000 is fully booked by Fish s2-pro.
This commit is contained in:
vh
2026-04-28 00:17:21 -07:00
parent 0304464b7d
commit fe01f73d84
2 changed files with 26 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/voxtral/.env.example
+10 -8
View File
@@ -20,15 +20,17 @@ VOXTRAL_BIND=0.0.0.0
# ── runtime / GPU ────────────────────────────────────────────────────
# GPU pinning. "0" = RTX 3090 (24 GB), "1" = RTX A6000 (48 GB).
# Voxtral 4B BF16 needs ~16 GB practical (model + KV + activation).
# Pinned to A6000 by default for headroom. The 3090 fits but is tight
# for long streaming sessions.
VOXTRAL_GPU_DEVICES=1
# Pinned to GPU 0 (3090) — the A6000 is fully booked by Fish S2-Pro
# (~17 GB) + Qwen3-TTS / IndexTTS-2 / VibeVoice slots. Voxtral 4B BF16
# needs ~10-12 GB practical (model + small KV); the 3090's 24 GB is
# enough alongside Kokoro (~1 GB) + Chatterbox (~3 GB) + Kyutai (~6 GB).
VOXTRAL_GPU_DEVICES=0
# vLLM GPU memory utilization fraction (0.0-1.0). 0.85 = leave 15%
# headroom for other processes / KV cache spikes. Lower if running
# alongside other GPU workloads on the same device.
VOXTRAL_GPU_UTIL=0.85
# vLLM GPU memory utilization fraction (0.0-1.0). On the 3090 alongside
# Kokoro/Chatterbox/Kyutai (~10 GB used), 0.5 = ~12 GB target gives
# Voxtral enough room for weights + KV. Bump to 0.85 if Voxtral ever
# moves to a dedicated GPU.
VOXTRAL_GPU_UTIL=0.5
# ── HuggingFace auth ─────────────────────────────────────────────────
# HF_TOKEN — required to dodge 429 rate limits on Voxtral download.