From 0304464b7dbe29b292f7de5c2fddf2b13cbc7b99 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 28 Apr 2026 00:05:28 -0700 Subject: [PATCH] voxtral: HF_TOKEN passthrough + Mistral-format flags (vLLM rejects HF format) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third voxtral attempt: image pulled clean (3 min, v0.18.0), entrypoint parsed correctly, vLLM started, but engine init failed two ways: 1. HF rate-limited the irv-ml1 IP (38.120.94.3) during the metadata fetch — 429 Too Many Requests from too many large unauthenticated pulls today (heretic, 27b, fish-s2, fish-s1-mini, voxtral). Added HF_TOKEN env passthrough; user generates a token at https://huggingface.co/settings/tokens and sets VOXTRAL_HF_TOKEN in .env. 2. Voxtral uses Mistral's native model format (params.json + tekken.json tokenizer + consolidated.safetensors single file), NOT HF transformers format (config.json + tokenizer.json + sharded .safetensors). vLLM errored with "ensure presence of params.json for Mistral models." Fix: pass --load-format=mistral --tokenizer-mode=mistral --config-format=mistral to vllm serve. Confirmed by inspecting the Voxtral-4B-TTS-2603 HF tree: 25 files, ships params.json + tekken.json + consolidated.safetensors. Both fixes baked into compose. User needs to drop their HF_TOKEN into .env once and recreate. Side note discovered while debugging: fish-s2 s1-mini variant uses the tiktoken tokenizer format; the wrapper can't load it (errors with "NoneType has no attribute encode" on warmup). So s1-mini isn't a drop-in optimization for s2-pro — different code path needed. Fish back on s2-pro for now. --- stacks/voxtral/.env.example | 8 ++++++++ stacks/voxtral/compose.yaml | 11 +++++++++++ 2 files changed, 19 insertions(+) diff --git a/stacks/voxtral/.env.example b/stacks/voxtral/.env.example index bef2e6f..7850f9f 100644 --- a/stacks/voxtral/.env.example +++ b/stacks/voxtral/.env.example @@ -30,6 +30,14 @@ VOXTRAL_GPU_DEVICES=1 # alongside other GPU workloads on the same device. VOXTRAL_GPU_UTIL=0.85 +# ── HuggingFace auth ───────────────────────────────────────────────── +# HF_TOKEN — required to dodge 429 rate limits on Voxtral download. +# HF aggressively throttles unauthenticated IPs that pull large repos. +# Generate one at https://huggingface.co/settings/tokens (a read-only +# token is sufficient). Without this, the first model download fails +# mid-stream and vLLM aborts engine init. +VOXTRAL_HF_TOKEN= + # ── persistent storage on the host ─────────────────────────────────── # HF cache — first start pulls the Voxtral checkpoint (~8 GB) into # this dir. Persistent across container recreates. diff --git a/stacks/voxtral/compose.yaml b/stacks/voxtral/compose.yaml index 591a1c0..3d23024 100644 --- a/stacks/voxtral/compose.yaml +++ b/stacks/voxtral/compose.yaml @@ -28,6 +28,10 @@ services: environment: - NVIDIA_VISIBLE_DEVICES=${VOXTRAL_GPU_DEVICES:-1} - HF_HOME=/root/.cache/huggingface + # HF_TOKEN required to dodge 429 rate limits on the model + # download (HF aggressively throttles unauthenticated IPs that + # pull large repos repeatedly). Set in .env — see .env.example. + - HF_TOKEN=${VOXTRAL_HF_TOKEN} # vLLM serving args — see https://docs.vllm.ai for full list. # We override the default model via cmd args below. volumes: @@ -43,6 +47,13 @@ services: - "--port=8000" - "--dtype=bfloat16" - "--gpu-memory-utilization=${VOXTRAL_GPU_UTIL:-0.85}" + # Voxtral uses the native Mistral model format (params.json, + # tekken.json tokenizer, consolidated.safetensors) — NOT HF + # transformers format. vLLM rejects it with "ensure presence + # of params.json" unless these three flags are set. + - "--load-format=mistral" + - "--tokenizer-mode=mistral" + - "--config-format=mistral" healthcheck: # vLLM-Omni exposes /health for liveness + /v1/models for readiness. # /health 200 means the server's listening; /v1/models 200 means