vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

voxtral: HF_TOKEN passthrough + Mistral-format flags (vLLM rejects HF format)

Browse Source 
Third voxtral attempt: image pulled clean (3 min, v0.18.0), entrypoint
parsed correctly, vLLM started, but engine init failed two ways:

1. HF rate-limited the irv-ml1 IP (38.120.94.3) during the metadata
   fetch — 429 Too Many Requests from too many large unauthenticated
   pulls today (heretic, 27b, fish-s2, fish-s1-mini, voxtral). Added
   HF_TOKEN env passthrough; user generates a token at
   https://huggingface.co/settings/tokens and sets VOXTRAL_HF_TOKEN
   in .env.

2. Voxtral uses Mistral's native model format (params.json +
   tekken.json tokenizer + consolidated.safetensors single file),
   NOT HF transformers format (config.json + tokenizer.json + sharded
   .safetensors). vLLM errored with "ensure presence of params.json
   for Mistral models." Fix: pass --load-format=mistral
   --tokenizer-mode=mistral --config-format=mistral to vllm serve.
   Confirmed by inspecting the Voxtral-4B-TTS-2603 HF tree:
   25 files, ships params.json + tekken.json + consolidated.safetensors.

Both fixes baked into compose. User needs to drop their HF_TOKEN into
.env once and recreate.

Side note discovered while debugging: fish-s2 s1-mini variant uses
the tiktoken tokenizer format; the wrapper can't load it (errors with
"NoneType has no attribute encode" on warmup). So s1-mini isn't a
drop-in optimization for s2-pro — different code path needed. Fish
back on s2-pro for now.
This commit is contained in:
vh
2026-04-28 00:05:28 -07:00
parent 5a1b715f3d
commit 0304464b7d
2 changed files with 19 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/voxtral/.env.example
+8
View File
@@ -30,6 +30,14 @@ VOXTRAL_GPU_DEVICES=1
# alongside other GPU workloads on the same device.
VOXTRAL_GPU_UTIL=0.85
# ── HuggingFace auth ─────────────────────────────────────────────────
# HF_TOKEN — required to dodge 429 rate limits on Voxtral download.
# HF aggressively throttles unauthenticated IPs that pull large repos.
# Generate one at https://huggingface.co/settings/tokens (a read-only
# token is sufficient). Without this, the first model download fails
# mid-stream and vLLM aborts engine init.
VOXTRAL_HF_TOKEN=
# ── persistent storage on the host ───────────────────────────────────
# HF cache — first start pulls the Voxtral checkpoint (~8 GB) into
# this dir. Persistent across container recreates.