voxtral: mount patched stage YAML to dodge hardcoded 0.8 GPU util; fish-s2: --half + streaming wins
Voxtral final fix (8th iteration): * The bundled voxtral_tts.yaml hardcodes gpu_memory_utilization: 0.8 on the language_model stage — overrides the CLI flag. Mounted a patched copy (0.4) at /etc/voxtral/voxtral_tts.yaml and pointed --stage-configs-path there. * With Kyutai stopped to free 5 GB on the 3090, both stages fit (target 9.4 + 2.4 GB ≈ 11.8 GB; 17 GB free post-kyutai-stop). * Voxtral now healthy on GPU 0 — bench: 1.9-2.7 s TTFB, real WAV. Fish s2-pro optimization (per-request sweep, no model swap): * `streaming: true` in request body drops TTFB from 7.7 s → 0.026 s (300×). Total time goes up ~1 s (chunked HTTP overhead) but perceived latency = TTFB. Use stream:true for any interactive use. * `latency: "balanced"` actually slower than default — bad name; skip. * `use_memory_cache: "on"` no measurable benefit. * `chunk_length: 100` (default 200) no TTFB benefit non-streaming. * Server-side `--half` (fp16 inference) added via compose `command` override — passes through start_server.sh's $@ unchanged into api_server.py. Should reduce total time too. Validation pending the post-restart bench. Kyutai stopped to free GPU 0 budget — the bench numbers earlier (3.4 s avg) were unimpressive vs Voxtral's 2.3 s in the same multilingual slot. Kept the stack files for future re-deploy if needed; just the running container is gone.
This commit is contained in:
@@ -80,6 +80,11 @@ services:
|
||||
- ${FISH_S2_REFERENCE_DIR}:/app/references
|
||||
# Persistent HF cache so model re-pull only happens on first deploy.
|
||||
- ${FISH_S2_CACHE_DIR}:/app/hf_cache
|
||||
# Pass --half to start_server.sh → enables fp16 inference on the
|
||||
# LLM half. Speeds up the autoregressive loop (the dominant cost
|
||||
# in TTFB). Also enables the streaming path's faster total time.
|
||||
# build_compile_args() echoes unknown args back to the exec line.
|
||||
command: ["--half"]
|
||||
healthcheck:
|
||||
# Fish ships /v1/health on the API server. python urllib instead
|
||||
# of curl because the upstream image is python-based and may not
|
||||
|
||||
Reference in New Issue
Block a user