voxtral: mount patched stage YAML to dodge hardcoded 0.8 GPU util; fish-s2: --half + streaming wins

Voxtral final fix (8th iteration):
* The bundled voxtral_tts.yaml hardcodes gpu_memory_utilization: 0.8
  on the language_model stage — overrides the CLI flag. Mounted a
  patched copy (0.4) at /etc/voxtral/voxtral_tts.yaml and pointed
  --stage-configs-path there.
* With Kyutai stopped to free 5 GB on the 3090, both stages fit
  (target 9.4 + 2.4 GB ≈ 11.8 GB; 17 GB free post-kyutai-stop).
* Voxtral now healthy on GPU 0 — bench: 1.9-2.7 s TTFB, real WAV.

Fish s2-pro optimization (per-request sweep, no model swap):
* `streaming: true` in request body drops TTFB from 7.7 s → 0.026 s
  (300×). Total time goes up ~1 s (chunked HTTP overhead) but
  perceived latency = TTFB. Use stream:true for any interactive use.
* `latency: "balanced"` actually slower than default — bad name; skip.
* `use_memory_cache: "on"` no measurable benefit.
* `chunk_length: 100` (default 200) no TTFB benefit non-streaming.
* Server-side `--half` (fp16 inference) added via compose `command`
  override — passes through start_server.sh's $@ unchanged into
  api_server.py. Should reduce total time too. Validation pending
  the post-restart bench.

Kyutai stopped to free GPU 0 budget — the bench numbers earlier
(3.4 s avg) were unimpressive vs Voxtral's 2.3 s in the same
multilingual slot. Kept the stack files for future re-deploy if
needed; just the running container is gone.
This commit is contained in:
vh
2026-04-28 00:35:55 -07:00
parent fe01f73d84
commit 68f3cd05fe
2 changed files with 12 additions and 1 deletions
+5
View File
@@ -80,6 +80,11 @@ services:
- ${FISH_S2_REFERENCE_DIR}:/app/references
# Persistent HF cache so model re-pull only happens on first deploy.
- ${FISH_S2_CACHE_DIR}:/app/hf_cache
# Pass --half to start_server.sh → enables fp16 inference on the
# LLM half. Speeds up the autoregressive loop (the dominant cost
# in TTFB). Also enables the streaming path's faster total time.
# build_compile_args() echoes unknown args back to the exec line.
command: ["--half"]
healthcheck:
# Fish ships /v1/health on the API server. python urllib instead
# of curl because the upstream image is python-based and may not