vh
14f052461e
New stack scaffolding for the Fish quantized-realtime experiment. Not deployed yet — this commit lands the canonical files; deploy follows. Architecture decisions made in Phase 1: * CUDA backend, NOT Vulkan. s2.cpp's CMakeLists exposes both -DS2_VULKAN and -DS2_CUDA; the most recent upstream commit (2026-04-12) was specifically about CUDA improvements, and CUDA on the A6000 will be substantially faster than Vulkan for ML matmul. -DS2_CUDA=ON in the Dockerfile build args. * Pinned to s2.cpp commit e48ce8e02d8335bd9a0ba94679f605724b31d12 (2026-04-12 HEAD of main). Repo is alpha software per README; pin tightly so future churn doesn't break our build. Bump deliberately when wanting upstream improvements. * Multi-stage Dockerfile: nvidia/cuda:12.6.0-devel for build (needs CMake + ninja + git + the CUDA toolchain) → nvidia/cuda:12.6.0-runtime for serve (slimmer; just the s2 binary + GGML libs + a small Python shim). Cuts image size by ~50% vs single-stage devel. * FastAPI shim (server.py) wraps s2.cpp CLI in Fish's `/v1/tts` contract so the same bench harness + clients work against fish-cpp with no changes. Per-request flow: decode optional reference WAV from base64 → write to temp → subprocess.run the s2 binary → stream resulting WAV back. Adds ~50-100ms per-request fork+exec overhead; negligible vs the multi-second generation cost. * `streaming: true` accepted in request body but IGNORED — s2.cpp writes a complete WAV before returning, so chunked output isn't available. Unlike fish-s2 (HF wrapper) where streaming drops TTFB to 26ms, fish-cpp's TTFB ≈ total wall time. Speed depends entirely on raw generation throughput. * q6_k as default quant — sweet spot per typical GGUF guidance: near-bf16 quality at ~5GB. Other variants (q4_k_m, q5_k_m, q8_0, f16) selectable via FISH_CPP_MODEL env. * Pinned to GPU 1 (A6000) by default to share with fish-s2 for direct A/B benching. q6_k weights ~5GB + runtime ~3GB ≈ 8GB — comfortable on either GPU. * Port 8199 (next free in the irv-ml1 TTS slate). Phase 2 (next) is the actual deploy + first build. Reserved 30-45 min for cold-cache build + weights pull.
fish-cpp
Fish s2-pro served via s2.cpp
— a pure C++/GGML inference engine for Fish s2-pro, with weights from
rodrigomt/s2-pro-gguf.
Wrapped by a tiny FastAPI shim exposing Fish's /v1/tts HTTP contract
so it slots into the same bench harness + client patterns as fish-s2.
Why this stack alongside fish-s2
fish-s2 (HF transformers wrapper) measured at 0.78× realtime on
the A6000 — phenomenal quality but sub-realtime, meaning streaming
clients hit buffer underruns on phrases longer than ~3-4 seconds of
audio. fish-cpp targets the same s2-pro architecture but runs it
through s2.cpp's C++/GGML/CUDA inference path with q6_k quantization
— typically 2-5× faster than HF transformers for equivalent precision
(GGML is what makes llama.cpp fast).
Goal: hit ≥ 1× realtime on the A6000 so streaming actually flows without stutters, while keeping Fish's quality near-equivalent to BF16.
Status: alpha
s2.cpp is alpha software per its upstream README. Pin the SHA in
.env, don't track main blindly — community alpha projects break
weekly.
What works (and what doesn't) vs fish-s2
| feature | fish-s2 (HF) | fish-cpp (this) |
|---|---|---|
/v1/tts POST endpoint |
✓ | ✓ |
text body field |
✓ | ✓ |
references body field (cloning) |
✓ | ✓ (single ref only) |
streaming: true |
✓ (TTFB → 26 ms) | ✗ accepted but ignored |
| Paralinguistic tags | ✓ | should work (same weights) |
| Quantization | bf16 | q4_k_m / q5_k_m / q6_k (default) / q8_0 |
| Realtime factor | 0.78× | targeting 1-1.5× |
The streaming gap matters: fish-s2 with streaming: true returns the
first audio chunk in 26 ms (perceived latency feels instant). fish-cpp
returns nothing until generation completes. So fish-cpp's appeal is
RAW THROUGHPUT, not perceived latency. Combined with realtime+
generation, total wall time stays low enough that polling clients
don't notice.
API
OpenAPI shape mirrors fish-s2:
# Basic — text only, default voice
curl -fsS -X POST http://10.100.79.3:8199/v1/tts \
-H 'Content-Type: application/json' \
-d '{"text":"Hello there."}' \
> out.wav
# With voice cloning — base64-encoded reference audio inline
B64=$(base64 -w 0 /worktank/fish-cpp/references/glados.wav) # on irv-ml1
echo "{\"text\":\"Welcome.\",\"references\":[{\"audio\":\"$B64\",\"text\":\"transcript\"}]}" \
| curl -fsS -X POST http://10.100.79.3:8199/v1/tts \
-H 'Content-Type: application/json' --data-binary @- \
> out.wav
Health probe at GET /v1/health.
Deploy
scripts/elway irv-ml1 --playbook playbooks/deploy-fish-cpp.yaml
Cold deploy ~30-45 min: ~10 min image build (CUDA dev toolchain + CMake + s2.cpp compile), ~3 min weights pull (~5 GB for q6_k + 12 MB tokenizer), ~1 min container boot.
Hardware footprint
- VRAM: ~8 GB practical for q6_k (5 GB weights + 3 GB runtime).
Pinned to GPU 1 (A6000) by default to share with
fish-s2for direct A/B comparison. Could also run on GPU 0 (3090) with room to spare. - Disk: ~5 GB for q6_k checkpoint + tokenizer.
Bench plan
Same 3-phrase suite as the other TTS:
P1 = "Hello, this is a test of the voice synthesis system. The quick brown fox jumps over the lazy dog."
P2 = "Oh my god, I cannot believe what just happened. That was absolutely incredible!"
P3 = "What the hell is going on. This is some bullshit and I am not putting up with it."
Compare:
- TTFB / total wall-clock per phrase
- audio-seconds / wall-seconds (realtime factor)
- Quality (ear test) vs
fish-s2BF16 baseline
If fish-cpp lands ≥ 1× realtime AND the q6_k quality holds up under ear test, this stack becomes the default Fish path. fish-s2 stays deployed for paralinguistic tag fidelity reference + streaming (if that turns out to matter for any specific use case).
Lessons learned
(Populate after deploy iteration.)