vh
14f052461e
New stack scaffolding for the Fish quantized-realtime experiment. Not deployed yet — this commit lands the canonical files; deploy follows. Architecture decisions made in Phase 1: * CUDA backend, NOT Vulkan. s2.cpp's CMakeLists exposes both -DS2_VULKAN and -DS2_CUDA; the most recent upstream commit (2026-04-12) was specifically about CUDA improvements, and CUDA on the A6000 will be substantially faster than Vulkan for ML matmul. -DS2_CUDA=ON in the Dockerfile build args. * Pinned to s2.cpp commit e48ce8e02d8335bd9a0ba94679f605724b31d12 (2026-04-12 HEAD of main). Repo is alpha software per README; pin tightly so future churn doesn't break our build. Bump deliberately when wanting upstream improvements. * Multi-stage Dockerfile: nvidia/cuda:12.6.0-devel for build (needs CMake + ninja + git + the CUDA toolchain) → nvidia/cuda:12.6.0-runtime for serve (slimmer; just the s2 binary + GGML libs + a small Python shim). Cuts image size by ~50% vs single-stage devel. * FastAPI shim (server.py) wraps s2.cpp CLI in Fish's `/v1/tts` contract so the same bench harness + clients work against fish-cpp with no changes. Per-request flow: decode optional reference WAV from base64 → write to temp → subprocess.run the s2 binary → stream resulting WAV back. Adds ~50-100ms per-request fork+exec overhead; negligible vs the multi-second generation cost. * `streaming: true` accepted in request body but IGNORED — s2.cpp writes a complete WAV before returning, so chunked output isn't available. Unlike fish-s2 (HF wrapper) where streaming drops TTFB to 26ms, fish-cpp's TTFB ≈ total wall time. Speed depends entirely on raw generation throughput. * q6_k as default quant — sweet spot per typical GGUF guidance: near-bf16 quality at ~5GB. Other variants (q4_k_m, q5_k_m, q8_0, f16) selectable via FISH_CPP_MODEL env. * Pinned to GPU 1 (A6000) by default to share with fish-s2 for direct A/B benching. q6_k weights ~5GB + runtime ~3GB ≈ 8GB — comfortable on either GPU. * Port 8199 (next free in the irv-ml1 TTS slate). Phase 2 (next) is the actual deploy + first build. Reserved 30-45 min for cold-cache build + weights pull.
109 lines
3.9 KiB
Markdown
109 lines
3.9 KiB
Markdown
# fish-cpp
|
||
|
||
Fish s2-pro served via [s2.cpp](https://github.com/rodrigomatta/s2.cpp)
|
||
— a pure C++/GGML inference engine for Fish s2-pro, with weights from
|
||
[rodrigomt/s2-pro-gguf](https://huggingface.co/rodrigomt/s2-pro-gguf).
|
||
Wrapped by a tiny FastAPI shim exposing Fish's `/v1/tts` HTTP contract
|
||
so it slots into the same bench harness + client patterns as `fish-s2`.
|
||
|
||
## Why this stack alongside `fish-s2`
|
||
|
||
`fish-s2` (HF transformers wrapper) measured at **0.78× realtime** on
|
||
the A6000 — phenomenal quality but sub-realtime, meaning streaming
|
||
clients hit buffer underruns on phrases longer than ~3-4 seconds of
|
||
audio. `fish-cpp` targets the same s2-pro architecture but runs it
|
||
through s2.cpp's C++/GGML/CUDA inference path with q6_k quantization
|
||
— typically 2-5× faster than HF transformers for equivalent precision
|
||
(GGML is what makes llama.cpp fast).
|
||
|
||
**Goal**: hit ≥ 1× realtime on the A6000 so streaming actually flows
|
||
without stutters, while keeping Fish's quality near-equivalent to BF16.
|
||
|
||
## Status: alpha
|
||
|
||
s2.cpp is alpha software per its upstream README. Pin the SHA in
|
||
`.env`, don't track main blindly — community alpha projects break
|
||
weekly.
|
||
|
||
## What works (and what doesn't) vs `fish-s2`
|
||
|
||
| feature | fish-s2 (HF) | fish-cpp (this) |
|
||
|---|---|---|
|
||
| `/v1/tts` POST endpoint | ✓ | ✓ |
|
||
| `text` body field | ✓ | ✓ |
|
||
| `references` body field (cloning) | ✓ | ✓ (single ref only) |
|
||
| `streaming: true` | ✓ (TTFB → 26 ms) | ✗ accepted but ignored |
|
||
| Paralinguistic tags | ✓ | should work (same weights) |
|
||
| Quantization | bf16 | q4_k_m / q5_k_m / **q6_k** (default) / q8_0 |
|
||
| Realtime factor | 0.78× | targeting 1-1.5× |
|
||
|
||
The streaming gap matters: fish-s2 with `streaming: true` returns the
|
||
first audio chunk in 26 ms (perceived latency feels instant). fish-cpp
|
||
returns nothing until generation completes. So fish-cpp's appeal is
|
||
RAW THROUGHPUT, not perceived latency. Combined with realtime+
|
||
generation, total wall time stays low enough that polling clients
|
||
don't notice.
|
||
|
||
## API
|
||
|
||
OpenAPI shape mirrors `fish-s2`:
|
||
|
||
```bash
|
||
# Basic — text only, default voice
|
||
curl -fsS -X POST http://10.100.79.3:8199/v1/tts \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{"text":"Hello there."}' \
|
||
> out.wav
|
||
|
||
# With voice cloning — base64-encoded reference audio inline
|
||
B64=$(base64 -w 0 /worktank/fish-cpp/references/glados.wav) # on irv-ml1
|
||
echo "{\"text\":\"Welcome.\",\"references\":[{\"audio\":\"$B64\",\"text\":\"transcript\"}]}" \
|
||
| curl -fsS -X POST http://10.100.79.3:8199/v1/tts \
|
||
-H 'Content-Type: application/json' --data-binary @- \
|
||
> out.wav
|
||
```
|
||
|
||
Health probe at `GET /v1/health`.
|
||
|
||
## Deploy
|
||
|
||
```bash
|
||
scripts/elway irv-ml1 --playbook playbooks/deploy-fish-cpp.yaml
|
||
```
|
||
|
||
Cold deploy ~30-45 min: ~10 min image build (CUDA dev toolchain +
|
||
CMake + s2.cpp compile), ~3 min weights pull (~5 GB for q6_k +
|
||
12 MB tokenizer), ~1 min container boot.
|
||
|
||
## Hardware footprint
|
||
|
||
- **VRAM**: ~8 GB practical for q6_k (5 GB weights + 3 GB runtime).
|
||
Pinned to GPU 1 (A6000) by default to share with `fish-s2` for
|
||
direct A/B comparison. Could also run on GPU 0 (3090) with room
|
||
to spare.
|
||
- **Disk**: ~5 GB for q6_k checkpoint + tokenizer.
|
||
|
||
## Bench plan
|
||
|
||
Same 3-phrase suite as the other TTS:
|
||
|
||
```
|
||
P1 = "Hello, this is a test of the voice synthesis system. The quick brown fox jumps over the lazy dog."
|
||
P2 = "Oh my god, I cannot believe what just happened. That was absolutely incredible!"
|
||
P3 = "What the hell is going on. This is some bullshit and I am not putting up with it."
|
||
```
|
||
|
||
Compare:
|
||
- TTFB / total wall-clock per phrase
|
||
- audio-seconds / wall-seconds (realtime factor)
|
||
- Quality (ear test) vs `fish-s2` BF16 baseline
|
||
|
||
If fish-cpp lands ≥ 1× realtime AND the q6_k quality holds up under
|
||
ear test, this stack becomes the default Fish path. fish-s2 stays
|
||
deployed for paralinguistic tag fidelity reference + streaming
|
||
(if that turns out to matter for any specific use case).
|
||
|
||
## Lessons learned
|
||
|
||
(Populate after deploy iteration.)
|