Subprocess-per-request architecture forced CUDA + model load on every
/v1/tts call (~10-20s init, then 5-15s generation). Even though CUDA
is now actually being used (`-c 0` fix landed), 32s for "Verify."
proved per-request init was the bottleneck.
s2.cpp ships a built-in HTTP server (`--server -H -P`) that keeps the
model resident on the GPU. Refactor:
* entrypoint.sh — backgrounds `s2 --server -P 3030 -c 0 -m ... -t ...`,
waits for it to bind 3030, then foregrounds uvicorn. tini supervises
via `wait -n` so either child dying takes down the container.
* server.py — drops subprocess.run; instead httpx-POSTs Fish-shaped
/v1/tts JSON to s2's localhost:3030/generate (multipart form: text
+ optional prompt_text/prompt_audio for cloning). Model load + CUDA
init now happen once at container start, not per-request.
* Dockerfile — added httpx (shim dep), curl (entrypoint readiness
probe), and the entrypoint.sh COPY+chmod. CMD now invokes
entrypoint.sh instead of uvicorn directly.
* deploy-fish-cpp.yaml — uploads entrypoint.sh alongside server.py.
s2.cpp's README example uses `-v 0` which is `--vulkan 0` (Vulkan
device 0), easy to misread as "voice 0". The shim copied that
verbatim, so even after fixing the libcuda.so build problem AND the
libgomp.so runtime dep, every synthesis ran on CPU because the wrong
backend was selected.
Direct verification: `[Model] NPU not compiled, falling back to CPU`
in stderr; nvidia-smi showed no s2 process; bench timed out at 60s
on phrases that fish-s2 (HF, GPU) does in 7s.
s2.cpp's CLI:
-v <id> = --vulkan <device>
-c <id> = --cuda <device>
-M = --metal (Apple Silicon)
Switched the shim to `-c 0`. The CUDA backend IS in the build (-DS2_CUDA=ON
worked, libggml-cuda.so links fine per ldd, libcuda.so.1 mounts at
runtime via NVIDIA container runtime) — just wasn't being told to use it.
Build succeeded after the libcuda.so symlink fix, but the first
/v1/tts request returned HTTP 500 with:
s2 binary failed (rc=127): /usr/local/bin/s2: error while loading
shared libraries: libgomp.so.1: cannot open shared object file
CMake auto-enabled OpenMP during the build (gcc's -fopenmp flag), so
the s2 binary dynamically links libgomp.so.1. The build-stage devel
image had it; the slim cuda:runtime base doesn't ship it by default.
Adding libgomp1 to the runtime image's apt install resolves it.
Second attempt's CMAKE_LIBRARY_PATH + LIBRARY_PATH didn't get picked
up by ggml's nested CMake — same linker errors as the first run.
Robust fix: symlink the stub at /usr/local/cuda/lib64/stubs/libcuda.so
into /usr/local/lib (which ld searches unconditionally) and provide
both libcuda.so AND libcuda.so.1 (the SONAME ggml-cuda's
libggml-cuda.so links against). ldconfig refreshes the cache.
The symlinks live only in the build stage. The runtime image inherits
the real driver-provided libcuda.so.1 via NVIDIA's container runtime
mount, so the stubs never get used at execution time.
Two issues from the first deploy attempt:
1) Build failure (real): linker errors on s2.cpp's CUDA build —
undefined references to cuMemSetAccess, cuDeviceGet, etc. These
are CUDA Driver API symbols (in libcuda.so), not Runtime API
(libcudart.so). The driver lib is provided by NVIDIA's container
runtime at RUN time, not BUILD time.
Fix: nvidia/cuda:devel images ship a stubs library at
/usr/local/cuda/lib64/stubs/libcuda.so that provides the symbols
for linking but is non-runnable. Adding that path via
LIBRARY_PATH + CMAKE_LIBRARY_PATH lets the linker resolve while
leaving runtime unchanged (real libcuda.so comes from the
driver mount).
2) Verify false positive: the /v1/tts verify step's last command was
`rm -f "$out"` — which always exits 0. This made the shell's
final exit code 0 regardless of whether curl/file/grep succeeded,
so verify reported OK even when nothing was running on host_port.
Fix: `set -e` at top + trap-based cleanup. Failures now propagate;
the rm still runs on either path via EXIT trap.
New stack scaffolding for the Fish quantized-realtime experiment. Not
deployed yet — this commit lands the canonical files; deploy follows.
Architecture decisions made in Phase 1:
* CUDA backend, NOT Vulkan. s2.cpp's CMakeLists exposes both
-DS2_VULKAN and -DS2_CUDA; the most recent upstream commit
(2026-04-12) was specifically about CUDA improvements, and CUDA
on the A6000 will be substantially faster than Vulkan for ML
matmul. -DS2_CUDA=ON in the Dockerfile build args.
* Pinned to s2.cpp commit e48ce8e02d8335bd9a0ba94679f605724b31d12
(2026-04-12 HEAD of main). Repo is alpha software per README;
pin tightly so future churn doesn't break our build. Bump
deliberately when wanting upstream improvements.
* Multi-stage Dockerfile: nvidia/cuda:12.6.0-devel for build (needs
CMake + ninja + git + the CUDA toolchain) → nvidia/cuda:12.6.0-runtime
for serve (slimmer; just the s2 binary + GGML libs + a small Python
shim). Cuts image size by ~50% vs single-stage devel.
* FastAPI shim (server.py) wraps s2.cpp CLI in Fish's `/v1/tts`
contract so the same bench harness + clients work against fish-cpp
with no changes. Per-request flow: decode optional reference WAV
from base64 → write to temp → subprocess.run the s2 binary → stream
resulting WAV back. Adds ~50-100ms per-request fork+exec overhead;
negligible vs the multi-second generation cost.
* `streaming: true` accepted in request body but IGNORED — s2.cpp
writes a complete WAV before returning, so chunked output isn't
available. Unlike fish-s2 (HF wrapper) where streaming drops TTFB
to 26ms, fish-cpp's TTFB ≈ total wall time. Speed depends entirely
on raw generation throughput.
* q6_k as default quant — sweet spot per typical GGUF guidance:
near-bf16 quality at ~5GB. Other variants (q4_k_m, q5_k_m, q8_0,
f16) selectable via FISH_CPP_MODEL env.
* Pinned to GPU 1 (A6000) by default to share with fish-s2 for
direct A/B benching. q6_k weights ~5GB + runtime ~3GB ≈ 8GB —
comfortable on either GPU.
* Port 8199 (next free in the irv-ml1 TTS slate).
Phase 2 (next) is the actual deploy + first build. Reserved 30-45 min
for cold-cache build + weights pull.
vh