vh
14f052461e
New stack scaffolding for the Fish quantized-realtime experiment. Not deployed yet — this commit lands the canonical files; deploy follows. Architecture decisions made in Phase 1: * CUDA backend, NOT Vulkan. s2.cpp's CMakeLists exposes both -DS2_VULKAN and -DS2_CUDA; the most recent upstream commit (2026-04-12) was specifically about CUDA improvements, and CUDA on the A6000 will be substantially faster than Vulkan for ML matmul. -DS2_CUDA=ON in the Dockerfile build args. * Pinned to s2.cpp commit e48ce8e02d8335bd9a0ba94679f605724b31d12 (2026-04-12 HEAD of main). Repo is alpha software per README; pin tightly so future churn doesn't break our build. Bump deliberately when wanting upstream improvements. * Multi-stage Dockerfile: nvidia/cuda:12.6.0-devel for build (needs CMake + ninja + git + the CUDA toolchain) → nvidia/cuda:12.6.0-runtime for serve (slimmer; just the s2 binary + GGML libs + a small Python shim). Cuts image size by ~50% vs single-stage devel. * FastAPI shim (server.py) wraps s2.cpp CLI in Fish's `/v1/tts` contract so the same bench harness + clients work against fish-cpp with no changes. Per-request flow: decode optional reference WAV from base64 → write to temp → subprocess.run the s2 binary → stream resulting WAV back. Adds ~50-100ms per-request fork+exec overhead; negligible vs the multi-second generation cost. * `streaming: true` accepted in request body but IGNORED — s2.cpp writes a complete WAV before returning, so chunked output isn't available. Unlike fish-s2 (HF wrapper) where streaming drops TTFB to 26ms, fish-cpp's TTFB ≈ total wall time. Speed depends entirely on raw generation throughput. * q6_k as default quant — sweet spot per typical GGUF guidance: near-bf16 quality at ~5GB. Other variants (q4_k_m, q5_k_m, q8_0, f16) selectable via FISH_CPP_MODEL env. * Pinned to GPU 1 (A6000) by default to share with fish-s2 for direct A/B benching. q6_k weights ~5GB + runtime ~3GB ≈ 8GB — comfortable on either GPU. * Port 8199 (next free in the irv-ml1 TTS slate). Phase 2 (next) is the actual deploy + first build. Reserved 30-45 min for cold-cache build + weights pull.
43 lines
2.3 KiB
Bash
43 lines
2.3 KiB
Bash
# fish-cpp stack tunables. Copy to `.env` on irv-ml1 before deploying.
|
|
|
|
# ── build pin ────────────────────────────────────────────────────────
|
|
# SHA of rodrigomatta/s2.cpp to build from. The repo is alpha — pin a
|
|
# specific SHA so future upstream churn doesn't break our build. Update
|
|
# deliberately when you want upstream improvements.
|
|
FISH_CPP_S2_SHA=e48ce8e02d8335bd9a0ba94679f605724b31d123
|
|
|
|
# Local image tag — bump when you change build context.
|
|
FISH_CPP_TAG=v1
|
|
|
|
# ── network ──────────────────────────────────────────────────────────
|
|
# Host port. Container listens on 8000 internally.
|
|
# Reservations on irv-ml1: 8188 ComfyUI, 8190 CosyVoice, 8191 Qwen3-TTS,
|
|
# 8192 IndexTTS-2, 8193 Kokoro, 8194 VibeVoice, 8195 fish-s2 (HF),
|
|
# 8196 Chatterbox, 8197 Voxtral, 8765 Parakeet ASR. 8199 picked here.
|
|
FISH_CPP_PORT=8199
|
|
FISH_CPP_BIND=0.0.0.0
|
|
|
|
# ── runtime / GPU ────────────────────────────────────────────────────
|
|
# GPU pinning. "0" = RTX 3090 (24 GB), "1" = RTX A6000 (48 GB).
|
|
# Pinned to GPU 1 to share with fish-s2 (HF) for direct A/B benching;
|
|
# q6_k weights need ~5 GB + 3 GB runtime ≈ 8 GB. A6000 has plenty.
|
|
FISH_CPP_GPU_DEVICES=1
|
|
|
|
# Quantization variant to load. Available files in
|
|
# rodrigomt/s2-pro-gguf:
|
|
# s2-pro-q4_k_m.gguf — smaller / faster, quality drops noticeably
|
|
# s2-pro-q5_k_m.gguf — middle ground
|
|
# s2-pro-q6_k.gguf — sweet spot (default), near-bf16 quality
|
|
# s2-pro-q8_0.gguf — closest to bf16, larger / slower
|
|
# s2-pro-f16.gguf — full precision, no quantization win
|
|
FISH_CPP_MODEL=s2-pro-q6_k.gguf
|
|
|
|
# ── persistent storage on the host ───────────────────────────────────
|
|
# Weights — pre-pulled by the deploy playbook (~5 GB for q6_k +
|
|
# ~12 MB tokenizer.json).
|
|
FISH_CPP_WEIGHTS_DIR=/worktank/fish-cpp/weights
|
|
|
|
# Reference audio for voice cloning. Drop ~5-15 s WAV/MP3/FLAC clips
|
|
# here; reference them by basename in the API request body.
|
|
FISH_CPP_REFERENCE_DIR=/worktank/fish-cpp/references
|