vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fish-cpp: delete the stack — s2.cpp is too alpha to use today

Browse Source 
Three deploy iterations + four backend attempts (subprocess CUDA,
resident-server CUDA, Vulkan rebuild) all failed to deliver speedup
over fish-s2:

* CUDA path: ggml_cuda_init succeeded, weights loaded onto GPU per
  s2's logs, but nvidia-smi showed 0% utilization during synthesis.
  Wall time 20s/long phrase vs fish-s2's 7.5s. The "CUDA get_rows
  unsupported for type q6_K" warning hints at incomplete op coverage
  in s2.cpp's alpha CUDA backend for fish-speech architecture.

* Vulkan path: vk::IncompatibleDriverError on container init. NVIDIA
  Vulkan ICD not accessible inside the container despite
  NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics. Would need
  host-side nvidia-utils-vulkan installation or manual ICD bind
  mount. Didn't pursue.

Both are fixable — CUDA needs op coverage upstream (author actively
working on it; "selective embedding dequant" commit landed 16 days
ago), Vulkan needs host-side ICD setup. Neither is a config-flip,
both are real work for marginal-or-zero return. Better to delete the
stack and revisit when s2.cpp matures or when we tackle FP8
quantization on ana-ml2's RTX 6000 Ada (sm_89, native FP8 hardware).

Local image rmi'd, /opt/docker/compose/fish-cpp removed on irv-ml1.
/worktank/fish-cpp left for user-side sudo cleanup.

Future Fish acceleration paths (in order of decreasing certainty):
1. Wait for s2.cpp CUDA op coverage to mature (track upstream commits).
2. Quantize Fish BF16 → FP8 via TransformerEngine, deploy on
   ana-ml2's RTX 6000 Ada (Ada has native FP8 tensor cores, A6000
   doesn't). ~2x speedup if it works.
3. vLLM port of Fish (no upstream support today).
This commit is contained in:
Vuong Hoang
2026-04-28 01:52:57 -07:00
parent 67813bbef4
commit 0ba41e02ea
7 changed files with 0 additions and 624 deletions
Download Patch File Download Diff File Expand all files Collapse all files
playbooks/deploy-fish-cpp.yaml
-142
View File
@@ -1,142 +0,0 @@
# Deploy fish-cpp (Fish s2-pro via s2.cpp + GGML CUDA inference) to irv-ml1.
#
# Builds the image locally — multi-stage CUDA devel base (CMake + s2.cpp
# compile, ~10 min cold) → CUDA runtime base + binary + python shim.
# Pre-pulls rodrigomt/s2-pro-gguf weights (q6_k default, ~5 GB) into
# the bind-mounted weights dir.
#
# Usage:
# scripts/elway irv-ml1 --playbook playbooks/deploy-fish-cpp.yaml
#
# Idempotent — every step is creates-/when-gated; rerun is safe.
vars:
compose_dir: /opt/docker/compose/fish-cpp
references_dir: /worktank/fish-cpp/references
weights_dir: /worktank/fish-cpp/weights
host_port: "8199"
weights_repo: rodrigomt/s2-pro-gguf
default_quant: s2-pro-q6_k.gguf
steps:
# ── host-side dirs ──────────────────────────────────────────────────
- name: Ensure /worktank/fish-cpp root exists (one-time, sudo)
shell: mkdir -p /worktank/fish-cpp
sudo: true
creates: /worktank/fish-cpp
- name: Chown /worktank/fish-cpp to lkraven
shell: chown -R lkraven:lkraven /worktank/fish-cpp
sudo: true
when: "[ \"$(stat -c %U /worktank/fish-cpp)\" != \"lkraven\" ]"
- name: Ensure references dir exists
shell: mkdir -p {{ references_dir }}
creates: "{{ references_dir }}"
- name: Ensure weights dir exists
shell: mkdir -p {{ weights_dir }}
creates: "{{ weights_dir }}"
- name: Ensure compose dir exists
shell: mkdir -p {{ compose_dir }}
creates: "{{ compose_dir }}"
# ── deploy build context ────────────────────────────────────────────
# s2.cpp is built INSIDE the docker image, but the Dockerfile + shim
# need to be present in the compose dir so `docker compose build`
# can find them.
- name: Upload compose.yaml
upload:
src: stacks/fish-cpp/compose.yaml
dest: "{{ compose_dir }}/compose.yaml"
mode: "0644"
- name: Upload Dockerfile
upload:
src: stacks/fish-cpp/Dockerfile
dest: "{{ compose_dir }}/Dockerfile"
mode: "0644"
- name: Upload server.py (FastAPI shim)
upload:
src: stacks/fish-cpp/server.py
dest: "{{ compose_dir }}/server.py"
mode: "0644"
- name: Upload entrypoint.sh (starts s2 server + uvicorn shim)
upload:
src: stacks/fish-cpp/entrypoint.sh
dest: "{{ compose_dir }}/entrypoint.sh"
mode: "0755"
- name: Seed .env from template (only if absent)
upload:
src: stacks/fish-cpp/.env.example
dest: "{{ compose_dir }}/.env"
mode: "0644"
when: "[ ! -f {{ compose_dir }}/.env ]"
# ── pre-pull weights ────────────────────────────────────────────────
# q6_k + tokenizer.json (~5 GB total). Same one-shot
# python:3.12-slim + huggingface_hub.snapshot_download + hf_transfer
# pattern we've used for fish-s2, voxtral, etc. Idempotent on rerun
# via `creates:` on the model file.
- name: Pre-pull rodrigomt/s2-pro-gguf weights (q6_k + tokenizer, ~5 GB)
shell: |
docker run --rm --user 1000:1000 \
-e HOME=/tmp/h -e HF_HUB_ENABLE_HF_TRANSFER=1 \
-v {{ weights_dir }}:/dest \
python:3.12-slim sh -c 'set -e; mkdir -p /tmp/h /tmp/pip /tmp/site; PIP_CACHE_DIR=/tmp/pip pip install --quiet --target /tmp/site huggingface_hub hf_transfer; PYTHONPATH=/tmp/site python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id=\"{{ weights_repo }}\", local_dir=\"/dest\", allow_patterns=[\"{{ default_quant }}\",\"tokenizer.json\"])"'
creates: "{{ weights_dir }}/{{ default_quant }}"
# ── build + bring up ────────────────────────────────────────────────
- name: docker compose build (~10 min first time; CUDA toolchain + s2.cpp compile)
shell: |
set -o pipefail
cd {{ compose_dir }} && docker compose build 2>&1 \
| grep -vE '^#[0-9]+ |^ => |^=> |Collecting|Downloading|Requirement|Using cached|Installing collected|Successfully (installed|built)|━'
- name: docker compose up -d
shell: cd {{ compose_dir }} && docker compose up -d
- name: Wait for /v1/health to respond
shell: |
for i in $(seq 1 60); do
curl -sf -o /dev/null --max-time 3 http://localhost:{{ host_port }}/v1/health && exit 0
sleep 5
done
exit 1
changed_when: "false"
verify:
- name: /v1/health returns 200 + reports model loaded
shell: |
curl -sf http://localhost:{{ host_port }}/v1/health \
| python3 -c "import json,sys; d=json.load(sys.stdin); assert d.get('status')=='ok' and d.get('model')"
changed_when: "false"
- name: /v1/tts returns a real WAV (POST with text body)
# `set -e` so curl/file/grep failures actually propagate. The
# previous version put `rm -f` as the last command, which always
# exits 0 — masking real failures (verify reported OK even when
# nothing was running on host_port). Trap-based cleanup runs the
# rm even on failure.
shell: |
set -e
out=$(mktemp --suffix=.wav)
trap 'rm -f "$out"' EXIT
curl -sf -X POST http://localhost:{{ host_port }}/v1/tts \
-H 'Content-Type: application/json' \
-d '{"text":"Verify."}' \
-o "$out" --max-time 60
file -b "$out" | grep -q '^RIFF.*WAVE'
changed_when: "false"
- name: Container is running
shell: docker inspect fish-cpp --format '{{.State.Status}}' | grep -q running
changed_when: "false"
stacks/fish-cpp/.env.example
-42
View File
@@ -1,42 +0,0 @@
# fish-cpp stack tunables. Copy to `.env` on irv-ml1 before deploying.
# ── build pin ────────────────────────────────────────────────────────
# SHA of rodrigomatta/s2.cpp to build from. The repo is alpha — pin a
# specific SHA so future upstream churn doesn't break our build. Update
# deliberately when you want upstream improvements.
FISH_CPP_S2_SHA=e48ce8e02d8335bd9a0ba94679f605724b31d123
# Local image tag — bump when you change build context.
FISH_CPP_TAG=v1
# ── network ──────────────────────────────────────────────────────────
# Host port. Container listens on 8000 internally.
# Reservations on irv-ml1: 8188 ComfyUI, 8190 CosyVoice, 8191 Qwen3-TTS,
# 8192 IndexTTS-2, 8193 Kokoro, 8194 VibeVoice, 8195 fish-s2 (HF),
# 8196 Chatterbox, 8197 Voxtral, 8765 Parakeet ASR. 8199 picked here.
FISH_CPP_PORT=8199
FISH_CPP_BIND=0.0.0.0
# ── runtime / GPU ────────────────────────────────────────────────────
# GPU pinning. "0" = RTX 3090 (24 GB), "1" = RTX A6000 (48 GB).
# Pinned to GPU 1 to share with fish-s2 (HF) for direct A/B benching;
# q6_k weights need ~5 GB + 3 GB runtime ≈ 8 GB. A6000 has plenty.
FISH_CPP_GPU_DEVICES=1
# Quantization variant to load. Available files in
# rodrigomt/s2-pro-gguf:
# s2-pro-q4_k_m.gguf — smaller / faster, quality drops noticeably
# s2-pro-q5_k_m.gguf — middle ground
# s2-pro-q6_k.gguf — sweet spot (default), near-bf16 quality
# s2-pro-q8_0.gguf — closest to bf16, larger / slower
# s2-pro-f16.gguf — full precision, no quantization win
FISH_CPP_MODEL=s2-pro-q6_k.gguf
# ── persistent storage on the host ───────────────────────────────────
# Weights — pre-pulled by the deploy playbook (~5 GB for q6_k +
# ~12 MB tokenizer.json).
FISH_CPP_WEIGHTS_DIR=/worktank/fish-cpp/weights
# Reference audio for voice cloning. Drop ~5-15 s WAV/MP3/FLAC clips
# here; reference them by basename in the API request body.
FISH_CPP_REFERENCE_DIR=/worktank/fish-cpp/references
stacks/fish-cpp/Dockerfile
-100
View File
@@ -1,100 +0,0 @@
# fish-cpp — s2.cpp (pure C++/GGML inference for Fish s2-pro GGUFs) +
# tiny FastAPI shim exposing Fish's /v1/tts contract.
#
# Two-stage build:
# 1. builder — compiles s2.cpp with CUDA backend
# 2. runtime — slim image with the s2 binary + Python shim
#
# The s2.cpp binary is the actual inference engine; the Python shim is
# just an HTTP-to-CLI bridge so this stack drops into the same fleet
# pattern as the other TTS (POST /v1/tts, Fish-shaped request body).
# ── Stage 1: build s2.cpp with CUDA ────────────────────────────────────
FROM nvidia/cuda:12.6.0-devel-ubuntu24.04 AS builder
ARG S2_CPP_SHA=e48ce8e02d8335bd9a0ba94679f605724b31d123
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
git ca-certificates cmake ninja-build build-essential pkg-config \
libvulkan-dev glslc \
&& rm -rf /var/lib/apt/lists/*
# libvulkan-dev = Vulkan headers + loader for build-time linking.
# glslc = GLSL→SPIR-V compiler; GGML's Vulkan backend compiles its
# shaders with this at build time. Without it, the cmake configure
# step disables Vulkan silently.
WORKDIR /src
RUN git clone --recurse-submodules https://github.com/rodrigomatta/s2.cpp.git \
&& cd s2.cpp \
&& git checkout ${S2_CPP_SHA} \
&& git submodule update --init --recursive
WORKDIR /src/s2.cpp
# CUDA Driver API symbols (cuMemSetAccess, cuDeviceGet, etc.) live in
# libcuda.so which the NVIDIA driver provides at RUNTIME. Build-time
# uses the stubs library at /usr/local/cuda/lib64/stubs/. CMake's
# find_library doesn't reliably pick it up via CMAKE_LIBRARY_PATH for
# nested ggml-cuda builds — the most robust fix is symlinking the stub
# into /usr/local/lib (which ld checks unconditionally) AND providing
# libcuda.so.1 (the SONAME ggml-cuda links against). The symlinks live
# only in this build stage; the runtime image gets the real driver-
# provided libcuda.so.1 via NVIDIA's container runtime.
RUN ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/lib/libcuda.so \
&& ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/lib/libcuda.so.1 \
&& ldconfig
# Build with BOTH CUDA and Vulkan backends — runtime selects via the
# entrypoint (-c <id> for CUDA, -v <id> for Vulkan). We've established
# that s2.cpp's CUDA path silently falls back to CPU on this model
# (alpha software, incomplete op coverage); Vulkan is the README's
# canonical example so it likely has more complete op coverage on
# fish-speech. Keep both built so we can flip without rebuilding.
RUN cmake -G Ninja -B build -DCMAKE_BUILD_TYPE=Release \
-DS2_CUDA=ON -DS2_VULKAN=ON \
&& cmake --build build --parallel $(nproc) --target s2
# ── Stage 2: runtime — slim image with the binary + python shim ────────
FROM nvidia/cuda:12.6.0-runtime-ubuntu24.04
ENV DEBIAN_FRONTEND=noninteractive \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1
RUN apt-get update && apt-get install -y --no-install-recommends \
python3 python3-pip python3-venv tini \
libgomp1 curl \
libvulkan1 \
&& rm -rf /var/lib/apt/lists/*
# libgomp1 = GNU OpenMP runtime — required by the s2 binary.
# curl = entrypoint uses it to wait for s2 server to bind :3030.
# libvulkan1 = Vulkan loader. The NVIDIA Vulkan ICD itself comes from
# the host driver via NVIDIA container runtime when
# NVIDIA_DRIVER_CAPABILITIES includes "graphics".
# Pull the shim deps into an isolated venv so we don't fight system pip.
RUN python3 -m venv /opt/venv
ENV PATH="/opt/venv/bin:${PATH}"
RUN pip install --no-cache-dir 'fastapi>=0.115' 'uvicorn[standard]>=0.30' 'pydantic>=2' 'httpx>=0.27'
# Copy the s2 binary + GGML runtime libs from the builder stage.
COPY --from=builder /src/s2.cpp/build/s2 /usr/local/bin/s2
COPY --from=builder /src/s2.cpp/build/ggml/src/libggml*.so /usr/local/lib/
COPY --from=builder /src/s2.cpp/build/ggml/src/ggml-cuda/libggml-cuda.so /usr/local/lib/
COPY --from=builder /src/s2.cpp/build/ggml/src/ggml-vulkan/libggml-vulkan.so /usr/local/lib/
RUN ldconfig
WORKDIR /app
COPY server.py /app/server.py
COPY entrypoint.sh /app/entrypoint.sh
RUN chmod +x /app/entrypoint.sh
# Bind-mounted at runtime: weights at /weights, references at /references.
VOLUME /weights
VOLUME /references
EXPOSE 8000
# tini supervises the entrypoint script which manages both s2 server
# (bound to localhost:3030, model resident) and the uvicorn shim
# (bound to 0.0.0.0:8000, proxies /v1/tts → s2's /generate).
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["/app/entrypoint.sh"]
stacks/fish-cpp/README.md
-108
View File
@@ -1,108 +0,0 @@
# fish-cpp
Fish s2-pro served via [s2.cpp](https://github.com/rodrigomatta/s2.cpp)
— a pure C++/GGML inference engine for Fish s2-pro, with weights from
[rodrigomt/s2-pro-gguf](https://huggingface.co/rodrigomt/s2-pro-gguf).
Wrapped by a tiny FastAPI shim exposing Fish's `/v1/tts` HTTP contract
so it slots into the same bench harness + client patterns as `fish-s2`.
## Why this stack alongside `fish-s2`
`fish-s2` (HF transformers wrapper) measured at **0.78× realtime** on
the A6000 — phenomenal quality but sub-realtime, meaning streaming
clients hit buffer underruns on phrases longer than ~3-4 seconds of
audio. `fish-cpp` targets the same s2-pro architecture but runs it
through s2.cpp's C++/GGML/CUDA inference path with q6_k quantization
— typically 2-5× faster than HF transformers for equivalent precision
(GGML is what makes llama.cpp fast).
**Goal**: hit ≥ 1× realtime on the A6000 so streaming actually flows
without stutters, while keeping Fish's quality near-equivalent to BF16.
## Status: alpha
s2.cpp is alpha software per its upstream README. Pin the SHA in
`.env`, don't track main blindly — community alpha projects break
weekly.
## What works (and what doesn't) vs `fish-s2`
| feature | fish-s2 (HF) | fish-cpp (this) |
|---|---|---|
| `/v1/tts` POST endpoint | ✓ | ✓ |
| `text` body field | ✓ | ✓ |
| `references` body field (cloning) | ✓ | ✓ (single ref only) |
| `streaming: true` | ✓ (TTFB → 26 ms) | ✗ accepted but ignored |
| Paralinguistic tags | ✓ | should work (same weights) |
| Quantization | bf16 | q4_k_m / q5_k_m / **q6_k** (default) / q8_0 |
| Realtime factor | 0.78× | targeting 1-1.5× |
The streaming gap matters: fish-s2 with `streaming: true` returns the
first audio chunk in 26 ms (perceived latency feels instant). fish-cpp
returns nothing until generation completes. So fish-cpp's appeal is
RAW THROUGHPUT, not perceived latency. Combined with realtime+
generation, total wall time stays low enough that polling clients
don't notice.
## API
OpenAPI shape mirrors `fish-s2`:
```bash
# Basic — text only, default voice
curl -fsS -X POST http://10.100.79.3:8199/v1/tts \
-H 'Content-Type: application/json' \
-d '{"text":"Hello there."}' \
> out.wav
# With voice cloning — base64-encoded reference audio inline
B64=$(base64 -w 0 /worktank/fish-cpp/references/glados.wav) # on irv-ml1
echo "{\"text\":\"Welcome.\",\"references\":[{\"audio\":\"$B64\",\"text\":\"transcript\"}]}" \
| curl -fsS -X POST http://10.100.79.3:8199/v1/tts \
-H 'Content-Type: application/json' --data-binary @- \
> out.wav
```
Health probe at `GET /v1/health`.
## Deploy
```bash
scripts/elway irv-ml1 --playbook playbooks/deploy-fish-cpp.yaml
```
Cold deploy ~30-45 min: ~10 min image build (CUDA dev toolchain +
CMake + s2.cpp compile), ~3 min weights pull (~5 GB for q6_k +
12 MB tokenizer), ~1 min container boot.
## Hardware footprint
- **VRAM**: ~8 GB practical for q6_k (5 GB weights + 3 GB runtime).
Pinned to GPU 1 (A6000) by default to share with `fish-s2` for
direct A/B comparison. Could also run on GPU 0 (3090) with room
to spare.
- **Disk**: ~5 GB for q6_k checkpoint + tokenizer.
## Bench plan
Same 3-phrase suite as the other TTS:
```
P1 = "Hello, this is a test of the voice synthesis system. The quick brown fox jumps over the lazy dog."
P2 = "Oh my god, I cannot believe what just happened. That was absolutely incredible!"
P3 = "What the hell is going on. This is some bullshit and I am not putting up with it."
```
Compare:
- TTFB / total wall-clock per phrase
- audio-seconds / wall-seconds (realtime factor)
- Quality (ear test) vs `fish-s2` BF16 baseline
If fish-cpp lands ≥ 1× realtime AND the q6_k quality holds up under
ear test, this stack becomes the default Fish path. fish-s2 stays
deployed for paralinguistic tag fidelity reference + streaming
(if that turns out to matter for any specific use case).
## Lessons learned
(Populate after deploy iteration.)
stacks/fish-cpp/compose.yaml
-64
View File
@@ -1,64 +0,0 @@
# fish-cpp — Fish s2-pro served via s2.cpp (pure C++/GGML inference,
# CUDA backend) with a tiny FastAPI shim exposing Fish's /v1/tts
# contract. Built locally from rodrigomatta/s2.cpp + a pinned SHA.
#
# Why this stack alongside fish-s2:
# * fish-s2 (HF transformers wrapper): ~7-8 s TTFB, 0.78× realtime.
# Phenomenal quality but sub-realtime, buffer-underruns on long
# phrases even with streaming.
# * fish-cpp (this): targets ~2-5× speedup from the GGML inference
# path + q6_k quantization. Goal: hit realtime for streaming use.
#
# CAVEATS:
# * s2.cpp is alpha software (per upstream README). Pin the SHA;
# don't track main blindly.
# * Per-request subprocess spawn — each /v1/tts call forks the s2
# binary. Adds ~50-100 ms over a long-running daemon. Negligible
# vs the multi-second generation cost.
# * No streaming — s2.cpp writes a complete WAV before returning,
# so the shim's `streaming: true` field is accepted but ignored.
# TTFB ≈ total wall time (fast generation is the only path to
# low-latency here, not chunked output).
services:
fish-cpp:
image: local/fish-cpp:${FISH_CPP_TAG}
build:
context: .
dockerfile: Dockerfile
args:
S2_CPP_SHA: ${FISH_CPP_S2_SHA:-e48ce8e02d8335bd9a0ba94679f605724b31d123}
container_name: fish-cpp
restart: unless-stopped
runtime: nvidia
ports:
- "${FISH_CPP_BIND:-0.0.0.0}:${FISH_CPP_PORT}:8000"
environment:
- NVIDIA_VISIBLE_DEVICES=${FISH_CPP_GPU_DEVICES:-1}
# Default nvidia-container-toolkit only mounts compute libs.
# Vulkan needs the graphics ICD too (libGLX_nvidia, vulkan ICD
# JSON). Without this, Vulkan init in the container fails with
# "no Vulkan ICD" even though the GPU is present.
- NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics
- FISH_CPP_MODEL=${FISH_CPP_MODEL:-s2-pro-q6_k.gguf}
- FISH_CPP_TOKENIZER=tokenizer.json
- FISH_CPP_DEVICE=0
# Backend selection for the s2 server in entrypoint.sh:
# cuda — -c <id>; alpha CUDA path, GPU 0% util on fish-speech (broken)
# vulkan — -v <id>; README's canonical example, more battle-tested
- FISH_CPP_BACKEND=${FISH_CPP_BACKEND:-vulkan}
volumes:
- ${FISH_CPP_WEIGHTS_DIR}:/weights:ro
- ${FISH_CPP_REFERENCE_DIR}:/references:ro
healthcheck:
test: ["CMD-SHELL", "python3 -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/v1/health', timeout=5).status==200 else 1)\""]
interval: 30s
timeout: 10s
retries: 3
start_period: 60s
labels:
- homepage.group=AI Systems
- homepage.name=Fish (s2.cpp)
- homepage.icon=mdi-fish
- homepage.description=Fish s2-pro via s2.cpp/GGML — quantized for realtime (irv-ml1)
- homepage.href=http://10.100.79.3:${FISH_CPP_PORT}
stacks/fish-cpp/entrypoint.sh
-63
View File
@@ -1,63 +0,0 @@
#!/bin/bash
# fish-cpp entrypoint — start s2 server + uvicorn shim.
#
# s2 server holds the model resident on the GPU; uvicorn proxies
# Fish-shaped /v1/tts JSON requests to s2's multipart /generate.
# Both processes share the container; tini supervises both via
# `wait` after backgrounding s2.
set -euo pipefail
MODEL_PATH="${WEIGHTS_DIR:-/weights}/${FISH_CPP_MODEL:-s2-pro-q6_k.gguf}"
TOKENIZER_PATH="${WEIGHTS_DIR:-/weights}/${FISH_CPP_TOKENIZER:-tokenizer.json}"
DEVICE="${FISH_CPP_DEVICE:-0}"
BACKEND="${FISH_CPP_BACKEND:-vulkan}"
case "$BACKEND" in
cuda) BACKEND_FLAG=(-c "$DEVICE") ;;
vulkan) BACKEND_FLAG=(-v "$DEVICE") ;;
cpu) BACKEND_FLAG=() ;; # no flag → s2 stays on CPU
*) echo "[entrypoint] unknown FISH_CPP_BACKEND=$BACKEND (use cuda/vulkan/cpu)" >&2; exit 2 ;;
esac
if [ ! -f "$MODEL_PATH" ]; then echo "missing model: $MODEL_PATH" >&2; exit 1; fi
if [ ! -f "$TOKENIZER_PATH" ]; then echo "missing tokenizer: $TOKENIZER_PATH" >&2; exit 1; fi
# Background s2 server. Loads model on GPU once (per backend), then
# accepts multipart POSTs on localhost:3030/generate.
echo "[entrypoint] starting s2 server on :3030 with $BACKEND backend, device $DEVICE"
/usr/local/bin/s2 \
--server -H 127.0.0.1 -P 3030 \
-m "$MODEL_PATH" \
-t "$TOKENIZER_PATH" \
"${BACKEND_FLAG[@]}" &
S2_PID=$!
# Wait for s2 to bind 3030 before starting the shim. Avoids the
# obvious /v1/health 502 race on first boot.
echo "[entrypoint] waiting for s2 server to bind :3030"
for i in $(seq 1 60); do
if curl -sf -o /dev/null --max-time 1 http://127.0.0.1:3030/ 2>/dev/null; then
echo "[entrypoint] s2 server up after ${i}s"
break
fi
if ! kill -0 "$S2_PID" 2>/dev/null; then
echo "[entrypoint] s2 server died during startup" >&2
exit 1
fi
sleep 1
done
# Foreground uvicorn. tini (PID 1) gets uvicorn signals; on SIGTERM
# uvicorn exits, then s2 gets reaped via `wait` below.
echo "[entrypoint] starting uvicorn shim on :8000"
uvicorn server:app --host 0.0.0.0 --port 8000 --no-access-log &
UV_PID=$!
# Block until either child exits; propagate exit code.
wait -n "$S2_PID" "$UV_PID"
EXIT=$?
echo "[entrypoint] one child exited (rc=$EXIT) — shutting down peer"
kill "$S2_PID" "$UV_PID" 2>/dev/null || true
wait || true
exit "$EXIT"
stacks/fish-cpp/server.py
-105
View File
@@ -1,105 +0,0 @@
"""fish-cpp — HTTP shim wrapping s2.cpp's built-in server in Fish's
/v1/tts API contract.
Architecture: an `s2 --server` instance runs IN THE SAME CONTAINER on
localhost:3030 with the model loaded once on the GPU; this FastAPI
shim translates each Fish-shape JSON POST into a multipart form POST
against localhost:3030/generate and streams the WAV back.
The earlier subprocess-per-request design paid CUDA init + model load
(~10-20s) on every call, which dominated wall time. Keeping s2 server
resident moves the cost to startup (paid once) so per-request latency
matches what the model can actually do.
The shim still accepts (and ignores) `streaming: true` in the request
body for contract parity with fish-s2 — s2.cpp's /generate is
synchronous, so TTFB ≈ total wall time. If/when s2.cpp grows
incremental output, wire it through here.
"""
from __future__ import annotations
import base64
import io
import os
import tempfile
from pathlib import Path
from typing import Optional
import httpx
from fastapi import FastAPI, HTTPException
from fastapi.responses import Response
from pydantic import BaseModel
S2_BASE_URL = os.environ.get("S2_BASE_URL", "http://127.0.0.1:3030")
WEIGHTS_DIR = Path(os.environ.get("WEIGHTS_DIR", "/weights"))
MODEL_FILE = os.environ.get("FISH_CPP_MODEL", "s2-pro-q6_k.gguf")
TOKENIZER_FILE = os.environ.get("FISH_CPP_TOKENIZER", "tokenizer.json")
MODEL_PATH = WEIGHTS_DIR / MODEL_FILE
TOKENIZER_PATH = WEIGHTS_DIR / TOKENIZER_FILE
app = FastAPI(title="fish-cpp")
class ReferenceAudio(BaseModel):
audio: str # base64-encoded WAV bytes
text: str # transcript of the reference clip
class TTSRequest(BaseModel):
text: str
references: list[ReferenceAudio] = []
format: str = "wav"
streaming: bool = False # accepted for parity; ignored
@app.get("/v1/health")
def health() -> dict:
if not MODEL_PATH.exists():
raise HTTPException(503, f"model missing: {MODEL_PATH}")
if not TOKENIZER_PATH.exists():
raise HTTPException(503, f"tokenizer missing: {TOKENIZER_PATH}")
# Probe the s2 server backend — readiness, not just liveness.
try:
r = httpx.get(f"{S2_BASE_URL}/", timeout=2.0)
except httpx.HTTPError as exc:
raise HTTPException(503, f"s2 server unreachable: {exc}")
return {"status": "ok", "model": MODEL_FILE, "s2_status": r.status_code}
@app.post("/v1/tts")
def tts(req: TTSRequest) -> Response:
if req.format not in ("wav",):
raise HTTPException(400, f"only wav format supported, got {req.format}")
# Translate Fish-shaped JSON to s2.cpp's multipart form.
files: dict[str, tuple] = {
"text": (None, req.text),
}
if req.references:
# s2.cpp accepts a single reference at a time. If the client
# sent multiple, use the first.
ref = req.references[0]
files["prompt_text"] = (None, ref.text)
files["prompt_audio"] = ("ref.wav", base64.b64decode(ref.audio), "audio/wav")
try:
# Long timeout — s2 generation is sub-realtime on q6_k. The
# /v1/health probe upstream already vetted s2 server is live.
r = httpx.post(
f"{S2_BASE_URL}/generate", files=files, timeout=180.0,
)
except httpx.HTTPError as exc:
raise HTTPException(502, f"s2 server request failed: {exc}")
if r.status_code != 200:
# s2 returns JSON `{"error": "..."}` on failure
raise HTTPException(
500, f"s2 server returned {r.status_code}: {r.text[:500]}"
)
return Response(
content=r.content,
media_type=r.headers.get("content-type", "audio/wav"),
)