vh
67813bbef4
CUDA backend confirmed broken for fish-speech ops on s2.cpp v0.x — alpha, incomplete op coverage, GPU stays at 0% during generation despite ggml_cuda_init succeeding. Vulkan was the original README example (`-v 0`), so likely the more battle-tested path. Build the image with BOTH backends so we can flip via env without rebuilding: * libvulkan-dev + glslc in the build stage (GGML's Vulkan backend compiles its shaders with glslc at build time; without it the cmake configure silently disables Vulkan). * libvulkan1 + the libggml-vulkan.so copy in the runtime stage. * compose env NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics — default nvidia-container-toolkit only mounts compute libs; Vulkan needs the graphics ICD (libGLX_nvidia + nvidia_icd.json) too. * entrypoint reads FISH_CPP_BACKEND (cuda/vulkan/cpu) and selects the appropriate -c/-v/no-flag invocation. * Default backend = vulkan.
65 lines
2.8 KiB
YAML
65 lines
2.8 KiB
YAML
# fish-cpp — Fish s2-pro served via s2.cpp (pure C++/GGML inference,
|
||
# CUDA backend) with a tiny FastAPI shim exposing Fish's /v1/tts
|
||
# contract. Built locally from rodrigomatta/s2.cpp + a pinned SHA.
|
||
#
|
||
# Why this stack alongside fish-s2:
|
||
# * fish-s2 (HF transformers wrapper): ~7-8 s TTFB, 0.78× realtime.
|
||
# Phenomenal quality but sub-realtime, buffer-underruns on long
|
||
# phrases even with streaming.
|
||
# * fish-cpp (this): targets ~2-5× speedup from the GGML inference
|
||
# path + q6_k quantization. Goal: hit realtime for streaming use.
|
||
#
|
||
# CAVEATS:
|
||
# * s2.cpp is alpha software (per upstream README). Pin the SHA;
|
||
# don't track main blindly.
|
||
# * Per-request subprocess spawn — each /v1/tts call forks the s2
|
||
# binary. Adds ~50-100 ms over a long-running daemon. Negligible
|
||
# vs the multi-second generation cost.
|
||
# * No streaming — s2.cpp writes a complete WAV before returning,
|
||
# so the shim's `streaming: true` field is accepted but ignored.
|
||
# TTFB ≈ total wall time (fast generation is the only path to
|
||
# low-latency here, not chunked output).
|
||
|
||
services:
|
||
fish-cpp:
|
||
image: local/fish-cpp:${FISH_CPP_TAG}
|
||
build:
|
||
context: .
|
||
dockerfile: Dockerfile
|
||
args:
|
||
S2_CPP_SHA: ${FISH_CPP_S2_SHA:-e48ce8e02d8335bd9a0ba94679f605724b31d123}
|
||
container_name: fish-cpp
|
||
restart: unless-stopped
|
||
runtime: nvidia
|
||
ports:
|
||
- "${FISH_CPP_BIND:-0.0.0.0}:${FISH_CPP_PORT}:8000"
|
||
environment:
|
||
- NVIDIA_VISIBLE_DEVICES=${FISH_CPP_GPU_DEVICES:-1}
|
||
# Default nvidia-container-toolkit only mounts compute libs.
|
||
# Vulkan needs the graphics ICD too (libGLX_nvidia, vulkan ICD
|
||
# JSON). Without this, Vulkan init in the container fails with
|
||
# "no Vulkan ICD" even though the GPU is present.
|
||
- NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics
|
||
- FISH_CPP_MODEL=${FISH_CPP_MODEL:-s2-pro-q6_k.gguf}
|
||
- FISH_CPP_TOKENIZER=tokenizer.json
|
||
- FISH_CPP_DEVICE=0
|
||
# Backend selection for the s2 server in entrypoint.sh:
|
||
# cuda — -c <id>; alpha CUDA path, GPU 0% util on fish-speech (broken)
|
||
# vulkan — -v <id>; README's canonical example, more battle-tested
|
||
- FISH_CPP_BACKEND=${FISH_CPP_BACKEND:-vulkan}
|
||
volumes:
|
||
- ${FISH_CPP_WEIGHTS_DIR}:/weights:ro
|
||
- ${FISH_CPP_REFERENCE_DIR}:/references:ro
|
||
healthcheck:
|
||
test: ["CMD-SHELL", "python3 -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8000/v1/health', timeout=5).status==200 else 1)\""]
|
||
interval: 30s
|
||
timeout: 10s
|
||
retries: 3
|
||
start_period: 60s
|
||
labels:
|
||
- homepage.group=AI Systems
|
||
- homepage.name=Fish (s2.cpp)
|
||
- homepage.icon=mdi-fish
|
||
- homepage.description=Fish s2-pro via s2.cpp/GGML — quantized for realtime (irv-ml1)
|
||
- homepage.href=http://10.100.79.3:${FISH_CPP_PORT}
|