vh
67813bbef4
CUDA backend confirmed broken for fish-speech ops on s2.cpp v0.x — alpha, incomplete op coverage, GPU stays at 0% during generation despite ggml_cuda_init succeeding. Vulkan was the original README example (`-v 0`), so likely the more battle-tested path. Build the image with BOTH backends so we can flip via env without rebuilding: * libvulkan-dev + glslc in the build stage (GGML's Vulkan backend compiles its shaders with glslc at build time; without it the cmake configure silently disables Vulkan). * libvulkan1 + the libggml-vulkan.so copy in the runtime stage. * compose env NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics — default nvidia-container-toolkit only mounts compute libs; Vulkan needs the graphics ICD (libGLX_nvidia + nvidia_icd.json) too. * entrypoint reads FISH_CPP_BACKEND (cuda/vulkan/cpu) and selects the appropriate -c/-v/no-flag invocation. * Default backend = vulkan.
101 lines
4.6 KiB
Docker
101 lines
4.6 KiB
Docker
# fish-cpp — s2.cpp (pure C++/GGML inference for Fish s2-pro GGUFs) +
|
|
# tiny FastAPI shim exposing Fish's /v1/tts contract.
|
|
#
|
|
# Two-stage build:
|
|
# 1. builder — compiles s2.cpp with CUDA backend
|
|
# 2. runtime — slim image with the s2 binary + Python shim
|
|
#
|
|
# The s2.cpp binary is the actual inference engine; the Python shim is
|
|
# just an HTTP-to-CLI bridge so this stack drops into the same fleet
|
|
# pattern as the other TTS (POST /v1/tts, Fish-shaped request body).
|
|
|
|
# ── Stage 1: build s2.cpp with CUDA ────────────────────────────────────
|
|
FROM nvidia/cuda:12.6.0-devel-ubuntu24.04 AS builder
|
|
|
|
ARG S2_CPP_SHA=e48ce8e02d8335bd9a0ba94679f605724b31d123
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
git ca-certificates cmake ninja-build build-essential pkg-config \
|
|
libvulkan-dev glslc \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
# libvulkan-dev = Vulkan headers + loader for build-time linking.
|
|
# glslc = GLSL→SPIR-V compiler; GGML's Vulkan backend compiles its
|
|
# shaders with this at build time. Without it, the cmake configure
|
|
# step disables Vulkan silently.
|
|
|
|
WORKDIR /src
|
|
RUN git clone --recurse-submodules https://github.com/rodrigomatta/s2.cpp.git \
|
|
&& cd s2.cpp \
|
|
&& git checkout ${S2_CPP_SHA} \
|
|
&& git submodule update --init --recursive
|
|
|
|
WORKDIR /src/s2.cpp
|
|
# CUDA Driver API symbols (cuMemSetAccess, cuDeviceGet, etc.) live in
|
|
# libcuda.so which the NVIDIA driver provides at RUNTIME. Build-time
|
|
# uses the stubs library at /usr/local/cuda/lib64/stubs/. CMake's
|
|
# find_library doesn't reliably pick it up via CMAKE_LIBRARY_PATH for
|
|
# nested ggml-cuda builds — the most robust fix is symlinking the stub
|
|
# into /usr/local/lib (which ld checks unconditionally) AND providing
|
|
# libcuda.so.1 (the SONAME ggml-cuda links against). The symlinks live
|
|
# only in this build stage; the runtime image gets the real driver-
|
|
# provided libcuda.so.1 via NVIDIA's container runtime.
|
|
RUN ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/lib/libcuda.so \
|
|
&& ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/lib/libcuda.so.1 \
|
|
&& ldconfig
|
|
# Build with BOTH CUDA and Vulkan backends — runtime selects via the
|
|
# entrypoint (-c <id> for CUDA, -v <id> for Vulkan). We've established
|
|
# that s2.cpp's CUDA path silently falls back to CPU on this model
|
|
# (alpha software, incomplete op coverage); Vulkan is the README's
|
|
# canonical example so it likely has more complete op coverage on
|
|
# fish-speech. Keep both built so we can flip without rebuilding.
|
|
RUN cmake -G Ninja -B build -DCMAKE_BUILD_TYPE=Release \
|
|
-DS2_CUDA=ON -DS2_VULKAN=ON \
|
|
&& cmake --build build --parallel $(nproc) --target s2
|
|
|
|
# ── Stage 2: runtime — slim image with the binary + python shim ────────
|
|
FROM nvidia/cuda:12.6.0-runtime-ubuntu24.04
|
|
|
|
ENV DEBIAN_FRONTEND=noninteractive \
|
|
PYTHONUNBUFFERED=1 \
|
|
PYTHONDONTWRITEBYTECODE=1
|
|
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
python3 python3-pip python3-venv tini \
|
|
libgomp1 curl \
|
|
libvulkan1 \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
# libgomp1 = GNU OpenMP runtime — required by the s2 binary.
|
|
# curl = entrypoint uses it to wait for s2 server to bind :3030.
|
|
# libvulkan1 = Vulkan loader. The NVIDIA Vulkan ICD itself comes from
|
|
# the host driver via NVIDIA container runtime when
|
|
# NVIDIA_DRIVER_CAPABILITIES includes "graphics".
|
|
|
|
# Pull the shim deps into an isolated venv so we don't fight system pip.
|
|
RUN python3 -m venv /opt/venv
|
|
ENV PATH="/opt/venv/bin:${PATH}"
|
|
RUN pip install --no-cache-dir 'fastapi>=0.115' 'uvicorn[standard]>=0.30' 'pydantic>=2' 'httpx>=0.27'
|
|
|
|
# Copy the s2 binary + GGML runtime libs from the builder stage.
|
|
COPY --from=builder /src/s2.cpp/build/s2 /usr/local/bin/s2
|
|
COPY --from=builder /src/s2.cpp/build/ggml/src/libggml*.so /usr/local/lib/
|
|
COPY --from=builder /src/s2.cpp/build/ggml/src/ggml-cuda/libggml-cuda.so /usr/local/lib/
|
|
COPY --from=builder /src/s2.cpp/build/ggml/src/ggml-vulkan/libggml-vulkan.so /usr/local/lib/
|
|
RUN ldconfig
|
|
|
|
WORKDIR /app
|
|
COPY server.py /app/server.py
|
|
COPY entrypoint.sh /app/entrypoint.sh
|
|
RUN chmod +x /app/entrypoint.sh
|
|
|
|
# Bind-mounted at runtime: weights at /weights, references at /references.
|
|
VOLUME /weights
|
|
VOLUME /references
|
|
|
|
EXPOSE 8000
|
|
# tini supervises the entrypoint script which manages both s2 server
|
|
# (bound to localhost:3030, model resident) and the uvicorn shim
|
|
# (bound to 0.0.0.0:8000, proxies /v1/tts → s2's /generate).
|
|
ENTRYPOINT ["/usr/bin/tini", "--"]
|
|
CMD ["/app/entrypoint.sh"]
|