vh
ee35fcd0a9
Two issues from the first deploy attempt: 1) Build failure (real): linker errors on s2.cpp's CUDA build — undefined references to cuMemSetAccess, cuDeviceGet, etc. These are CUDA Driver API symbols (in libcuda.so), not Runtime API (libcudart.so). The driver lib is provided by NVIDIA's container runtime at RUN time, not BUILD time. Fix: nvidia/cuda:devel images ship a stubs library at /usr/local/cuda/lib64/stubs/libcuda.so that provides the symbols for linking but is non-runnable. Adding that path via LIBRARY_PATH + CMAKE_LIBRARY_PATH lets the linker resolve while leaving runtime unchanged (real libcuda.so comes from the driver mount). 2) Verify false positive: the /v1/tts verify step's last command was `rm -f "$out"` — which always exits 0. This made the shell's final exit code 0 regardless of whether curl/file/grep succeeded, so verify reported OK even when nothing was running on host_port. Fix: `set -e` at top + trap-based cleanup. Failures now propagate; the rm still runs on either path via EXIT trap.
72 lines
2.9 KiB
Docker
72 lines
2.9 KiB
Docker
# fish-cpp — s2.cpp (pure C++/GGML inference for Fish s2-pro GGUFs) +
|
|
# tiny FastAPI shim exposing Fish's /v1/tts contract.
|
|
#
|
|
# Two-stage build:
|
|
# 1. builder — compiles s2.cpp with CUDA backend
|
|
# 2. runtime — slim image with the s2 binary + Python shim
|
|
#
|
|
# The s2.cpp binary is the actual inference engine; the Python shim is
|
|
# just an HTTP-to-CLI bridge so this stack drops into the same fleet
|
|
# pattern as the other TTS (POST /v1/tts, Fish-shaped request body).
|
|
|
|
# ── Stage 1: build s2.cpp with CUDA ────────────────────────────────────
|
|
FROM nvidia/cuda:12.6.0-devel-ubuntu24.04 AS builder
|
|
|
|
ARG S2_CPP_SHA=e48ce8e02d8335bd9a0ba94679f605724b31d123
|
|
ENV DEBIAN_FRONTEND=noninteractive
|
|
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
git ca-certificates cmake ninja-build build-essential pkg-config \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
WORKDIR /src
|
|
RUN git clone --recurse-submodules https://github.com/rodrigomatta/s2.cpp.git \
|
|
&& cd s2.cpp \
|
|
&& git checkout ${S2_CPP_SHA} \
|
|
&& git submodule update --init --recursive
|
|
|
|
WORKDIR /src/s2.cpp
|
|
# CUDA Driver API symbols (cuMemSetAccess, cuDeviceGet, etc.) live in
|
|
# libcuda.so which the NVIDIA driver provides at RUNTIME via --gpus
|
|
# mount. At build time there's no GPU, so we use the stubs library
|
|
# at /usr/local/cuda/lib64/stubs/ which provides the symbols for
|
|
# linking but is NOT runnable. The runtime image uses the real
|
|
# driver-provided libcuda.so via NVIDIA's container runtime.
|
|
ENV LIBRARY_PATH=/usr/local/cuda/lib64/stubs:${LIBRARY_PATH}
|
|
RUN cmake -G Ninja -B build -DCMAKE_BUILD_TYPE=Release -DS2_CUDA=ON \
|
|
-DCMAKE_LIBRARY_PATH=/usr/local/cuda/lib64/stubs \
|
|
&& cmake --build build --parallel $(nproc) --target s2
|
|
|
|
# ── Stage 2: runtime — slim image with the binary + python shim ────────
|
|
FROM nvidia/cuda:12.6.0-runtime-ubuntu24.04
|
|
|
|
ENV DEBIAN_FRONTEND=noninteractive \
|
|
PYTHONUNBUFFERED=1 \
|
|
PYTHONDONTWRITEBYTECODE=1
|
|
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
python3 python3-pip python3-venv tini \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Pull the shim deps into an isolated venv so we don't fight system pip.
|
|
RUN python3 -m venv /opt/venv
|
|
ENV PATH="/opt/venv/bin:${PATH}"
|
|
RUN pip install --no-cache-dir 'fastapi>=0.115' 'uvicorn[standard]>=0.30' 'pydantic>=2'
|
|
|
|
# Copy the s2 binary + GGML runtime libs from the builder stage.
|
|
COPY --from=builder /src/s2.cpp/build/s2 /usr/local/bin/s2
|
|
COPY --from=builder /src/s2.cpp/build/ggml/src/libggml*.so /usr/local/lib/
|
|
COPY --from=builder /src/s2.cpp/build/ggml/src/ggml-cuda/libggml-cuda.so /usr/local/lib/
|
|
RUN ldconfig
|
|
|
|
WORKDIR /app
|
|
COPY server.py /app/server.py
|
|
|
|
# Bind-mounted at runtime: weights at /weights, references at /references.
|
|
VOLUME /weights
|
|
VOLUME /references
|
|
|
|
EXPOSE 8000
|
|
ENTRYPOINT ["/usr/bin/tini", "--"]
|
|
CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8000", "--no-access-log"]
|