vh
a2b5b58eee
Replaces a bare nohup on irv-ml1:8410 that would not have survived a reboot, handed over by waterland-dev. Tracks vh/waterland @ main (PR #4 merged; main HEAD is exactly the pinned 8025366). Build context is a checkout at /opt/waterland-studio/src, deliberately OUTSIDE the compose dir — deploy-stack.sh rsyncs stacks/<stack>/ with --delete and would otherwise eat it. The Dockerfile is passed out-of-context. Three landmines, all measured: 1. Both uv extras are load-bearing at build AND run. jobs.py shells the renderer out as a literal with no --extra flags, so uv would re-sync at runtime and prune cupy — silently dropping to the numpy path at ~21x wall time. UV_NO_SYNC pins it; UV_OFFLINE makes any failure loud instead of quietly slow. 2. cupy needs CUDA HEADERS for its NVRTC compile, not just the driver and the wheel's runtime libs. The host has a system CUDA toolkit so the nohup process found them by accident; a slim image does not, and every render died 1.7s in with 'Failed to find CUDA headers' printed through argparse's usage banner — which reads like a CLI bug, not a missing toolkit. Fixed with cupy-cuda12x[ctk] (hundreds of MB, vs ~6 GB for a -devel base). 3. The A6000 is host device 1 but container device 0, since compose exposes exactly one GPU. CUDA_VISIBLE_DEVICES_TARGET=0 inside; copying the host's value selects a device that does not exist. /root/.cupy is a volume because the NVRTC compile costs ~17s: verified at 23.3s cold vs 6.1s warm, and re-verified across a restart (23.2s on a fresh cache volume, 6.0s once populated). Warm 256^2+anim beats the 7.4s recorded against bare metal, so containerising cost nothing. Job store seeded with the 4 jobs from the displaced instance. Serial by design (one replica, one card) and unauthenticated, so it stays LAN/WireGuard-only.
68 lines
3.2 KiB
Docker
68 lines
3.2 KiB
Docker
# waterland studio — GPU watercolour render service.
|
|
#
|
|
# Build context is a CHECKOUT OF vh/waterland, not this directory. See
|
|
# compose.yaml: context is /opt/waterland-studio/src and this Dockerfile is
|
|
# passed out-of-context so `deploy-stack.sh --delete` can never eat the
|
|
# checkout. Refresh the checkout with ./update.sh.
|
|
|
|
FROM python:3.12-slim
|
|
|
|
# ffmpeg is not optional — the CLI shells out to it for the VP9 encode of the
|
|
# reveal animation. Without it, plate renders succeed and animated ones fail
|
|
# at the very end of a 30s GPU job.
|
|
RUN apt-get update \
|
|
&& apt-get install -y --no-install-recommends ffmpeg ca-certificates \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
COPY --from=ghcr.io/astral-sh/uv:0.9.9 /uv /usr/local/bin/uv
|
|
|
|
WORKDIR /app
|
|
COPY . /app
|
|
|
|
# ⚠️ BOTH extras are load-bearing. `gpu` carries cupy-cuda12x; a bare
|
|
# `uv sync` PRUNES it and the renderer silently drops to the numpy path at
|
|
# roughly 21x the wall time — it does not error, it just gets slow. `studio`
|
|
# carries fastapi/uvicorn/python-multipart.
|
|
RUN uv sync --frozen --extra studio --extra gpu
|
|
|
|
# ⚠️ CUDA HEADERS — the dependency the host never had to declare.
|
|
# cupy compiles kernels at runtime through NVRTC, which needs the CUDA toolkit
|
|
# HEADERS present, not just the driver and the runtime libs bundled in the
|
|
# cupy-cuda12x wheel. On irv-ml1 that requirement was invisible: a CUDA toolkit
|
|
# is installed system-wide, so the bare `nohup` process found headers by
|
|
# accident. In a slim image there are none, and every render dies 1.7s in with
|
|
# RuntimeError: Failed to find CUDA headers.
|
|
# printed through argparse's usage banner, which makes it read like a CLI
|
|
# argument bug rather than a missing toolkit.
|
|
#
|
|
# The [ctk] extra pulls the header packages as wheels — a few hundred MB
|
|
# against ~6 GB for a -devel base image. It is installed AFTER the sync above
|
|
# because `uv sync` prunes anything it does not know about.
|
|
RUN uv pip install "cupy-cuda12x[ctk]"
|
|
|
|
ENV PATH="/app/.venv/bin:${PATH}" \
|
|
# ⚠️ THE SAME PRUNE TRAP, AT RUNTIME. studio/jobs.py shells the renderer
|
|
# out as a literal `uv run waterland ...` (cwd=WATERLAND_STUDIO_REPO), and
|
|
# that invocation carries no --extra flags. Left to itself uv would
|
|
# re-sync the project to its default extras and prune cupy right back out
|
|
# from under the venv built above. UV_NO_SYNC stops it re-syncing;
|
|
# UV_OFFLINE means that if the pin ever stops working the job fails LOUDLY
|
|
# instead of quietly rebuilding a slower environment.
|
|
UV_NO_SYNC=1 \
|
|
UV_OFFLINE=1 \
|
|
WATERLAND_STUDIO_REPO=/app \
|
|
WATERLAND_STUDIO_DATA=/data \
|
|
WATERLAND_STUDIO_BACKEND=cupy \
|
|
# PCI_BUS_ID index of the A6000 *as seen inside the container*. The
|
|
# compose file exposes exactly one GPU, so that GPU is index 0 here — even
|
|
# though it is index 1 on the host. Do not copy the host's value.
|
|
CUDA_VISIBLE_DEVICES_TARGET=0
|
|
|
|
EXPOSE 8410
|
|
|
|
# uvicorn is invoked from the venv directly rather than through `uv run`: the
|
|
# server has no reason to re-enter uv, and one less uv invocation is one less
|
|
# chance to trip the prune above. The app is a FACTORY, hence --factory.
|
|
CMD ["uvicorn", "--factory", "waterland.studio.app:app", \
|
|
"--host", "0.0.0.0", "--port", "8410"]
|