feat(waterland-studio): containerise the GPU render service on irv-ml1

Replaces a bare nohup on irv-ml1:8410 that would not have survived a reboot, handed over by waterland-dev. Tracks vh/waterland @ main (PR #4 merged; main HEAD is exactly the pinned 8025366). Build context is a checkout at /opt/waterland-studio/src, deliberately OUTSIDE the compose dir — deploy-stack.sh rsyncs stacks/<stack>/ with --delete and would otherwise eat it. The Dockerfile is passed out-of-context. Three landmines, all measured: 1. Both uv extras are load-bearing at build AND run. jobs.py shells the renderer out as a literal with no --extra flags, so uv would re-sync at runtime and prune cupy — silently dropping to the numpy path at ~21x wall time. UV_NO_SYNC pins it; UV_OFFLINE makes any failure loud instead of quietly slow. 2. cupy needs CUDA HEADERS for its NVRTC compile, not just the driver and the wheel's runtime libs. The host has a system CUDA toolkit so the nohup process found them by accident; a slim image does not, and every render died 1.7s in with 'Failed to find CUDA headers' printed through argparse's usage banner — which reads like a CLI bug, not a missing toolkit. Fixed with cupy-cuda12x[ctk] (hundreds of MB, vs ~6 GB for a -devel base). 3. The A6000 is host device 1 but container device 0, since compose exposes exactly one GPU. CUDA_VISIBLE_DEVICES_TARGET=0 inside; copying the host's value selects a device that does not exist. /root/.cupy is a volume because the NVRTC compile costs ~17s: verified at 23.3s cold vs 6.1s warm, and re-verified across a restart (23.2s on a fresh cache volume, 6.0s once populated). Warm 256^2+anim beats the 7.4s recorded against bare metal, so containerising cost nothing. Job store seeded with the 4 jobs from the displaced instance. Serial by design (one replica, one card) and unauthenticated, so it stays LAN/WireGuard-only.
2026-08-19 00:42:12 -07:00
parent df68dd2753
commit a2b5b58eee
5 changed files with 335 additions and 0 deletions
@@ -0,0 +1,11 @@
+# waterland-studio on irv-ml1. Real .env lives on the host; every value below
+# is the compose default, so an absent .env is a working configuration.
+WLS_PORT=8410
+WLS_CONTAINER=waterland-studio
+WLS_BACKEND=cupy
+# Docker's device index for the A6000 on this host. Device 0 is the 3090 and
+# hosts the TTS zoo — do not point this at it.
+WLS_GPU_ID=1
+# The A6000's index INSIDE the container. Exactly one GPU is exposed, so it is
+# 0 here even though it is 1 on the host. See the Dockerfile.
+WLS_CUDA_TARGET=0
@@ -0,0 +1,67 @@
+# waterland studio — GPU watercolour render service.
+#
+# Build context is a CHECKOUT OF vh/waterland, not this directory. See
+# compose.yaml: context is /opt/waterland-studio/src and this Dockerfile is
+# passed out-of-context so `deploy-stack.sh --delete` can never eat the
+# checkout. Refresh the checkout with ./update.sh.
+
+FROM python:3.12-slim
+
+# ffmpeg is not optional — the CLI shells out to it for the VP9 encode of the
+# reveal animation. Without it, plate renders succeed and animated ones fail
+# at the very end of a 30s GPU job.
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends ffmpeg ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY --from=ghcr.io/astral-sh/uv:0.9.9 /uv /usr/local/bin/uv
+
+WORKDIR /app
+COPY . /app
+
+# ⚠️ BOTH extras are load-bearing. `gpu` carries cupy-cuda12x; a bare
+# `uv sync` PRUNES it and the renderer silently drops to the numpy path at
+# roughly 21x the wall time — it does not error, it just gets slow. `studio`
+# carries fastapi/uvicorn/python-multipart.
+RUN uv sync --frozen --extra studio --extra gpu
+
+# ⚠️ CUDA HEADERS — the dependency the host never had to declare.
+# cupy compiles kernels at runtime through NVRTC, which needs the CUDA toolkit
+# HEADERS present, not just the driver and the runtime libs bundled in the
+# cupy-cuda12x wheel. On irv-ml1 that requirement was invisible: a CUDA toolkit
+# is installed system-wide, so the bare `nohup` process found headers by
+# accident. In a slim image there are none, and every render dies 1.7s in with
+#   RuntimeError: Failed to find CUDA headers.
+# printed through argparse's usage banner, which makes it read like a CLI
+# argument bug rather than a missing toolkit.
+#
+# The [ctk] extra pulls the header packages as wheels — a few hundred MB
+# against ~6 GB for a -devel base image. It is installed AFTER the sync above
+# because `uv sync` prunes anything it does not know about.
+RUN uv pip install "cupy-cuda12x[ctk]"
+
+ENV PATH="/app/.venv/bin:${PATH}" \
+    # ⚠️ THE SAME PRUNE TRAP, AT RUNTIME. studio/jobs.py shells the renderer
+    # out as a literal `uv run waterland ...` (cwd=WATERLAND_STUDIO_REPO), and
+    # that invocation carries no --extra flags. Left to itself uv would
+    # re-sync the project to its default extras and prune cupy right back out
+    # from under the venv built above. UV_NO_SYNC stops it re-syncing;
+    # UV_OFFLINE means that if the pin ever stops working the job fails LOUDLY
+    # instead of quietly rebuilding a slower environment.
+    UV_NO_SYNC=1 \
+    UV_OFFLINE=1 \
+    WATERLAND_STUDIO_REPO=/app \
+    WATERLAND_STUDIO_DATA=/data \
+    WATERLAND_STUDIO_BACKEND=cupy \
+    # PCI_BUS_ID index of the A6000 *as seen inside the container*. The
+    # compose file exposes exactly one GPU, so that GPU is index 0 here — even
+    # though it is index 1 on the host. Do not copy the host's value.
+    CUDA_VISIBLE_DEVICES_TARGET=0
+
+EXPOSE 8410
+
+# uvicorn is invoked from the venv directly rather than through `uv run`: the
+# server has no reason to re-enter uv, and one less uv invocation is one less
+# chance to trip the prune above. The app is a FACTORY, hence --factory.
+CMD ["uvicorn", "--factory", "waterland.studio.app:app", \
+     "--host", "0.0.0.0", "--port", "8410"]
@@ -0,0 +1,115 @@
+# waterland-studio — watercolour render service (irv-ml1)
+
+FastAPI + vanilla-JS SPA fronting the `waterland` CLI: upload an image, get a
+watercolour plate and a painted-in reveal animation. Every job shells out to
+the CLI, which runs a fluid simulation on the **A6000**.
+
+- **Host:** irv-ml1 (10.100.79.3, WireGuard-only) · **Port:** 8410
+- **URL:** http://10.100.79.3:8410/ · **Health:** `GET /api/health`
+- **Source:** `gitea.phasefinal.com/vh/waterland`, tracking **`main`**
+
+Handed over by `waterland-dev` on 2026-08-19, replacing a bare `nohup` that
+would not have survived a reboot.
+
+## Layout — the build context is deliberately outside this directory
+
+| path | what |
+|---|---|
+| `/opt/waterland-studio/src` | checkout of `vh/waterland` @ `main` — the build context |
+| `/opt/docker/compose/waterland-studio/` | `compose.yaml`, `Dockerfile`, `update.sh`, `.env` |
+| volume `waterland-studio_waterland_studio_data` | job store (`/data`) |
+| volume `waterland-studio_waterland_studio_kernels` | cupy JIT cache (`/root/.cupy`) |
+
+**The checkout must NOT live under the compose directory.** `deploy-stack.sh`
+rsyncs `stacks/<stack>/` with `--delete`, so a checkout kept beside
+`compose.yaml` would be destroyed by the next deploy of this stack. The
+Dockerfile is passed out-of-context to keep both trees clean.
+
+Refresh source + rebuild:
+
+```bash
+ssh infra-ops@10.100.79.3 /opt/docker/compose/waterland-studio/update.sh
+```
+
+## Three landmines, all of them measured rather than guessed
+
+**1. Both uv extras are load-bearing, at build AND at run.** `gpu` carries
+`cupy-cuda12x`; a bare `uv sync` prunes it and the renderer silently drops to
+the numpy path at ~21x the wall time — it does not error, it just gets slow.
+Worse, `studio/jobs.py` shells the renderer out as a literal `uv run waterland`
+with no `--extra` flags, so uv would re-sync at *runtime* and prune cupy right
+back out. `UV_NO_SYNC=1` stops that; `UV_OFFLINE=1` means that if the pin ever
+stops working the job fails loudly instead of quietly rebuilding a slower
+environment.
+
+**2. cupy needs CUDA *headers*, which the host never had to declare.** cupy
+compiles kernels at runtime through NVRTC, which needs toolkit headers — not
+just the driver and the runtime libs bundled in the wheel. On irv-ml1 a CUDA
+toolkit is installed system-wide, so the bare `nohup` process found them by
+accident; a slim image has none. Every render died 1.7s in with
+
+```
+RuntimeError: Failed to find CUDA headers.
+```
+
+printed *through argparse's usage banner*, which makes it read like a CLI
+argument bug rather than a missing toolkit — that misdirection is the reason
+this is written down. Fixed with `uv pip install "cupy-cuda12x[ctk]"`, which
+pulls the headers as wheels: a few hundred MB against ~6 GB for a `-devel`
+base image. It runs *after* `uv sync`, because sync prunes what it does not
+know about.
+
+**3. The GPU index inside the container is not the host's.** The app pins
+`CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects `CUDA_VISIBLE_DEVICES_TARGET`
+(default `1`, correct on the host). Compose exposes exactly one GPU
+(`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the
+container that card is index **0** — hence `CUDA_VISIBLE_DEVICES_TARGET=0`.
+Copying the host's value selects a device that does not exist. Device 0 on the
+host is the 3090, which hosts the TTS zoo and must not be touched.
+
+## Performance, measured on this host
+
+| job | wall |
+|---|---|
+| 256², animation, **cold container** | 23.3 s |
+| 256², animation, warm | 6.1 s |
+| 256², plate only (`--codec none`) | 3.9 s |
+| 512², plate only | 6.4 s |
+
+Warm numbers beat the 7.4 s at 256² recorded against the bare-metal process, so
+containerising cost nothing. The cold-vs-warm gap is **cupy's NVRTC compile**,
+which is why `/root/.cupy` is a volume: verified by recreating the container
+(fresh cache → 23.2 s first render) and then restarting it (populated cache →
+6.0 s). Without that volume every restart makes the next user wait 4x and the
+service merely looks slow.
+
+## Operational constraints — from waterland-dev, not inferred
+
+- **Serial by design. One replica, one card.** A render is 20–45 s of near-full
+  GPU and the app runs a single worker thread. Two of these on the same A6000
+  would OOM or thrash. Throughput is a conversation about hardware, not replica
+  count.
+- **No authentication, and it accepts arbitrary file uploads.** It must stay
+  inside the LAN / WireGuard boundary. Do **not** paper over this with a proxy
+  password — waterland-dev has offered to add a real auth layer if it ever
+  needs wider reach. Ask.
+- Job store is scratch output, not source-of-truth: ~12 MB per animated job,
+  self-evicting at 40 retained jobs (`RETAIN` in `studio/jobs.py`), so steady
+  state is bounded around 500 MB. The 4 jobs from the bare-metal instance were
+  copied in at cutover.
+- Internal render timeout is 480 s, which is why the healthcheck interval is
+  loose — an aggressive probe would measure queue depth rather than liveness.
+
+## ⚠️ `update.sh` needs a credential this host does not have
+
+The repo is not anonymously readable — an unauthenticated clone 403s. The
+initial checkout was made with the operator's `vh` site-admin token passed
+inline and **not persisted**: the on-disk remote is the plain URL and
+`.git/config` holds no token (verified). Consequently `git fetch` in
+`update.sh` will fail until the host has a credential of its own.
+
+`claude-bot` 404s on `vh/waterland`, so it currently lacks read access. The
+right fix is a read-only deploy token for this host, or granting `claude-bot`
+read on the repo — **not** writing the site-admin token to disk on a GPU box.
+Raised with waterland-dev; until then, re-run the authenticated clone by hand
+to update.
@@ -0,0 +1,96 @@
+# waterland-studio — watercolour render service on irv-ml1, port 8410.
+#
+# FastAPI + vanilla-JS SPA fronting the waterland CLI: upload an image, get a
+# watercolour plate and a painted-in reveal animation. Every job shells out to
+# the CLI, which runs a fluid simulation on the A6000.
+#
+# Handed over by waterland-dev 2026-08-19, replacing a bare `nohup` that would
+# not have survived a reboot.
+#
+# ⚠️ THE BUILD CONTEXT LIVES OUTSIDE THIS DIRECTORY, DELIBERATELY.
+#   /opt/waterland-studio/src is a checkout of vh/waterland @ main.
+#   `deploy-stack.sh` rsyncs this stack dir with --delete, so a checkout kept
+#   in here would be destroyed on the next deploy. Refresh it with ./update.sh.
+#
+# ⚠️ SERIAL BY DESIGN — ONE REPLICA, ONE CARD. A render is 20-45s of near-full
+#   GPU and the app runs a single worker thread. Two of these on the same A6000
+#   would OOM or thrash. Throughput is a conversation about hardware, not about
+#   replica count (waterland-dev, explicitly).
+#
+# ⚠️ NO AUTHENTICATION, AND IT ACCEPTS ARBITRARY FILE UPLOADS. It must stay
+#   inside the LAN / WireGuard boundary. Do NOT paper over this by putting it
+#   behind a proxy with a password — waterland-dev has offered to add a real
+#   auth layer if it ever needs wider reach. Ask, don't improvise.
+
+name: waterland-studio
+
+services:
+  waterland-studio:
+    build:
+      # Absolute paths: the context is the source checkout, the Dockerfile is
+      # this version-controlled one, and the two live in different trees.
+      context: /opt/waterland-studio/src
+      dockerfile: /opt/docker/compose/waterland-studio/Dockerfile
+    image: waterland-studio:local
+    container_name: ${WLS_CONTAINER:-waterland-studio}
+    restart: unless-stopped
+    ports:
+      - "${WLS_PORT:-8410}:8410"
+    volumes:
+      # Job store: uploaded sources plus rendered plates and animations.
+      # ~12 MB per job with an animation; the app self-evicts at 40 retained
+      # jobs (RETAIN in studio/jobs.py), so steady state is bounded ~500 MB.
+      # Scratch output, not source-of-truth — losing it costs a re-render.
+      - waterland_studio_data:/data
+      # cupy JIT kernel cache. Not optional for good behaviour: cupy compiles
+      # its kernels through NVRTC on first use, and measured on this host that
+      # cold compile costs ~17s — the first 256^2 render after a fresh
+      # container took 23.3s against 6.1s warm. Without this volume every
+      # restart makes the next user wait 4x, and it looks like the service is
+      # slow rather than warming up.
+      - waterland_studio_kernels:/root/.cupy
+    environment:
+      - WATERLAND_STUDIO_DATA=/data
+      - WATERLAND_STUDIO_REPO=/app
+      - WATERLAND_STUDIO_BACKEND=${WLS_BACKEND:-cupy}
+      # 0, not 1 — see the Dockerfile. Exactly one GPU is exposed below, so
+      # inside this container the A6000 is index 0 under PCI_BUS_ID ordering.
+      - CUDA_VISIBLE_DEVICES_TARGET=${WLS_CUDA_TARGET:-0}
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              # "1" is the A6000 in DOCKER's device ordering, matching the
+              # comfyui stack on this host. Device 0 is the 3090, which hosts
+              # the TTS zoo and must not be touched.
+              device_ids: ["${WLS_GPU_ID:-1}"]
+              capabilities: [gpu]
+    healthcheck:
+      # /api/health touches no GPU and is safe to poll. The interval is
+      # deliberately loose: a render holds the GPU for 20-45s and the app's own
+      # job timeout is 480s, so an aggressive probe would be measuring queue
+      # depth rather than liveness.
+      test: ["CMD-SHELL", "python -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8410/api/health', timeout=5).status==200 else 1)\""]
+      interval: 60s
+      timeout: 10s
+      retries: 3
+      start_period: 30s
+    networks:
+      - tnet
+    labels:
+      - homepage.group=AI - Image & Media
+      - homepage.name=Waterland Studio
+      - homepage.icon=mdi-watercolor
+      - homepage.description=Watercolour plate + reveal animation renderer (irv-ml1, A6000)
+      - homepage.href=http://10.100.79.3:${WLS_PORT:-8410}/
+      - homepage.siteMonitor=http://10.100.79.3:${WLS_PORT:-8410}/api/health
+
+volumes:
+  waterland_studio_data: {}
+  waterland_studio_kernels: {}
+
+networks:
+  tnet:
+    name: traefik-net
+    external: true
@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+# Refresh the waterland checkout and rebuild the studio image.
+#
+# The checkout deliberately lives OUTSIDE the compose directory:
+# deploy-stack.sh rsyncs stacks/<stack>/ with --delete, so a checkout kept
+# beside compose.yaml would be deleted by the next deploy of this stack.
+#
+# Run ON irv-ml1:
+#   /opt/docker/compose/waterland-studio/update.sh
+set -euo pipefail
+
+SRC=/opt/waterland-studio/src
+COMPOSE_DIR=/opt/docker/compose/waterland-studio
+REPO_URL=${WATERLAND_REPO_URL:-https://gitea.phasefinal.com/vh/waterland.git}
+REF=${WATERLAND_REF:-main}
+
+if [ ! -d "$SRC/.git" ]; then
+  echo "no checkout at $SRC — clone it first:"
+  echo "  sudo mkdir -p $(dirname "$SRC")"
+  echo "  sudo git clone $REPO_URL $SRC"
+  exit 1
+fi
+
+echo "==> fetching $REF"
+git -C "$SRC" fetch --prune origin
+git -C "$SRC" checkout -q "$REF"
+git -C "$SRC" reset --hard "origin/$REF"
+echo "==> now at $(git -C "$SRC" rev-parse --short HEAD): $(git -C "$SRC" log -1 --format=%s)"
+
+echo "==> rebuilding"
+cd "$COMPOSE_DIR"
+docker compose build --pull
+
+echo "==> restarting"
+docker compose up -d
+
+echo "==> waiting for health"
+for _ in $(seq 1 30); do
+  if curl -fsS -m 5 http://127.0.0.1:8410/api/health >/dev/null 2>&1; then
+    echo "healthy: $(curl -fsS -m 5 http://127.0.0.1:8410/api/health)"
+    exit 0
+  fi
+  sleep 3
+done
+echo "did not come healthy within 90s — check: docker compose logs --tail 50" >&2
+exit 1