feat(waterland-studio): containerise the GPU render service on irv-ml1

Replaces a bare nohup on irv-ml1:8410 that would not have survived a reboot, handed over by waterland-dev. Tracks vh/waterland @ main (PR #4 merged; main HEAD is exactly the pinned 8025366). Build context is a checkout at /opt/waterland-studio/src, deliberately OUTSIDE the compose dir — deploy-stack.sh rsyncs stacks/<stack>/ with --delete and would otherwise eat it. The Dockerfile is passed out-of-context. Three landmines, all measured: 1. Both uv extras are load-bearing at build AND run. jobs.py shells the renderer out as a literal with no --extra flags, so uv would re-sync at runtime and prune cupy — silently dropping to the numpy path at ~21x wall time. UV_NO_SYNC pins it; UV_OFFLINE makes any failure loud instead of quietly slow. 2. cupy needs CUDA HEADERS for its NVRTC compile, not just the driver and the wheel's runtime libs. The host has a system CUDA toolkit so the nohup process found them by accident; a slim image does not, and every render died 1.7s in with 'Failed to find CUDA headers' printed through argparse's usage banner — which reads like a CLI bug, not a missing toolkit. Fixed with cupy-cuda12x[ctk] (hundreds of MB, vs ~6 GB for a -devel base). 3. The A6000 is host device 1 but container device 0, since compose exposes exactly one GPU. CUDA_VISIBLE_DEVICES_TARGET=0 inside; copying the host's value selects a device that does not exist. /root/.cupy is a volume because the NVRTC compile costs ~17s: verified at 23.3s cold vs 6.1s warm, and re-verified across a restart (23.2s on a fresh cache volume, 6.0s once populated). Warm 256^2+anim beats the 7.4s recorded against bare metal, so containerising cost nothing. Job store seeded with the 4 jobs from the displaced instance. Serial by design (one replica, one card) and unauthenticated, so it stays LAN/WireGuard-only.
2026-08-19 00:42:12 -07:00
parent df68dd2753
commit a2b5b58eee
5 changed files with 335 additions and 0 deletions
@@ -0,0 +1,96 @@
+# waterland-studio — watercolour render service on irv-ml1, port 8410.
+#
+# FastAPI + vanilla-JS SPA fronting the waterland CLI: upload an image, get a
+# watercolour plate and a painted-in reveal animation. Every job shells out to
+# the CLI, which runs a fluid simulation on the A6000.
+#
+# Handed over by waterland-dev 2026-08-19, replacing a bare `nohup` that would
+# not have survived a reboot.
+#
+# ⚠️ THE BUILD CONTEXT LIVES OUTSIDE THIS DIRECTORY, DELIBERATELY.
+#   /opt/waterland-studio/src is a checkout of vh/waterland @ main.
+#   `deploy-stack.sh` rsyncs this stack dir with --delete, so a checkout kept
+#   in here would be destroyed on the next deploy. Refresh it with ./update.sh.
+#
+# ⚠️ SERIAL BY DESIGN — ONE REPLICA, ONE CARD. A render is 20-45s of near-full
+#   GPU and the app runs a single worker thread. Two of these on the same A6000
+#   would OOM or thrash. Throughput is a conversation about hardware, not about
+#   replica count (waterland-dev, explicitly).
+#
+# ⚠️ NO AUTHENTICATION, AND IT ACCEPTS ARBITRARY FILE UPLOADS. It must stay
+#   inside the LAN / WireGuard boundary. Do NOT paper over this by putting it
+#   behind a proxy with a password — waterland-dev has offered to add a real
+#   auth layer if it ever needs wider reach. Ask, don't improvise.
+
+name: waterland-studio
+
+services:
+  waterland-studio:
+    build:
+      # Absolute paths: the context is the source checkout, the Dockerfile is
+      # this version-controlled one, and the two live in different trees.
+      context: /opt/waterland-studio/src
+      dockerfile: /opt/docker/compose/waterland-studio/Dockerfile
+    image: waterland-studio:local
+    container_name: ${WLS_CONTAINER:-waterland-studio}
+    restart: unless-stopped
+    ports:
+      - "${WLS_PORT:-8410}:8410"
+    volumes:
+      # Job store: uploaded sources plus rendered plates and animations.
+      # ~12 MB per job with an animation; the app self-evicts at 40 retained
+      # jobs (RETAIN in studio/jobs.py), so steady state is bounded ~500 MB.
+      # Scratch output, not source-of-truth — losing it costs a re-render.
+      - waterland_studio_data:/data
+      # cupy JIT kernel cache. Not optional for good behaviour: cupy compiles
+      # its kernels through NVRTC on first use, and measured on this host that
+      # cold compile costs ~17s — the first 256^2 render after a fresh
+      # container took 23.3s against 6.1s warm. Without this volume every
+      # restart makes the next user wait 4x, and it looks like the service is
+      # slow rather than warming up.
+      - waterland_studio_kernels:/root/.cupy
+    environment:
+      - WATERLAND_STUDIO_DATA=/data
+      - WATERLAND_STUDIO_REPO=/app
+      - WATERLAND_STUDIO_BACKEND=${WLS_BACKEND:-cupy}
+      # 0, not 1 — see the Dockerfile. Exactly one GPU is exposed below, so
+      # inside this container the A6000 is index 0 under PCI_BUS_ID ordering.
+      - CUDA_VISIBLE_DEVICES_TARGET=${WLS_CUDA_TARGET:-0}
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              # "1" is the A6000 in DOCKER's device ordering, matching the
+              # comfyui stack on this host. Device 0 is the 3090, which hosts
+              # the TTS zoo and must not be touched.
+              device_ids: ["${WLS_GPU_ID:-1}"]
+              capabilities: [gpu]
+    healthcheck:
+      # /api/health touches no GPU and is safe to poll. The interval is
+      # deliberately loose: a render holds the GPU for 20-45s and the app's own
+      # job timeout is 480s, so an aggressive probe would be measuring queue
+      # depth rather than liveness.
+      test: ["CMD-SHELL", "python -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8410/api/health', timeout=5).status==200 else 1)\""]
+      interval: 60s
+      timeout: 10s
+      retries: 3
+      start_period: 30s
+    networks:
+      - tnet
+    labels:
+      - homepage.group=AI - Image & Media
+      - homepage.name=Waterland Studio
+      - homepage.icon=mdi-watercolor
+      - homepage.description=Watercolour plate + reveal animation renderer (irv-ml1, A6000)
+      - homepage.href=http://10.100.79.3:${WLS_PORT:-8410}/
+      - homepage.siteMonitor=http://10.100.79.3:${WLS_PORT:-8410}/api/health
+
+volumes:
+  waterland_studio_data: {}
+  waterland_studio_kernels: {}
+
+networks:
+  tnet:
+    name: traefik-net
+    external: true