vh
a2b5b58eee
Replaces a bare nohup on irv-ml1:8410 that would not have survived a reboot, handed over by waterland-dev. Tracks vh/waterland @ main (PR #4 merged; main HEAD is exactly the pinned 8025366). Build context is a checkout at /opt/waterland-studio/src, deliberately OUTSIDE the compose dir — deploy-stack.sh rsyncs stacks/<stack>/ with --delete and would otherwise eat it. The Dockerfile is passed out-of-context. Three landmines, all measured: 1. Both uv extras are load-bearing at build AND run. jobs.py shells the renderer out as a literal with no --extra flags, so uv would re-sync at runtime and prune cupy — silently dropping to the numpy path at ~21x wall time. UV_NO_SYNC pins it; UV_OFFLINE makes any failure loud instead of quietly slow. 2. cupy needs CUDA HEADERS for its NVRTC compile, not just the driver and the wheel's runtime libs. The host has a system CUDA toolkit so the nohup process found them by accident; a slim image does not, and every render died 1.7s in with 'Failed to find CUDA headers' printed through argparse's usage banner — which reads like a CLI bug, not a missing toolkit. Fixed with cupy-cuda12x[ctk] (hundreds of MB, vs ~6 GB for a -devel base). 3. The A6000 is host device 1 but container device 0, since compose exposes exactly one GPU. CUDA_VISIBLE_DEVICES_TARGET=0 inside; copying the host's value selects a device that does not exist. /root/.cupy is a volume because the NVRTC compile costs ~17s: verified at 23.3s cold vs 6.1s warm, and re-verified across a restart (23.2s on a fresh cache volume, 6.0s once populated). Warm 256^2+anim beats the 7.4s recorded against bare metal, so containerising cost nothing. Job store seeded with the 4 jobs from the displaced instance. Serial by design (one replica, one card) and unauthenticated, so it stays LAN/WireGuard-only.
97 lines
4.3 KiB
YAML
97 lines
4.3 KiB
YAML
# waterland-studio — watercolour render service on irv-ml1, port 8410.
|
|
#
|
|
# FastAPI + vanilla-JS SPA fronting the waterland CLI: upload an image, get a
|
|
# watercolour plate and a painted-in reveal animation. Every job shells out to
|
|
# the CLI, which runs a fluid simulation on the A6000.
|
|
#
|
|
# Handed over by waterland-dev 2026-08-19, replacing a bare `nohup` that would
|
|
# not have survived a reboot.
|
|
#
|
|
# ⚠️ THE BUILD CONTEXT LIVES OUTSIDE THIS DIRECTORY, DELIBERATELY.
|
|
# /opt/waterland-studio/src is a checkout of vh/waterland @ main.
|
|
# `deploy-stack.sh` rsyncs this stack dir with --delete, so a checkout kept
|
|
# in here would be destroyed on the next deploy. Refresh it with ./update.sh.
|
|
#
|
|
# ⚠️ SERIAL BY DESIGN — ONE REPLICA, ONE CARD. A render is 20-45s of near-full
|
|
# GPU and the app runs a single worker thread. Two of these on the same A6000
|
|
# would OOM or thrash. Throughput is a conversation about hardware, not about
|
|
# replica count (waterland-dev, explicitly).
|
|
#
|
|
# ⚠️ NO AUTHENTICATION, AND IT ACCEPTS ARBITRARY FILE UPLOADS. It must stay
|
|
# inside the LAN / WireGuard boundary. Do NOT paper over this by putting it
|
|
# behind a proxy with a password — waterland-dev has offered to add a real
|
|
# auth layer if it ever needs wider reach. Ask, don't improvise.
|
|
|
|
name: waterland-studio
|
|
|
|
services:
|
|
waterland-studio:
|
|
build:
|
|
# Absolute paths: the context is the source checkout, the Dockerfile is
|
|
# this version-controlled one, and the two live in different trees.
|
|
context: /opt/waterland-studio/src
|
|
dockerfile: /opt/docker/compose/waterland-studio/Dockerfile
|
|
image: waterland-studio:local
|
|
container_name: ${WLS_CONTAINER:-waterland-studio}
|
|
restart: unless-stopped
|
|
ports:
|
|
- "${WLS_PORT:-8410}:8410"
|
|
volumes:
|
|
# Job store: uploaded sources plus rendered plates and animations.
|
|
# ~12 MB per job with an animation; the app self-evicts at 40 retained
|
|
# jobs (RETAIN in studio/jobs.py), so steady state is bounded ~500 MB.
|
|
# Scratch output, not source-of-truth — losing it costs a re-render.
|
|
- waterland_studio_data:/data
|
|
# cupy JIT kernel cache. Not optional for good behaviour: cupy compiles
|
|
# its kernels through NVRTC on first use, and measured on this host that
|
|
# cold compile costs ~17s — the first 256^2 render after a fresh
|
|
# container took 23.3s against 6.1s warm. Without this volume every
|
|
# restart makes the next user wait 4x, and it looks like the service is
|
|
# slow rather than warming up.
|
|
- waterland_studio_kernels:/root/.cupy
|
|
environment:
|
|
- WATERLAND_STUDIO_DATA=/data
|
|
- WATERLAND_STUDIO_REPO=/app
|
|
- WATERLAND_STUDIO_BACKEND=${WLS_BACKEND:-cupy}
|
|
# 0, not 1 — see the Dockerfile. Exactly one GPU is exposed below, so
|
|
# inside this container the A6000 is index 0 under PCI_BUS_ID ordering.
|
|
- CUDA_VISIBLE_DEVICES_TARGET=${WLS_CUDA_TARGET:-0}
|
|
deploy:
|
|
resources:
|
|
reservations:
|
|
devices:
|
|
- driver: nvidia
|
|
# "1" is the A6000 in DOCKER's device ordering, matching the
|
|
# comfyui stack on this host. Device 0 is the 3090, which hosts
|
|
# the TTS zoo and must not be touched.
|
|
device_ids: ["${WLS_GPU_ID:-1}"]
|
|
capabilities: [gpu]
|
|
healthcheck:
|
|
# /api/health touches no GPU and is safe to poll. The interval is
|
|
# deliberately loose: a render holds the GPU for 20-45s and the app's own
|
|
# job timeout is 480s, so an aggressive probe would be measuring queue
|
|
# depth rather than liveness.
|
|
test: ["CMD-SHELL", "python -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8410/api/health', timeout=5).status==200 else 1)\""]
|
|
interval: 60s
|
|
timeout: 10s
|
|
retries: 3
|
|
start_period: 30s
|
|
networks:
|
|
- tnet
|
|
labels:
|
|
- homepage.group=AI - Image & Media
|
|
- homepage.name=Waterland Studio
|
|
- homepage.icon=mdi-watercolor
|
|
- homepage.description=Watercolour plate + reveal animation renderer (irv-ml1, A6000)
|
|
- homepage.href=http://10.100.79.3:${WLS_PORT:-8410}/
|
|
- homepage.siteMonitor=http://10.100.79.3:${WLS_PORT:-8410}/api/health
|
|
|
|
volumes:
|
|
waterland_studio_data: {}
|
|
waterland_studio_kernels: {}
|
|
|
|
networks:
|
|
tnet:
|
|
name: traefik-net
|
|
external: true
|