esh-pfi-infrastructure/stacks/waterland-studio

waterland-studio — watercolour render service (irv-ml1)

FastAPI + vanilla-JS SPA fronting the waterland CLI: upload an image, get a watercolour plate and a painted-in reveal animation. Every job shells out to the CLI, which runs a fluid simulation on the A6000.

• Host: irv-ml1 (10.100.79.3, WireGuard-only) · Port: 8410
• URL: http://10.100.79.3:8410/ · Health: GET /api/health
• Source: gitea.phasefinal.com/vh/waterland, tracking main

Handed over by waterland-dev on 2026-08-19, replacing a bare nohup that would not have survived a reboot.

Layout — the build context is deliberately outside this directory

path	what
/opt/waterland-studio/src	checkout of vh/waterland @ main — the build context
/opt/docker/compose/waterland-studio/	compose.yaml, Dockerfile, update.sh, .env
volume waterland-studio_waterland_studio_data	job store (/data)
volume waterland-studio_waterland_studio_kernels	cupy JIT cache (/root/.cupy)

The checkout must NOT live under the compose directory. deploy-stack.sh rsyncs stacks/<stack>/ with --delete, so a checkout kept beside compose.yaml would be destroyed by the next deploy of this stack. The Dockerfile is passed out-of-context to keep both trees clean.

Refresh source + rebuild:

ssh infra-ops@10.100.79.3 /opt/docker/compose/waterland-studio/update.sh

Three landmines, all of them measured rather than guessed

1. Both uv extras are load-bearing, at build AND at run. gpu carries cupy-cuda12x; a bare uv sync prunes it and the renderer silently drops to the numpy path at ~21x the wall time — it does not error, it just gets slow. Worse, studio/jobs.py shells the renderer out as a literal uv run waterland with no --extra flags, so uv would re-sync at runtime and prune cupy right back out. UV_NO_SYNC=1 stops that; UV_OFFLINE=1 means that if the pin ever stops working the job fails loudly instead of quietly rebuilding a slower environment.

2. cupy needs CUDA headers, which the host never had to declare. cupy compiles kernels at runtime through NVRTC, which needs toolkit headers — not just the driver and the runtime libs bundled in the wheel. On irv-ml1 a CUDA toolkit is installed system-wide, so the bare nohup process found them by accident; a slim image has none. Every render died 1.7s in with

RuntimeError: Failed to find CUDA headers.

printed through argparse's usage banner, which makes it read like a CLI argument bug rather than a missing toolkit — that misdirection is the reason this is written down. Fixed with uv pip install "cupy-cuda12x[ctk]", which pulls the headers as wheels: a few hundred MB against ~6 GB for a -devel base image. It runs after uv sync, because sync prunes what it does not know about.

3. The GPU index inside the container is not the host's. The app pins CUDA_DEVICE_ORDER=PCI_BUS_ID and selects CUDA_VISIBLE_DEVICES_TARGET (default 1, correct on the host). Compose exposes exactly one GPU (device_ids: ["1"], the A6000 in Docker's ordering), so inside the container that card is index 0 — hence CUDA_VISIBLE_DEVICES_TARGET=0. Copying the host's value selects a device that does not exist. Device 0 on the host is the 3090, which hosts the TTS zoo and must not be touched.

Performance, measured on this host

job	wall
256², animation, cold container	23.3 s
256², animation, warm	6.1 s
256², plate only (--codec none)	3.9 s
512², plate only	6.4 s

Warm numbers beat the 7.4 s at 256² recorded against the bare-metal process, so containerising cost nothing. The cold-vs-warm gap is cupy's NVRTC compile, which is why /root/.cupy is a volume: verified by recreating the container (fresh cache → 23.2 s first render) and then restarting it (populated cache → 6.0 s). Without that volume every restart makes the next user wait 4x and the service merely looks slow.

Operational constraints — from waterland-dev, not inferred

• Serial by design. One replica, one card. A render is 20–45 s of near-full GPU and the app runs a single worker thread. Two of these on the same A6000 would OOM or thrash. Throughput is a conversation about hardware, not replica count.
• No authentication, and it accepts arbitrary file uploads. It must stay inside the LAN / WireGuard boundary. Do not paper over this with a proxy password — waterland-dev has offered to add a real auth layer if it ever needs wider reach. Ask.
• Job store is scratch output, not source-of-truth: ~12 MB per animated job, self-evicting at 40 retained jobs (RETAIN in studio/jobs.py), so steady state is bounded around 500 MB. The 4 jobs from the bare-metal instance were copied in at cutover.
• Internal render timeout is 480 s, which is why the healthcheck interval is loose — an aggressive probe would measure queue depth rather than liveness.

Source credential — claude-bot, read-only

The repo is not anonymously readable (an unauthenticated clone 403s), so the host needs a credential to fetch. Operator granted claude-bot read on vh/waterland on 2026-08-19; verified scoped correctly:

admin: False | push: False | pull: True

The token lives on irv-ml1 at /root/.config/waterland-studio/git-credentials, mode 0600, root-owned, and is wired as a repo-scoped credential helper — not a global one:

git -C /opt/waterland-studio/src config credential.helper \
    'store --file=/root/.config/waterland-studio/git-credentials'

.git/config itself holds no token (verified), so the remote URL stays clean in any diff, log or backup of the checkout.

The operator's vh site-admin token was used only for the initial clone and the grant itself, passed inline and never written to disk on this host. Do not reintroduce it: a site-admin credential on a GPU box is a blast radius nobody needs for a read-only fetch.

⚠️ Upstream finding: the on-disk job store grows without bound

The job index is memory-only — JobStore._jobs is a plain dict, and nothing scans WATERLAND_STUDIO_DATA at startup. Two consequences:

1. After any restart the UI and /api/jobs list only jobs created since that restart, even though every earlier job's files are still on the volume. This is cosmetic and not data loss — verified at cutover: the API reported 1 job while the volume held all 16 directories (60.6 MB).
2. More importantly, the RETAIN = 40 self-eviction only ever sees in-memory jobs, so directories orphaned by a restart are never reclaimed. The handover's "bounded around 500 MB" holds within a single process lifetime; across restarts the store grows monotonically at ~12 MB per animated job.

Growth is slow, so this is not urgent, but it is unbounded and the volume will need occasional pruning until upstream rehydrates the index (or sweeps orphans) at startup. Reported to waterland-dev — it is their call, not a thing to patch from the infrastructure side.