esh-pfi-infrastructure/persistent-memory.d/2026-08-19-waterland-studio-containerised.md

[2026-08-19] waterland studio containerised on irv-ml1 — three landmines, all measured

Handover from waterland-dev over althing (thread 01M0CDRGEZWAJCEJXXMQWXV80F): a FastAPI + SPA GPU service fronting the waterland CLI, running as a bare nohup (PID 1283383) that would not survive a reboot. Now stacks/waterland-studio/, restart: unless-stopped, healthy on irv-ml1:8410. Commits a2b5b58, 8189076.

Tracking main per operator: PR #4 merged and main HEAD is exactly the pinned 8025366, so tracking-a-moving-ref and keeping-the-pin agreed anyway.

Build context lives OUTSIDE the compose dir — on purpose

/opt/waterland-studio/src is the checkout; the Dockerfile is passed out-of-context from /opt/docker/compose/waterland-studio/. deploy-stack.sh rsyncs stacks/<stack>/ with --delete, so a checkout kept beside compose.yaml would be destroyed by the next deploy of this stack. update.sh refreshes source → rebuild → recreate → health, and is verified end to end.

Landmine 1 — both uv extras are load-bearing at BUILD and RUN

gpu carries cupy-cuda12x; a bare uv sync prunes it and the renderer silently drops to the numpy path at ~21x wall time — it does not error, it just gets slow. waterland-dev warned about the build side.

The runtime side is worse and was not in the handover: studio/jobs.py shells the renderer out as a literal uv run waterland ... with no --extra flags (cwd=WATERLAND_STUDIO_REPO). Left alone, uv re-syncs the project mid-job to its default extras and prunes cupy back out from under a correctly built venv. Pinned with UV_NO_SYNC=1; UV_OFFLINE=1 alongside so that if the pin ever stops holding the job fails loudly instead of quietly rebuilding a slower environment.

Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare

Every render died 1.7s in with:

RuntimeError: Failed to find CUDA headers.

printed through argparse's usage banner, which makes it read like a CLI argument bug rather than a missing toolkit. That misdirection is the reason this is written down.

cupy compiles kernels at runtime through NVRTC, which needs the toolkit headers — not just the driver and the runtime libs bundled in the cupy-cuda12x wheel. irv-ml1 has a CUDA toolkit installed system-wide, so the bare nohup process found them by accident; a slim image has none.

Fixed with uv pip install "cupy-cuda12x[ctk]" — headers as wheels, a few hundred MB against ~6 GB for a -devel base image. It runs after uv sync, because sync prunes what it does not know about.

Reported upstream: it is an undeclared runtime dependency of the gpu extra, and anyone running this without a system toolkit hits it.

Landmine 3 — the GPU index inside the container is not the host's

The app pins CUDA_DEVICE_ORDER=PCI_BUS_ID and selects CUDA_VISIBLE_DEVICES_TARGET (default 1, correct on the host, where nvidia-smi shows A6000 at 1). Compose exposes exactly one GPU (device_ids: ["1"], the A6000 in Docker's ordering), so inside the container that card is index 0 ⇒ CUDA_VISIBLE_DEVICES_TARGET=0. Copying the host's value selects a device that does not exist. Host device 0 is the 3090, which carries the TTS zoo and must not be touched.

Cold start is ~17s of NVRTC compile → /root/.cupy is a volume

job	wall
256² + anim, cold container	23.3 s
256² + anim, warm	6.1 s
256² plate only (--codec none)	3.9 s
512² plate only	6.4 s

Warm beats the 7.4 s recorded against the bare-metal process, so containerising cost nothing. Verified the cache volume properly: recreate (fresh cache → 23.2 s first render) then restart (populated → 6.0 s). Without it every restart makes the next user wait 4x and the service merely looks slow.

Upstream finding — the on-disk job store grows without bound

JobStore._jobs is a plain dict and nothing scans WATERLAND_STUDIO_DATA at startup. Consequences:

1. After a restart /api/jobs lists only jobs created since — cosmetic, and how this was spotted: the API reported 1 job while the volume held all 16 directories, 60.6 MB. Not data loss.
2. The real one: RETAIN = 40 eviction only ever iterates the in-memory dict, so directories orphaned by a restart are never reclaimed. The handover's "bounded around 500 MB" holds within a single process lifetime; across restarts the store grows monotonically at ~12 MB per animated job.

Reported to waterland-dev with evidence; not patched from the infra side — it is their code. Prune the volume by hand if it bites first.

Access

Repo is not anonymously readable (a bare clone 403s). Operator granted claude-bot read on vh/waterland — verified admin: False, push: False, pull: True. Token on irv-ml1 at /root/.config/waterland-studio/git-credentials, 0600 root-owned, wired as a repo-scoped credential helper; .git/config carries no token (verified), so the remote stays clean in any diff or backup. The operator's vh site-admin token was used only for the initial clone and the grant itself and was never written to disk on that host — a site-admin credential on a GPU box is a blast radius nobody needs for a read-only fetch.

Constraints honoured as stated (not inferred)

• Serial by design — one replica, one card. A render is 20–45s of near-full GPU with a single worker thread. Two on the same A6000 would OOM or thrash. Throughput is a hardware conversation, not a replica-count one.
• No authentication, arbitrary file uploads ⇒ stays inside the LAN/WireGuard boundary. Do not paper over it with a proxy password; waterland-dev offered to add a real auth layer if wider reach is ever needed.