vh
a2b5b58eee
Replaces a bare nohup on irv-ml1:8410 that would not have survived a reboot, handed over by waterland-dev. Tracks vh/waterland @ main (PR #4 merged; main HEAD is exactly the pinned 8025366). Build context is a checkout at /opt/waterland-studio/src, deliberately OUTSIDE the compose dir — deploy-stack.sh rsyncs stacks/<stack>/ with --delete and would otherwise eat it. The Dockerfile is passed out-of-context. Three landmines, all measured: 1. Both uv extras are load-bearing at build AND run. jobs.py shells the renderer out as a literal with no --extra flags, so uv would re-sync at runtime and prune cupy — silently dropping to the numpy path at ~21x wall time. UV_NO_SYNC pins it; UV_OFFLINE makes any failure loud instead of quietly slow. 2. cupy needs CUDA HEADERS for its NVRTC compile, not just the driver and the wheel's runtime libs. The host has a system CUDA toolkit so the nohup process found them by accident; a slim image does not, and every render died 1.7s in with 'Failed to find CUDA headers' printed through argparse's usage banner — which reads like a CLI bug, not a missing toolkit. Fixed with cupy-cuda12x[ctk] (hundreds of MB, vs ~6 GB for a -devel base). 3. The A6000 is host device 1 but container device 0, since compose exposes exactly one GPU. CUDA_VISIBLE_DEVICES_TARGET=0 inside; copying the host's value selects a device that does not exist. /root/.cupy is a volume because the NVRTC compile costs ~17s: verified at 23.3s cold vs 6.1s warm, and re-verified across a restart (23.2s on a fresh cache volume, 6.0s once populated). Warm 256^2+anim beats the 7.4s recorded against bare metal, so containerising cost nothing. Job store seeded with the 4 jobs from the displaced instance. Serial by design (one replica, one card) and unauthenticated, so it stays LAN/WireGuard-only.
waterland-studio — watercolour render service (irv-ml1)
FastAPI + vanilla-JS SPA fronting the waterland CLI: upload an image, get a
watercolour plate and a painted-in reveal animation. Every job shells out to
the CLI, which runs a fluid simulation on the A6000.
- Host: irv-ml1 (10.100.79.3, WireGuard-only) · Port: 8410
- URL: http://10.100.79.3:8410/ · Health:
GET /api/health - Source:
gitea.phasefinal.com/vh/waterland, trackingmain
Handed over by waterland-dev on 2026-08-19, replacing a bare nohup that
would not have survived a reboot.
Layout — the build context is deliberately outside this directory
| path | what |
|---|---|
/opt/waterland-studio/src |
checkout of vh/waterland @ main — the build context |
/opt/docker/compose/waterland-studio/ |
compose.yaml, Dockerfile, update.sh, .env |
volume waterland-studio_waterland_studio_data |
job store (/data) |
volume waterland-studio_waterland_studio_kernels |
cupy JIT cache (/root/.cupy) |
The checkout must NOT live under the compose directory. deploy-stack.sh
rsyncs stacks/<stack>/ with --delete, so a checkout kept beside
compose.yaml would be destroyed by the next deploy of this stack. The
Dockerfile is passed out-of-context to keep both trees clean.
Refresh source + rebuild:
ssh infra-ops@10.100.79.3 /opt/docker/compose/waterland-studio/update.sh
Three landmines, all of them measured rather than guessed
1. Both uv extras are load-bearing, at build AND at run. gpu carries
cupy-cuda12x; a bare uv sync prunes it and the renderer silently drops to
the numpy path at ~21x the wall time — it does not error, it just gets slow.
Worse, studio/jobs.py shells the renderer out as a literal uv run waterland
with no --extra flags, so uv would re-sync at runtime and prune cupy right
back out. UV_NO_SYNC=1 stops that; UV_OFFLINE=1 means that if the pin ever
stops working the job fails loudly instead of quietly rebuilding a slower
environment.
2. cupy needs CUDA headers, which the host never had to declare. cupy
compiles kernels at runtime through NVRTC, which needs toolkit headers — not
just the driver and the runtime libs bundled in the wheel. On irv-ml1 a CUDA
toolkit is installed system-wide, so the bare nohup process found them by
accident; a slim image has none. Every render died 1.7s in with
RuntimeError: Failed to find CUDA headers.
printed through argparse's usage banner, which makes it read like a CLI
argument bug rather than a missing toolkit — that misdirection is the reason
this is written down. Fixed with uv pip install "cupy-cuda12x[ctk]", which
pulls the headers as wheels: a few hundred MB against ~6 GB for a -devel
base image. It runs after uv sync, because sync prunes what it does not
know about.
3. The GPU index inside the container is not the host's. The app pins
CUDA_DEVICE_ORDER=PCI_BUS_ID and selects CUDA_VISIBLE_DEVICES_TARGET
(default 1, correct on the host). Compose exposes exactly one GPU
(device_ids: ["1"], the A6000 in Docker's ordering), so inside the
container that card is index 0 — hence CUDA_VISIBLE_DEVICES_TARGET=0.
Copying the host's value selects a device that does not exist. Device 0 on the
host is the 3090, which hosts the TTS zoo and must not be touched.
Performance, measured on this host
| job | wall |
|---|---|
| 256², animation, cold container | 23.3 s |
| 256², animation, warm | 6.1 s |
256², plate only (--codec none) |
3.9 s |
| 512², plate only | 6.4 s |
Warm numbers beat the 7.4 s at 256² recorded against the bare-metal process, so
containerising cost nothing. The cold-vs-warm gap is cupy's NVRTC compile,
which is why /root/.cupy is a volume: verified by recreating the container
(fresh cache → 23.2 s first render) and then restarting it (populated cache →
6.0 s). Without that volume every restart makes the next user wait 4x and the
service merely looks slow.
Operational constraints — from waterland-dev, not inferred
- Serial by design. One replica, one card. A render is 20–45 s of near-full GPU and the app runs a single worker thread. Two of these on the same A6000 would OOM or thrash. Throughput is a conversation about hardware, not replica count.
- No authentication, and it accepts arbitrary file uploads. It must stay inside the LAN / WireGuard boundary. Do not paper over this with a proxy password — waterland-dev has offered to add a real auth layer if it ever needs wider reach. Ask.
- Job store is scratch output, not source-of-truth: ~12 MB per animated job,
self-evicting at 40 retained jobs (
RETAINinstudio/jobs.py), so steady state is bounded around 500 MB. The 4 jobs from the bare-metal instance were copied in at cutover. - Internal render timeout is 480 s, which is why the healthcheck interval is loose — an aggressive probe would measure queue depth rather than liveness.
⚠️ update.sh needs a credential this host does not have
The repo is not anonymously readable — an unauthenticated clone 403s. The
initial checkout was made with the operator's vh site-admin token passed
inline and not persisted: the on-disk remote is the plain URL and
.git/config holds no token (verified). Consequently git fetch in
update.sh will fail until the host has a credential of its own.
claude-bot 404s on vh/waterland, so it currently lacks read access. The
right fix is a read-only deploy token for this host, or granting claude-bot
read on the repo — not writing the site-admin token to disk on a GPU box.
Raised with waterland-dev; until then, re-run the authenticated clone by hand
to update.