# waterland-studio — watercolour render service (irv-ml1)

FastAPI + vanilla-JS SPA fronting the `waterland` CLI: upload an image, get a
watercolour plate and a painted-in reveal animation. Every job shells out to
the CLI, which runs a fluid simulation on the **A6000**.

- **Host:** irv-ml1 (10.100.79.3, WireGuard-only) · **Port:** 8410
- **URL:** http://10.100.79.3:8410/ · **Health:** `GET /api/health`
- **Source:** `gitea.phasefinal.com/vh/waterland`, tracking **`main`**

Handed over by `waterland-dev` on 2026-08-19, replacing a bare `nohup` that
would not have survived a reboot.

## Layout — the build context is deliberately outside this directory

| path | what |
|---|---|
| `/opt/waterland-studio/src` | checkout of `vh/waterland` @ `main` — the build context |
| `/opt/docker/compose/waterland-studio/` | `compose.yaml`, `Dockerfile`, `update.sh`, `.env` |
| volume `waterland-studio_waterland_studio_data` | job store (`/data`) |
| volume `waterland-studio_waterland_studio_kernels` | cupy JIT cache (`/root/.cupy`) |

**The checkout must NOT live under the compose directory.** `deploy-stack.sh`
rsyncs `stacks/<stack>/` with `--delete`, so a checkout kept beside
`compose.yaml` would be destroyed by the next deploy of this stack. The
Dockerfile is passed out-of-context to keep both trees clean.

Refresh source + rebuild:

```bash
ssh infra-ops@10.100.79.3 /opt/docker/compose/waterland-studio/update.sh
```

## Three landmines, all of them measured rather than guessed

**1. Both uv extras are load-bearing, at build AND at run.** `gpu` carries
`cupy-cuda12x`; a bare `uv sync` prunes it and the renderer silently drops to
the numpy path at ~21x the wall time — it does not error, it just gets slow.
Worse, `studio/jobs.py` shells the renderer out as a literal `uv run waterland`
with no `--extra` flags, so uv would re-sync at *runtime* and prune cupy right
back out. `UV_NO_SYNC=1` stops that; `UV_OFFLINE=1` means that if the pin ever
stops working the job fails loudly instead of quietly rebuilding a slower
environment.

**2. cupy needs CUDA *headers*, which the host never had to declare.** cupy
compiles kernels at runtime through NVRTC, which needs toolkit headers — not
just the driver and the runtime libs bundled in the wheel. On irv-ml1 a CUDA
toolkit is installed system-wide, so the bare `nohup` process found them by
accident; a slim image has none. Every render died 1.7s in with

```
RuntimeError: Failed to find CUDA headers.
```

printed *through argparse's usage banner*, which makes it read like a CLI
argument bug rather than a missing toolkit — that misdirection is the reason
this is written down. Fixed with `uv pip install "cupy-cuda12x[ctk]"`, which
pulls the headers as wheels: a few hundred MB against ~6 GB for a `-devel`
base image. It runs *after* `uv sync`, because sync prunes what it does not
know about.

**3. The GPU index inside the container is not the host's.** The app pins
`CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects `CUDA_VISIBLE_DEVICES_TARGET`
(default `1`, correct on the host). Compose exposes exactly one GPU
(`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the
container that card is index **0** — hence `CUDA_VISIBLE_DEVICES_TARGET=0`.
Copying the host's value selects a device that does not exist. Device 0 on the
host is the 3090, which hosts the TTS zoo and must not be touched.

## Performance, measured on this host

| job | wall |
|---|---|
| 256², animation, **cold container** | 23.3 s |
| 256², animation, warm | 6.1 s |
| 256², plate only (`--codec none`) | 3.9 s |
| 512², plate only | 6.4 s |

Warm numbers beat the 7.4 s at 256² recorded against the bare-metal process, so
containerising cost nothing. The cold-vs-warm gap is **cupy's NVRTC compile**,
which is why `/root/.cupy` is a volume: verified by recreating the container
(fresh cache → 23.2 s first render) and then restarting it (populated cache →
6.0 s). Without that volume every restart makes the next user wait 4x and the
service merely looks slow.

## Operational constraints — from waterland-dev, not inferred

- **Serial by design. One replica, one card.** A render is 20–45 s of near-full
  GPU and the app runs a single worker thread. Two of these on the same A6000
  would OOM or thrash. Throughput is a conversation about hardware, not replica
  count.
- **No authentication, and it accepts arbitrary file uploads.** It must stay
  inside the LAN / WireGuard boundary. Do **not** paper over this with a proxy
  password — waterland-dev has offered to add a real auth layer if it ever
  needs wider reach. Ask.
- Job store is scratch output, not source-of-truth: ~12 MB per animated job,
  self-evicting at 40 retained jobs (`RETAIN` in `studio/jobs.py`), so steady
  state is bounded around 500 MB. As of `b72425b` the index is rehydrated from
  disk at startup, so that bound holds **across restarts** and not merely
  within one process — see the resolved-finding section below for why that
  distinction mattered. The 4 jobs from the bare-metal instance were copied in
  at cutover.
- Internal render timeout is 480 s, which is why the healthcheck interval is
  loose — an aggressive probe would measure queue depth rather than liveness.

## Source credential — `claude-bot`, read-only

The repo is not anonymously readable (an unauthenticated clone 403s), so the
host needs a credential to fetch. Operator granted **`claude-bot` read on
`vh/waterland`** on 2026-08-19; verified scoped correctly:

```
admin: False | push: False | pull: True
```

The token lives on irv-ml1 at `/root/.config/waterland-studio/git-credentials`,
mode `0600`, root-owned, and is wired as a **repo-scoped** credential helper —
not a global one:

```bash
git -C /opt/waterland-studio/src config credential.helper \
    'store --file=/root/.config/waterland-studio/git-credentials'
```

`.git/config` itself holds no token (verified), so the remote URL stays clean
in any diff, log or backup of the checkout.

The operator's `vh` site-admin token was used **only** for the initial clone
and the grant itself, passed inline and never written to disk on this host.
Do not reintroduce it: a site-admin credential on a GPU box is a blast radius
nobody needs for a read-only fetch.

## ✅ Resolved upstream: the on-disk job store used to grow without bound

**Fixed in `b72425b` and deployed here.** Kept on the record because the
symptom is easy to misread and the fix has a property worth knowing about.

The job index used to be **memory-only** — `JobStore._jobs` was a plain dict
and nothing scanned `WATERLAND_STUDIO_DATA` at startup. Two consequences:

1. After any restart the UI and `/api/jobs` listed only jobs created *since*
   that restart, even though every earlier job's files were still on the
   volume. Cosmetic, not data loss — and it is how this was spotted: the API
   reported 1 job while the volume held all 16 directories (60.6 MB).
2. The real one: `RETAIN = 40` self-eviction only ever walked the in-memory
   dict, so directories orphaned by a restart were **never** reclaimed. The
   handover's "bounded around 500 MB" held within a single process lifetime;
   across restarts the store grew monotonically at ~12 MB per animated job.

Upstream now **rehydrates the index at startup** by scanning the data dir, and
deliberately *adopts* directories it cannot fully parse rather than skipping
them — a directory that never enters the index is precisely the one eviction
can never reclaim. Sidecar present → restored; no sidecar → adopted with
dimensions read from the PNG IHDR; corrupt sidecar → inferred, no startup
crash. **Neither source nor sidecar → skipped on purpose**, because adopting
those would make eviction a delete-arbitrary-directories primitive pointed at
this volume. Jobs left `running`/`queued` by a mid-render restart are marked
`failed`, so they stop being unreclaimable phantoms that inflate `queue_depth`.

Verified here after the update: `/api/jobs` went **1 → 16** against 16
directories on disk, with nothing reclaimed — correct, since 16 is under
`RETAIN=40`; adoption only made them visible. The bound now holds across
restarts, so no hand-pruning is needed.

**Don't chase the one `failed` job in that list.** Adoption marks a job
`failed` on exactly two conditions: a sidecar saying `running`/`queued`, or a
directory with no `plate.png`. The renders that died 1.7 s in on the missing
CUDA headers (before that fix) wrote a source and never a plate, so they land
in the second branch. It is adopted history being labelled honestly, not a
current fault.