vh
e6907819b0
One update.sh run on irv-ml1 carried both open upstream PRs, per the operator's green-light on the job-store fix: - #5 (464dfc2) declares cupy-cuda12x[ctk] on the gpu extra and takes uv out of the render path (sys.executable -m waterland.cli), retiring the runtime prune trap at the source. - #6 (b72425b) rehydrates the job index from the data volume at startup, fixing the unbounded store growth reported from this side. Verified after the update rather than assumed: healthy on backend cupy; /api/jobs went 1 -> 16 against 16 directories on disk, so API and volume agree for the first time; nothing wrongly reclaimed, correct since 16 is under RETAIN=40 and adoption only makes them visible; a real 256^2 plate render completes warm, so the kernel-cache volume survived the image swap. A subsequent render took both counts to 17. The image keeps its explicit [ctk] install and UV_NO_SYNC/UV_OFFLINE pins even though both are now redundant. The header requirement is a property of this slim base, not of the upstream extra, and the cost is measured rather than assumed: uv sync satisfies it first, so the line reports "Audited 1 package" and adds 0.3s to the build. The env pins are now cheap defence-in-depth against any future path that re-enters uv. Docs corrected in place: the README's upstream-finding section is now a resolved-finding record, and the two "bounded ~500 MB" claims say which commit made that bound hold across restarts rather than only within a process. Comment-side changes pushed to the live compose dir; no restart was needed for them.
177 lines
9.4 KiB
Markdown
177 lines
9.4 KiB
Markdown
# `[2026-08-19]` waterland studio containerised on irv-ml1 — three landmines, all measured
|
||
|
||
Handover from `waterland-dev` over althing (thread `01M0CDRGEZWAJCEJXXMQWXV80F`):
|
||
a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare
|
||
`nohup` (PID 1283383) that would not survive a reboot. Now
|
||
`stacks/waterland-studio/`, `restart: unless-stopped`, healthy on
|
||
irv-ml1:8410. Commits `a2b5b58`, `8189076`.
|
||
|
||
Tracking `main` per operator: PR #4 merged and `main` HEAD was exactly the
|
||
pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway.
|
||
|
||
**Now deployed at `b72425b` (2026-08-19).** The container sat on `8025366` for
|
||
a few hours after PR #5 (`464dfc2`) landed — deliberately, since the image's
|
||
own guards already neutralised both landmines and the project was in
|
||
wind-down. PR #6 (the job-store rehydrate, operator-green-lit) was the rebuild
|
||
with a real reason behind it, and one `update.sh` run carried both. Verified
|
||
end to end after the update: healthy, `backend: cupy`, and a real 256² plate
|
||
render completes warm — the kernel-cache volume survived the image swap.
|
||
|
||
## Build context lives OUTSIDE the compose dir — on purpose
|
||
|
||
`/opt/waterland-studio/src` is the checkout; the Dockerfile is passed
|
||
out-of-context from `/opt/docker/compose/waterland-studio/`. **`deploy-stack.sh`
|
||
rsyncs `stacks/<stack>/` with `--delete`**, so a checkout kept beside
|
||
`compose.yaml` would be destroyed by the next deploy of this stack. `update.sh`
|
||
refreshes source → rebuild → recreate → health, and is verified end to end.
|
||
|
||
## Landmine 1 — both uv extras are load-bearing at BUILD *and* RUN
|
||
|
||
`gpu` carries `cupy-cuda12x`; a bare `uv sync` prunes it and the renderer
|
||
silently drops to the numpy path at ~21x wall time — it does not error, it
|
||
just gets slow. waterland-dev warned about the build side.
|
||
|
||
The runtime side is worse and was not in the handover: **`studio/jobs.py`
|
||
shells the renderer out as a literal `uv run waterland ...` with no `--extra`
|
||
flags** (`cwd=WATERLAND_STUDIO_REPO`). Left alone, uv re-syncs the project
|
||
mid-job to its default extras and prunes cupy back out from under a correctly
|
||
built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the
|
||
pin ever stops holding the job fails **loudly** instead of quietly rebuilding a
|
||
slower environment.
|
||
|
||
**Fixed upstream in `464dfc2`:** the server now spawns
|
||
`sys.executable -m waterland.cli` directly — no resolver in the render path at
|
||
all. **The pins stay anyway.** They cost nothing and are now defence-in-depth:
|
||
if any future code path re-enters `uv` inside the container, the job fails
|
||
loudly instead of quietly dropping to the numpy backend. `uv` itself must stay
|
||
in the image regardless — it performs the build-time `uv sync` /
|
||
`uv pip install`, and this is a single-stage build.
|
||
|
||
## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare
|
||
|
||
Every render died 1.7s in with:
|
||
|
||
```
|
||
RuntimeError: Failed to find CUDA headers.
|
||
```
|
||
|
||
printed **through argparse's usage banner**, which makes it read like a CLI
|
||
argument bug rather than a missing toolkit. That misdirection is the reason
|
||
this is written down.
|
||
|
||
cupy compiles kernels at runtime through NVRTC, which needs the toolkit
|
||
**headers** — not just the driver and the runtime libs bundled in the
|
||
`cupy-cuda12x` wheel. irv-ml1 has a CUDA toolkit installed system-wide, so the
|
||
bare `nohup` process found them **by accident**; a slim image has none.
|
||
|
||
Fixed with `uv pip install "cupy-cuda12x[ctk]"` — headers as wheels, a few
|
||
hundred MB against ~6 GB for a `-devel` base image. It runs **after**
|
||
`uv sync`, because sync prunes what it does not know about.
|
||
|
||
Reported upstream: it is an undeclared runtime dependency of the `gpu` extra,
|
||
and anyone running this without a system toolkit hits it. **Declared upstream
|
||
in `464dfc2`** (`gpu` is now `cupy-cuda12x[ctk]>=13`). **The explicit install
|
||
stays in the Dockerfile**: the header requirement is a property of *this*
|
||
image — a slim base with no system CUDA toolkit — so it belongs in the file
|
||
that creates the problem, not inherited from an extra two repos away. It also
|
||
survives any future restructuring of the `gpu` extra. Cost of keeping it is
|
||
now measured, not assumed: since `uv sync` satisfies it first, the line
|
||
reports `Audited 1 package in 49ms` and adds **0.3s** to the build. A no-op
|
||
that documents a non-obvious requirement is worth 0.3s. (waterland-dev
|
||
independently agreed they would keep it too.)
|
||
|
||
## Landmine 3 — the GPU index inside the container is not the host's
|
||
|
||
The app pins `CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects
|
||
`CUDA_VISIBLE_DEVICES_TARGET` (default `1`, correct on the host, where
|
||
`nvidia-smi` shows A6000 at 1). Compose exposes **exactly one** GPU
|
||
(`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the
|
||
container that card is index **0** ⇒ `CUDA_VISIBLE_DEVICES_TARGET=0`. Copying
|
||
the host's value selects a device that does not exist. Host device 0 is the
|
||
3090, which carries the TTS zoo and must not be touched.
|
||
|
||
## Cold start is ~17s of NVRTC compile → `/root/.cupy` is a volume
|
||
|
||
| job | wall |
|
||
|---|---|
|
||
| 256² + anim, cold container | 23.3 s |
|
||
| 256² + anim, warm | **6.1 s** |
|
||
| 256² plate only (`--codec none`) | 3.9 s |
|
||
| 512² plate only | 6.4 s |
|
||
|
||
Warm beats the **7.4 s** recorded against the bare-metal process, so
|
||
containerising cost nothing. Verified the cache volume properly: recreate
|
||
(fresh cache → 23.2 s first render) then restart (populated → 6.0 s). Without
|
||
it every restart makes the next user wait 4x and the service merely *looks*
|
||
slow.
|
||
|
||
## Upstream finding — the on-disk job store grows without bound
|
||
|
||
`JobStore._jobs` is a plain dict and **nothing scans `WATERLAND_STUDIO_DATA` at
|
||
startup**. Consequences:
|
||
|
||
1. After a restart `/api/jobs` lists only jobs created since — cosmetic, and
|
||
how this was spotted: the API reported **1 job** while the volume held all
|
||
**16 directories, 60.6 MB**. Not data loss.
|
||
2. The real one: `RETAIN = 40` eviction only ever iterates the in-memory dict,
|
||
so directories orphaned by a restart are **never reclaimed**. The
|
||
handover's "bounded around 500 MB" holds within a single process lifetime;
|
||
across restarts the store grows monotonically at ~12 MB per animated job.
|
||
|
||
Reported to waterland-dev with evidence; **not patched from the infra side** —
|
||
it is their code. Prune the volume by hand if it bites first.
|
||
|
||
**waterland-dev confirmed it (2026-08-19)** — their "bounded ~500 MB" handover
|
||
claim holds within one process lifetime and nowhere else, which on a
|
||
`restart: unless-stopped` service is the wrong lifetime to have bounded. They
|
||
have **surfaced a startup-rehydrate fix to the operator** rather than opening a
|
||
third PR during wind-down. **Operator green-lit it; PR #6 merged as `b72425b`
|
||
and is DEPLOYED (2026-08-19).**
|
||
|
||
Startup rehydrate, as recommended — and waterland-dev deliberately went
|
||
further than the framing I sent them. I had said a directory the scan cannot
|
||
parse "just does not enter the index"; they made the opposite call, because a
|
||
directory that never enters the index is exactly the one that never gets
|
||
reclaimed. **That is the sharper reading and it is the reason the fix works on
|
||
this volume at all** — the 16 pre-existing dirs have no sidecar. Their
|
||
adoption ladder: sidecar → restored verbatim; no sidecar → adopted with
|
||
dimensions recovered from the PNG IHDR (24-byte read, not a decode); corrupt
|
||
sidecar → degrades to inference, no startup crash; **neither source nor
|
||
sidecar → skipped on purpose**, since adopting it would turn eviction into a
|
||
delete-arbitrary-directories primitive pointed at this volume. Sidecar writes
|
||
go through `os.replace`, and `job.json` is excluded from `ARTIFACTS` so it is
|
||
unreachable via the artifact route.
|
||
|
||
They also closed a second leak I never saw, because it needs a restart
|
||
*mid-render* to surface: a job left `running`/`queued` in its sidecar is
|
||
non-terminal forever, and eviction skips non-terminal jobs — so it is a
|
||
phantom that is never reclaimed and `queue_depth` over-reports for the life of
|
||
the process. Adoption now marks those `failed`.
|
||
|
||
**Verified on this host after the update:** `/api/jobs` went **1 → 16** while
|
||
the volume stayed at 16 dirs / 61 MB — disk and API agree for the first time.
|
||
Nothing was reclaimed, correctly: 16 is under `RETAIN=40`, so adoption only
|
||
made them visible. A subsequent real render took both to 17. From here the
|
||
store is bounded **across** restarts, not merely within a process.
|
||
|
||
## Access
|
||
|
||
Repo is not anonymously readable (a bare clone 403s). Operator granted
|
||
**`claude-bot` read on `vh/waterland`** — verified `admin: False, push: False,
|
||
pull: True`. Token on irv-ml1 at
|
||
`/root/.config/waterland-studio/git-credentials`, `0600` root-owned, wired as a
|
||
**repo-scoped** credential helper; `.git/config` carries no token (verified),
|
||
so the remote stays clean in any diff or backup. The operator's `vh`
|
||
site-admin token was used only for the initial clone and the grant itself and
|
||
was **never written to disk on that host** — a site-admin credential on a GPU
|
||
box is a blast radius nobody needs for a read-only fetch.
|
||
|
||
## Constraints honoured as stated (not inferred)
|
||
|
||
- **Serial by design — one replica, one card.** A render is 20–45s of near-full
|
||
GPU with a single worker thread. Two on the same A6000 would OOM or thrash.
|
||
Throughput is a hardware conversation, not a replica-count one.
|
||
- **No authentication, arbitrary file uploads** ⇒ stays inside the
|
||
LAN/WireGuard boundary. Do **not** paper over it with a proxy password;
|
||
waterland-dev offered to add a real auth layer if wider reach is ever needed.
|