feat(waterland-studio): deploy b72425b — all three upstream findings fixed

One update.sh run on irv-ml1 carried both open upstream PRs, per the
operator's green-light on the job-store fix:

- #5 (464dfc2) declares cupy-cuda12x[ctk] on the gpu extra and takes uv
  out of the render path (sys.executable -m waterland.cli), retiring the
  runtime prune trap at the source.
- #6 (b72425b) rehydrates the job index from the data volume at startup,
  fixing the unbounded store growth reported from this side.

Verified after the update rather than assumed: healthy on backend cupy;
/api/jobs went 1 -> 16 against 16 directories on disk, so API and volume
agree for the first time; nothing wrongly reclaimed, correct since 16 is
under RETAIN=40 and adoption only makes them visible; a real 256^2 plate
render completes warm, so the kernel-cache volume survived the image swap.
A subsequent render took both counts to 17.

The image keeps its explicit [ctk] install and UV_NO_SYNC/UV_OFFLINE pins
even though both are now redundant. The header requirement is a property
of this slim base, not of the upstream extra, and the cost is measured
rather than assumed: uv sync satisfies it first, so the line reports
"Audited 1 package" and adds 0.3s to the build. The env pins are now
cheap defence-in-depth against any future path that re-enters uv.

Docs corrected in place: the README's upstream-finding section is now a
resolved-finding record, and the two "bounded ~500 MB" claims say which
commit made that bound hold across restarts rather than only within a
process. Comment-side changes pushed to the live compose dir; no restart
was needed for them.
This commit is contained in:
vh
2026-08-19 01:39:13 -07:00
parent a2b6bf409e
commit e6907819b0
4 changed files with 76 additions and 33 deletions
+33 -17
View File
@@ -95,8 +95,11 @@ service merely looks slow.
needs wider reach. Ask.
- Job store is scratch output, not source-of-truth: ~12 MB per animated job,
self-evicting at 40 retained jobs (`RETAIN` in `studio/jobs.py`), so steady
state is bounded around 500 MB. The 4 jobs from the bare-metal instance were
copied in at cutover.
state is bounded around 500 MB. As of `b72425b` the index is rehydrated from
disk at startup, so that bound holds **across restarts** and not merely
within one process — see the resolved-finding section below for why that
distinction mattered. The 4 jobs from the bare-metal instance were copied in
at cutover.
- Internal render timeout is 480 s, which is why the healthcheck interval is
loose — an aggressive probe would measure queue depth rather than liveness.
@@ -127,21 +130,34 @@ and the grant itself, passed inline and never written to disk on this host.
Do not reintroduce it: a site-admin credential on a GPU box is a blast radius
nobody needs for a read-only fetch.
## ⚠️ Upstream finding: the on-disk job store grows without bound
## ✅ Resolved upstream: the on-disk job store used to grow without bound
The job index is **memory-only** — `JobStore._jobs` is a plain dict, and
nothing scans `WATERLAND_STUDIO_DATA` at startup. Two consequences:
**Fixed in `b72425b` and deployed here.** Kept on the record because the
symptom is easy to misread and the fix has a property worth knowing about.
1. After any restart the UI and `/api/jobs` list only jobs created *since*
that restart, even though every earlier job's files are still on the volume.
This is cosmetic and not data loss — verified at cutover: the API reported
1 job while the volume held all 16 directories (60.6 MB).
2. More importantly, the `RETAIN = 40` self-eviction only ever sees in-memory
jobs, so directories orphaned by a restart are **never** reclaimed. The
handover's "bounded around 500 MB" holds within a single process lifetime;
across restarts the store grows monotonically at ~12 MB per animated job.
The job index used to be **memory-only** — `JobStore._jobs` was a plain dict
and nothing scanned `WATERLAND_STUDIO_DATA` at startup. Two consequences:
Growth is slow, so this is not urgent, but it is unbounded and the volume will
need occasional pruning until upstream rehydrates the index (or sweeps orphans)
at startup. Reported to waterland-dev — it is their call, not a thing to patch
from the infrastructure side.
1. After any restart the UI and `/api/jobs` listed only jobs created *since*
that restart, even though every earlier job's files were still on the
volume. Cosmetic, not data loss — and it is how this was spotted: the API
reported 1 job while the volume held all 16 directories (60.6 MB).
2. The real one: `RETAIN = 40` self-eviction only ever walked the in-memory
dict, so directories orphaned by a restart were **never** reclaimed. The
handover's "bounded around 500 MB" held within a single process lifetime;
across restarts the store grew monotonically at ~12 MB per animated job.
Upstream now **rehydrates the index at startup** by scanning the data dir, and
deliberately *adopts* directories it cannot fully parse rather than skipping
them — a directory that never enters the index is precisely the one eviction
can never reclaim. Sidecar present → restored; no sidecar → adopted with
dimensions read from the PNG IHDR; corrupt sidecar → inferred, no startup
crash. **Neither source nor sidecar → skipped on purpose**, because adopting
those would make eviction a delete-arbitrary-directories primitive pointed at
this volume. Jobs left `running`/`queued` by a mid-render restart are marked
`failed`, so they stop being unreclaimable phantoms that inflate `queue_depth`.
Verified here after the update: `/api/jobs` went **1 → 16** against 16
directories on disk, with nothing reclaimed — correct, since 16 is under
`RETAIN=40`; adoption only made them visible. The bound now holds across
restarts, so no hand-pruning is needed.
+3
View File
@@ -40,6 +40,9 @@ services:
# Job store: uploaded sources plus rendered plates and animations.
# ~12 MB per job with an animation; the app self-evicts at 40 retained
# jobs (RETAIN in studio/jobs.py), so steady state is bounded ~500 MB.
# That bound holds ACROSS restarts only as of b72425b, which rehydrates
# the job index from this volume at startup; before that, eviction was
# blind to anything a restart had orphaned. See the README.
# Scratch output, not source-of-truth — losing it costs a re-render.
- waterland_studio_data:/data
# cupy JIT kernel cache. Not optional for good behaviour: cupy compiles