feat(waterland-studio): deploy b72425b — all three upstream findings fixed
One update.sh run on irv-ml1 carried both open upstream PRs, per the operator's green-light on the job-store fix: - #5 (464dfc2) declares cupy-cuda12x[ctk] on the gpu extra and takes uv out of the render path (sys.executable -m waterland.cli), retiring the runtime prune trap at the source. - #6 (b72425b) rehydrates the job index from the data volume at startup, fixing the unbounded store growth reported from this side. Verified after the update rather than assumed: healthy on backend cupy; /api/jobs went 1 -> 16 against 16 directories on disk, so API and volume agree for the first time; nothing wrongly reclaimed, correct since 16 is under RETAIN=40 and adoption only makes them visible; a real 256^2 plate render completes warm, so the kernel-cache volume survived the image swap. A subsequent render took both counts to 17. The image keeps its explicit [ctk] install and UV_NO_SYNC/UV_OFFLINE pins even though both are now redundant. The header requirement is a property of this slim base, not of the upstream extra, and the cost is measured rather than assumed: uv sync satisfies it first, so the line reports "Audited 1 package" and adds 0.3s to the build. The env pins are now cheap defence-in-depth against any future path that re-enters uv. Docs corrected in place: the README's upstream-finding section is now a resolved-finding record, and the two "bounded ~500 MB" claims say which commit made that bound hold across restarts rather than only within a process. Comment-side changes pushed to the live compose dir; no restart was needed for them.
This commit is contained in:
@@ -95,8 +95,11 @@ service merely looks slow.
|
||||
needs wider reach. Ask.
|
||||
- Job store is scratch output, not source-of-truth: ~12 MB per animated job,
|
||||
self-evicting at 40 retained jobs (`RETAIN` in `studio/jobs.py`), so steady
|
||||
state is bounded around 500 MB. The 4 jobs from the bare-metal instance were
|
||||
copied in at cutover.
|
||||
state is bounded around 500 MB. As of `b72425b` the index is rehydrated from
|
||||
disk at startup, so that bound holds **across restarts** and not merely
|
||||
within one process — see the resolved-finding section below for why that
|
||||
distinction mattered. The 4 jobs from the bare-metal instance were copied in
|
||||
at cutover.
|
||||
- Internal render timeout is 480 s, which is why the healthcheck interval is
|
||||
loose — an aggressive probe would measure queue depth rather than liveness.
|
||||
|
||||
@@ -127,21 +130,34 @@ and the grant itself, passed inline and never written to disk on this host.
|
||||
Do not reintroduce it: a site-admin credential on a GPU box is a blast radius
|
||||
nobody needs for a read-only fetch.
|
||||
|
||||
## ⚠️ Upstream finding: the on-disk job store grows without bound
|
||||
## ✅ Resolved upstream: the on-disk job store used to grow without bound
|
||||
|
||||
The job index is **memory-only** — `JobStore._jobs` is a plain dict, and
|
||||
nothing scans `WATERLAND_STUDIO_DATA` at startup. Two consequences:
|
||||
**Fixed in `b72425b` and deployed here.** Kept on the record because the
|
||||
symptom is easy to misread and the fix has a property worth knowing about.
|
||||
|
||||
1. After any restart the UI and `/api/jobs` list only jobs created *since*
|
||||
that restart, even though every earlier job's files are still on the volume.
|
||||
This is cosmetic and not data loss — verified at cutover: the API reported
|
||||
1 job while the volume held all 16 directories (60.6 MB).
|
||||
2. More importantly, the `RETAIN = 40` self-eviction only ever sees in-memory
|
||||
jobs, so directories orphaned by a restart are **never** reclaimed. The
|
||||
handover's "bounded around 500 MB" holds within a single process lifetime;
|
||||
across restarts the store grows monotonically at ~12 MB per animated job.
|
||||
The job index used to be **memory-only** — `JobStore._jobs` was a plain dict
|
||||
and nothing scanned `WATERLAND_STUDIO_DATA` at startup. Two consequences:
|
||||
|
||||
Growth is slow, so this is not urgent, but it is unbounded and the volume will
|
||||
need occasional pruning until upstream rehydrates the index (or sweeps orphans)
|
||||
at startup. Reported to waterland-dev — it is their call, not a thing to patch
|
||||
from the infrastructure side.
|
||||
1. After any restart the UI and `/api/jobs` listed only jobs created *since*
|
||||
that restart, even though every earlier job's files were still on the
|
||||
volume. Cosmetic, not data loss — and it is how this was spotted: the API
|
||||
reported 1 job while the volume held all 16 directories (60.6 MB).
|
||||
2. The real one: `RETAIN = 40` self-eviction only ever walked the in-memory
|
||||
dict, so directories orphaned by a restart were **never** reclaimed. The
|
||||
handover's "bounded around 500 MB" held within a single process lifetime;
|
||||
across restarts the store grew monotonically at ~12 MB per animated job.
|
||||
|
||||
Upstream now **rehydrates the index at startup** by scanning the data dir, and
|
||||
deliberately *adopts* directories it cannot fully parse rather than skipping
|
||||
them — a directory that never enters the index is precisely the one eviction
|
||||
can never reclaim. Sidecar present → restored; no sidecar → adopted with
|
||||
dimensions read from the PNG IHDR; corrupt sidecar → inferred, no startup
|
||||
crash. **Neither source nor sidecar → skipped on purpose**, because adopting
|
||||
those would make eviction a delete-arbitrary-directories primitive pointed at
|
||||
this volume. Jobs left `running`/`queued` by a mid-render restart are marked
|
||||
`failed`, so they stop being unreclaimable phantoms that inflate `queue_depth`.
|
||||
|
||||
Verified here after the update: `/api/jobs` went **1 → 16** against 16
|
||||
directories on disk, with nothing reclaimed — correct, since 16 is under
|
||||
`RETAIN=40`; adoption only made them visible. The bound now holds across
|
||||
restarts, so no hand-pruning is needed.
|
||||
|
||||
@@ -40,6 +40,9 @@ services:
|
||||
# Job store: uploaded sources plus rendered plates and animations.
|
||||
# ~12 MB per job with an animation; the app self-evicts at 40 retained
|
||||
# jobs (RETAIN in studio/jobs.py), so steady state is bounded ~500 MB.
|
||||
# That bound holds ACROSS restarts only as of b72425b, which rehydrates
|
||||
# the job index from this volume at startup; before that, eviction was
|
||||
# blind to anything a restart had orphaned. See the README.
|
||||
# Scratch output, not source-of-truth — losing it costs a re-render.
|
||||
- waterland_studio_data:/data
|
||||
# cupy JIT kernel cache. Not optional for good behaviour: cupy compiles
|
||||
|
||||
Reference in New Issue
Block a user