waterland-dev confirmed the mechanism: adoption marks a job failed on a
sidecar saying running/queued, or on a directory with no plate.png. The
pre-header-fix renders died 1.7s in with a source and no plate, so they
land in the second branch. Recorded so nobody investigates adopted
history as a live fault.
One update.sh run on irv-ml1 carried both open upstream PRs, per the
operator's green-light on the job-store fix:
- #5 (464dfc2) declares cupy-cuda12x[ctk] on the gpu extra and takes uv
out of the render path (sys.executable -m waterland.cli), retiring the
runtime prune trap at the source.
- #6 (b72425b) rehydrates the job index from the data volume at startup,
fixing the unbounded store growth reported from this side.
Verified after the update rather than assumed: healthy on backend cupy;
/api/jobs went 1 -> 16 against 16 directories on disk, so API and volume
agree for the first time; nothing wrongly reclaimed, correct since 16 is
under RETAIN=40 and adoption only makes them visible; a real 256^2 plate
render completes warm, so the kernel-cache volume survived the image swap.
A subsequent render took both counts to 17.
The image keeps its explicit [ctk] install and UV_NO_SYNC/UV_OFFLINE pins
even though both are now redundant. The header requirement is a property
of this slim base, not of the upstream extra, and the cost is measured
rather than assumed: uv sync satisfies it first, so the line reports
"Audited 1 package" and adds 0.3s to the build. The env pins are now
cheap defence-in-depth against any future path that re-enters uv.
Docs corrected in place: the README's upstream-finding section is now a
resolved-finding record, and the two "bounded ~500 MB" claims say which
commit made that bound hold across restarts rather than only within a
process. Comment-side changes pushed to the live compose dir; no restart
was needed for them.
Operator granted claude-bot read on vh/waterland; verified scoped correctly
(admin false, push false, pull true). Token is on irv-ml1 at
/root/.config/waterland-studio/git-credentials, 0600 root-owned, wired as a
REPO-SCOPED credential helper rather than a global one, and .git/config holds
no token so the remote stays clean in any diff or backup. The vh site-admin
token was used only for the initial clone and the grant itself and was never
written to disk on that host.
update.sh now runs end to end: fetch, rebuild, recreate, health. Verified the
kernel-cache volume survives a recreate (warm 256^2+anim render 6.6s straight
after) and the job store survives with all 16 directories intact.
Records an upstream finding surfaced by that check: JobStore._jobs is
memory-only and nothing scans the data dir at startup, so after a restart the
API lists only new jobs while old ones persist on disk — cosmetic — but the
RETAIN=40 eviction only sees in-memory jobs, so restart-orphaned directories
are never reclaimed. The handover's ~500 MB bound holds per process lifetime,
not across restarts. Reported to waterland-dev; upstream's call to fix.
Replaces a bare nohup on irv-ml1:8410 that would not have survived a reboot,
handed over by waterland-dev. Tracks vh/waterland @ main (PR #4 merged; main
HEAD is exactly the pinned 8025366).
Build context is a checkout at /opt/waterland-studio/src, deliberately OUTSIDE
the compose dir — deploy-stack.sh rsyncs stacks/<stack>/ with --delete and
would otherwise eat it. The Dockerfile is passed out-of-context.
Three landmines, all measured:
1. Both uv extras are load-bearing at build AND run. jobs.py shells the
renderer out as a literal with no --extra flags, so uv
would re-sync at runtime and prune cupy — silently dropping to the numpy
path at ~21x wall time. UV_NO_SYNC pins it; UV_OFFLINE makes any failure
loud instead of quietly slow.
2. cupy needs CUDA HEADERS for its NVRTC compile, not just the driver and the
wheel's runtime libs. The host has a system CUDA toolkit so the nohup
process found them by accident; a slim image does not, and every render
died 1.7s in with 'Failed to find CUDA headers' printed through argparse's
usage banner — which reads like a CLI bug, not a missing toolkit. Fixed
with cupy-cuda12x[ctk] (hundreds of MB, vs ~6 GB for a -devel base).
3. The A6000 is host device 1 but container device 0, since compose exposes
exactly one GPU. CUDA_VISIBLE_DEVICES_TARGET=0 inside; copying the host's
value selects a device that does not exist.
/root/.cupy is a volume because the NVRTC compile costs ~17s: verified at
23.3s cold vs 6.1s warm, and re-verified across a restart (23.2s on a fresh
cache volume, 6.0s once populated). Warm 256^2+anim beats the 7.4s recorded
against bare metal, so containerising cost nothing.
Job store seeded with the 4 jobs from the displaced instance. Serial by design
(one replica, one card) and unauthenticated, so it stays LAN/WireGuard-only.
vh