vh
39da1d4a97
The board mixed tools with endpoints. A vLLM seat whose href is a /docs page
sat in the same band as ComfyUI; the MQTT broker and the RustDesk relay, which
have no page at all, sat in Apps; and `Service Networking` was thirteen members
spanning three AdGuards, five Dockges, two Traefiks and four headless agents.
Every group is now one of two kinds and they never mix. TOOLS are expanded and
sit at the top of their tab. ENDPOINTS — an API, a broker, a background agent,
an href that is /docs or /ping or nothing — carry `initiallyCollapsed: true`
and sit at the bottom. Collapsed is not hidden: the eyebrow and its rule still
render, so the tab still says the thing exists and one click expands it.
A second rule fell out of the same pass and now shapes the group boundaries: a
group's members should all carry a widget or none should. A stat strip makes a
card ~50px taller, so one widget card in a row of plain ones opens a void under
the plain ones. That is why AdGuard and Traefik get their own groups rather
than sharing one with Dockge, and it is most of why the old Service Networking
band looked broken. AdGuard (ANA) was the last short card in its row and now
carries the same query/blocked/latency strip as its two siblings — one
infra-ops AdGuard login authenticates against all three instances, verified
against each; it lives in that stack's .env on the host and is vaulted.
The sixteen GPU-backed model seats were deliberately NOT relabelled.
`homepage.group` is read at container creation, so clearer names for
`AI - Inference` and friends would have cost a recreate on six vLLM seats, four
eval seats and four TTS engines — multi-minute model reloads on endpoints peers
reach through the gateway. Order plus `initiallyCollapsed` buys the same
separation for nothing, so those names stay as they are on purpose.
28 containers that ARE cheap to bounce were relabelled, across five hosts, via
rerunnable elway playbooks. Their label steps are gated on the old value still
being present, so a second run reports skipped rather than churning. Two verify
steps were wrong on first contact and are fixed with the reason recorded: the
traefik check raced its own recreate, and asserting a model seat is "running"
cannot answer "did I bounce it" when a seat may be legitimately stopped —
container age can, and now does.
The canonical stacks/ tree was synced to the deployed labels afterwards, so
intent and reality agree again on all fourteen tracked stacks.
Also documents the real nature of the post-recreate blank dashboard, which cost
~25 minutes here and an hour on 2026-08-19. `initialSettings":{}` in the served
HTML is the catch branch of the page's data loader, not a warm-up and not a
cache — and the error can vanish entirely, because the logger is assigned inside
the same try and the catch only logs if the logger exists. Ruled out by
measurement this time: all four API routes return 200 with correct content while
the page serves {}, and the previous known-good settings.yaml reproduces it
identically. The README now carries the one-command test and the next lead.
Before/after, all four tabs: http://10.100.10.50:8090/b/homepage-relayout/
waterland-studio — watercolour render service (irv-ml1)
FastAPI + vanilla-JS SPA fronting the waterland CLI: upload an image, get a
watercolour plate and a painted-in reveal animation. Every job shells out to
the CLI, which runs a fluid simulation on the A6000.
- Host: irv-ml1 (10.100.79.3, WireGuard-only) · Port: 8410
- URL: http://10.100.79.3:8410/ · Health:
GET /api/health - Source:
gitea.phasefinal.com/vh/waterland, trackingmain
Handed over by waterland-dev on 2026-08-19, replacing a bare nohup that
would not have survived a reboot.
Layout — the build context is deliberately outside this directory
| path | what |
|---|---|
/opt/waterland-studio/src |
checkout of vh/waterland @ main — the build context |
/opt/docker/compose/waterland-studio/ |
compose.yaml, Dockerfile, update.sh, .env |
volume waterland-studio_waterland_studio_data |
job store (/data) |
volume waterland-studio_waterland_studio_kernels |
cupy JIT cache (/root/.cupy) |
The checkout must NOT live under the compose directory. deploy-stack.sh
rsyncs stacks/<stack>/ with --delete, so a checkout kept beside
compose.yaml would be destroyed by the next deploy of this stack. The
Dockerfile is passed out-of-context to keep both trees clean.
Refresh source + rebuild:
ssh infra-ops@10.100.79.3 /opt/docker/compose/waterland-studio/update.sh
Three landmines, all of them measured rather than guessed
1. Both uv extras are load-bearing, at build AND at run. gpu carries
cupy-cuda12x; a bare uv sync prunes it and the renderer silently drops to
the numpy path at ~21x the wall time — it does not error, it just gets slow.
Worse, studio/jobs.py shells the renderer out as a literal uv run waterland
with no --extra flags, so uv would re-sync at runtime and prune cupy right
back out. UV_NO_SYNC=1 stops that; UV_OFFLINE=1 means that if the pin ever
stops working the job fails loudly instead of quietly rebuilding a slower
environment.
2. cupy needs CUDA headers, which the host never had to declare. cupy
compiles kernels at runtime through NVRTC, which needs toolkit headers — not
just the driver and the runtime libs bundled in the wheel. On irv-ml1 a CUDA
toolkit is installed system-wide, so the bare nohup process found them by
accident; a slim image has none. Every render died 1.7s in with
RuntimeError: Failed to find CUDA headers.
printed through argparse's usage banner, which makes it read like a CLI
argument bug rather than a missing toolkit — that misdirection is the reason
this is written down. Fixed with uv pip install "cupy-cuda12x[ctk]", which
pulls the headers as wheels: a few hundred MB against ~6 GB for a -devel
base image. It runs after uv sync, because sync prunes what it does not
know about.
3. The GPU index inside the container is not the host's. The app pins
CUDA_DEVICE_ORDER=PCI_BUS_ID and selects CUDA_VISIBLE_DEVICES_TARGET
(default 1, correct on the host). Compose exposes exactly one GPU
(device_ids: ["1"], the A6000 in Docker's ordering), so inside the
container that card is index 0 — hence CUDA_VISIBLE_DEVICES_TARGET=0.
Copying the host's value selects a device that does not exist. Device 0 on the
host is the 3090, which hosts the TTS zoo and must not be touched.
Performance, measured on this host
| job | wall |
|---|---|
| 256², animation, cold container | 23.3 s |
| 256², animation, warm | 6.1 s |
256², plate only (--codec none) |
3.9 s |
| 512², plate only | 6.4 s |
Warm numbers beat the 7.4 s at 256² recorded against the bare-metal process, so
containerising cost nothing. The cold-vs-warm gap is cupy's NVRTC compile,
which is why /root/.cupy is a volume: verified by recreating the container
(fresh cache → 23.2 s first render) and then restarting it (populated cache →
6.0 s). Without that volume every restart makes the next user wait 4x and the
service merely looks slow.
Operational constraints — from waterland-dev, not inferred
- Serial by design. One replica, one card. A render is 20–45 s of near-full GPU and the app runs a single worker thread. Two of these on the same A6000 would OOM or thrash. Throughput is a conversation about hardware, not replica count.
- No authentication, and it accepts arbitrary file uploads. It must stay inside the LAN / WireGuard boundary. Do not paper over this with a proxy password — waterland-dev has offered to add a real auth layer if it ever needs wider reach. Ask.
- Job store is scratch output, not source-of-truth: ~12 MB per animated job,
self-evicting at 40 retained jobs (
RETAINinstudio/jobs.py), so steady state is bounded around 500 MB. As ofb72425bthe index is rehydrated from disk at startup, so that bound holds across restarts and not merely within one process — see the resolved-finding section below for why that distinction mattered. The 4 jobs from the bare-metal instance were copied in at cutover. - Internal render timeout is 480 s, which is why the healthcheck interval is loose — an aggressive probe would measure queue depth rather than liveness.
Source credential — claude-bot, read-only
The repo is not anonymously readable (an unauthenticated clone 403s), so the
host needs a credential to fetch. Operator granted claude-bot read on
vh/waterland on 2026-08-19; verified scoped correctly:
admin: False | push: False | pull: True
The token lives on irv-ml1 at /root/.config/waterland-studio/git-credentials,
mode 0600, root-owned, and is wired as a repo-scoped credential helper —
not a global one:
git -C /opt/waterland-studio/src config credential.helper \
'store --file=/root/.config/waterland-studio/git-credentials'
.git/config itself holds no token (verified), so the remote URL stays clean
in any diff, log or backup of the checkout.
The operator's vh site-admin token was used only for the initial clone
and the grant itself, passed inline and never written to disk on this host.
Do not reintroduce it: a site-admin credential on a GPU box is a blast radius
nobody needs for a read-only fetch.
✅ Resolved upstream: the on-disk job store used to grow without bound
Fixed in b72425b and deployed here. Kept on the record because the
symptom is easy to misread and the fix has a property worth knowing about.
The job index used to be memory-only — JobStore._jobs was a plain dict
and nothing scanned WATERLAND_STUDIO_DATA at startup. Two consequences:
- After any restart the UI and
/api/jobslisted only jobs created since that restart, even though every earlier job's files were still on the volume. Cosmetic, not data loss — and it is how this was spotted: the API reported 1 job while the volume held all 16 directories (60.6 MB). - The real one:
RETAIN = 40self-eviction only ever walked the in-memory dict, so directories orphaned by a restart were never reclaimed. The handover's "bounded around 500 MB" held within a single process lifetime; across restarts the store grew monotonically at ~12 MB per animated job.
Upstream now rehydrates the index at startup by scanning the data dir, and
deliberately adopts directories it cannot fully parse rather than skipping
them — a directory that never enters the index is precisely the one eviction
can never reclaim. Sidecar present → restored; no sidecar → adopted with
dimensions read from the PNG IHDR; corrupt sidecar → inferred, no startup
crash. Neither source nor sidecar → skipped on purpose, because adopting
those would make eviction a delete-arbitrary-directories primitive pointed at
this volume. Jobs left running/queued by a mid-render restart are marked
failed, so they stop being unreclaimable phantoms that inflate queue_depth.
Verified here after the update: /api/jobs went 1 → 16 against 16
directories on disk, with nothing reclaimed — correct, since 16 is under
RETAIN=40; adoption only made them visible. The bound now holds across
restarts, so no hand-pruning is needed.
Don't chase the one failed job in that list. Adoption marks a job
failed on exactly two conditions: a sidecar saying running/queued, or a
directory with no plate.png. The renders that died 1.7 s in on the missing
CUDA headers (before that fix) wrote a source and never a plate, so they land
in the second branch. It is adopted history being labelled honestly, not a
current fault.