From a2b6bf409ed11f5d390f5e4530d6ae1886938efa Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 19 Aug 2026 01:23:47 -0700 Subject: [PATCH] memory: waterland-studio upstream fixes landed, container stays pinned MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit waterland-dev merged PR #5 (main now 464dfc2), fixing both landmines at source: the gpu extra declares cupy-cuda12x[ctk], and the renderer spawns sys.executable -m waterland.cli instead of re-entering uv mid-job. The running container deliberately stays on 8025366. Its own [ctk] install and UV_NO_SYNC/UV_OFFLINE pins already neutralise both defects, so a rebuild would buy reliability that is already present — and the project is in wind-down. Both guards are kept rather than dropped: the header requirement is a property of this slim image, not of the upstream extra, and the uv pins are now cheap defence-in-depth against any future path that re-enters uv. Also records waterland-dev's confirmation of the unbounded job-store growth and the operator's green-light on their startup-rehydrate fix. That PR merging is the rebuild trigger: one update.sh run lands the rehydrate and 464dfc2 together. Marks the inbox drained. --- ...26-08-19-waterland-studio-containerised.md | 38 ++++++++++++++++++- persistent-memory.md | 4 +- 2 files changed, 38 insertions(+), 4 deletions(-) diff --git a/persistent-memory.d/2026-08-19-waterland-studio-containerised.md b/persistent-memory.d/2026-08-19-waterland-studio-containerised.md index 168fadf..6a197fd 100644 --- a/persistent-memory.d/2026-08-19-waterland-studio-containerised.md +++ b/persistent-memory.d/2026-08-19-waterland-studio-containerised.md @@ -6,9 +6,17 @@ a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare `stacks/waterland-studio/`, `restart: unless-stopped`, healthy on irv-ml1:8410. Commits `a2b5b58`, `8189076`. -Tracking `main` per operator: PR #4 merged and `main` HEAD is exactly the +Tracking `main` per operator: PR #4 merged and `main` HEAD was exactly the pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway. +**`main` has since moved to `464dfc2` (PR #5, 2026-08-19) — the running +container is deliberately still on `8025366`.** #5 fixes landmines 1 and 2 at +the source (see below). Nothing on the service is broken by staying behind: +the image's own `[ctk]` install and env pins already neutralise both, so a +rebuild buys reliability that is already present, and the operator has called +wind-down on the project. `update.sh` picks up `main` on the next rebuild that +has a real reason behind it. + ## Build context lives OUTSIDE the compose dir — on purpose `/opt/waterland-studio/src` is the checkout; the Dockerfile is passed @@ -31,6 +39,14 @@ built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the pin ever stops holding the job fails **loudly** instead of quietly rebuilding a slower environment. +**Fixed upstream in `464dfc2`:** the server now spawns +`sys.executable -m waterland.cli` directly — no resolver in the render path at +all. **The pins stay anyway.** They cost nothing and are now defence-in-depth: +if any future code path re-enters `uv` inside the container, the job fails +loudly instead of quietly dropping to the numpy backend. `uv` itself must stay +in the image regardless — it performs the build-time `uv sync` / +`uv pip install`, and this is a single-stage build. + ## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare Every render died 1.7s in with: @@ -53,7 +69,12 @@ hundred MB against ~6 GB for a `-devel` base image. It runs **after** `uv sync`, because sync prunes what it does not know about. Reported upstream: it is an undeclared runtime dependency of the `gpu` extra, -and anyone running this without a system toolkit hits it. +and anyone running this without a system toolkit hits it. **Declared upstream +in `464dfc2`** (`gpu` is now `cupy-cuda12x[ctk]>=13`). **The explicit install +stays in the Dockerfile**: the header requirement is a property of *this* +image — a slim base with no system CUDA toolkit — so it belongs in the file +that creates the problem, not inherited from an extra two repos away. It also +survives any future restructuring of the `gpu` extra. ## Landmine 3 — the GPU index inside the container is not the host's @@ -96,6 +117,19 @@ startup**. Consequences: Reported to waterland-dev with evidence; **not patched from the infra side** — it is their code. Prune the volume by hand if it bites first. +**waterland-dev confirmed it (2026-08-19)** — their "bounded ~500 MB" handover +claim holds within one process lifetime and nowhere else, which on a +`restart: unless-stopped` service is the wrong lifetime to have bounded. They +have **surfaced a startup-rehydrate fix to the operator** rather than opening a +third PR during wind-down. **Operator green-lit it (2026-08-19)** — a PR is +expected. Volume at **61 MB / 16 job dirs** as of 2026-08-19 08:30Z — +unchanged since first observed, so there is no clock on it. + +**This is the rebuild trigger.** When the rehydrate PR merges, one `update.sh` +run on irv-ml1 moves the container off `8025366` and picks up `464dfc2`'s +fixes in the same motion — the "rebuild with a real reason behind it" named +above. Nothing else on the infra side needs coordinating. + ## Access Repo is not anonymously readable (a bare clone 403s). Operator granted diff --git a/persistent-memory.md b/persistent-memory.md index 9effc51..27083bd 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -109,11 +109,11 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-08-19 (late) — a long infra session, all of it landed and committed: **fleet `*.internal` DNS is live** (git-sourced, 42 names, three resolvers incl. a NEW colo one), **waterland studio containerised** on irv-ml1, **Homepage cleaned up and themed** with Australis Skyfall + an Arbo-generated background, and **four unmanaged stacks adopted** into `stacks/` (two of which turned out to be quietly broken — SearXNG's healthcheck, SeaFile down 3 months). ⚠ 17 commits unpushed. ⚠ 2 unread althing messages from waterland-dev left deliberately undrained — operator deferred the inbox to after the `/clear`._ +_As of 2026-08-19 (late) — a long infra session, all of it landed and committed: **fleet `*.internal` DNS is live** (git-sourced, 42 names, three resolvers incl. a NEW colo one), **waterland studio containerised** on irv-ml1, **Homepage cleaned up and themed** with Australis Skyfall + an Arbo-generated background, and **four unmanaged stacks adopted** into `stacks/` (two of which turned out to be quietly broken — SearXNG's healthcheck, SeaFile down 3 months). ⚠ 17 commits unpushed. ~~⚠ 2 unread althing messages from waterland-dev left deliberately undrained~~ — **drained 2026-08-19 post-`/clear`; thread closed, see below.**_ - **🟢 FLEET `.internal` DNS — LIVE 2026-08-19.** `..internal`, sites `ana`/`esh`/`nh3`. `dns/internal.yaml` is the source of truth; `scripts/dns-sync.py` reconciles the three AdGuard resolvers (diff → prompt → apply, idempotent). 42 names resolving from all three sites. **Colo got its first resolver ever** (`stacks/adguard-ana/`, API on **8053** not 8080, no blocklists by design) — before this, ana-docker resolved straight against `1.1.1.1`. Auth = a dedicated `infra-ops` AdGuard user, password vaulted `nh3-dev/adguard-infra-ops-password`. **⏳ TWO OPEN, both operator's to schedule:** (a) colo hosts still point at `1.1.1.1` so they do not yet *use* the new resolver — repointing a site's DNS is a separate change; (b) the static-v6 convention (each server at its site's `/64` with low bits echoing the v4 octet, `esh-docker-vm` → `…::45`) is **proposed, not ruled on**. The `v6:` column is empty and correct — no fleet host has a global v6 address yet. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md` -- **🟢 WATERLAND STUDIO — containerised + live, irv-ml1:8410.** Displaced a bare `nohup` (PID 1283383). Warm 256²+anim render **6.1s**, beating the 7.4s bare-metal figure. `claude-bot` now has **read** on `vh/waterland` (operator-granted); token on irv-ml1 root-owned 0600, repo-scoped helper, nothing in `.git/config`. **⏳ Reported upstream, NOT patched by us:** the on-disk job store grows without bound across restarts (`RETAIN=40` eviction only sees the in-memory dict). → `persistent-memory.d/2026-08-19-waterland-studio-containerised.md` +- **🟢 WATERLAND STUDIO — containerised + live, irv-ml1:8410.** Displaced a bare `nohup` (PID 1283383). Warm 256²+anim render **6.1s**, beating the 7.4s bare-metal figure. `claude-bot` now has **read** on `vh/waterland` (operator-granted); token on irv-ml1 root-owned 0600, repo-scoped helper, nothing in `.git/config`. **⏳ Reported upstream, NOT patched by us:** the on-disk job store grows without bound across restarts (`RETAIN=40` eviction only sees the in-memory dict) — waterland-dev **confirmed it as a real defect**, and the **operator green-lit their startup-rehydrate fix** — PR expected (61 MB / 16 dirs, no clock on it). Upstream `main` is now `464dfc2` (PR #5 fixes the uv-prune and CUDA-header landmines at source); **the container deliberately stays on `8025366`** and keeps its `[ctk]` install + `UV_NO_SYNC`/`UV_OFFLINE` pins as defence-in-depth. **⏳ NEXT: when the rehydrate PR merges, one `update.sh` on irv-ml1 lands it and `464dfc2` together** — that merge is the rebuild trigger. Thread closed. → `persistent-memory.d/2026-08-19-waterland-studio-containerised.md` - **🟢 HOMEPAGE — cleaned + themed (Australis Skyfall).** Three real defects fixed (UltraSeedbox on all tabs, Uptime Kuma double-rendered, fiction column counts), AI tab reordered by clickability, then themed from the operator's Skyfall handoff bundle with a background generated by **Arbo** (`t2i-ui-background`, job `13f0891f4e42`). ⚠ **After any recreate the tab bar/wallpaper/i18n vanish for up to ~an hour and then return on their own — do not chase it.** ⚠ CSS is served per-request: a theme change needs a **reload**, not a recreate, and candidate CSS can be injected live via Playwright for seconds-long iteration. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md`