memory: waterland-studio upstream fixes landed, container stays pinned
waterland-dev merged PR #5 (main now 464dfc2), fixing both landmines at source: the gpu extra declares cupy-cuda12x[ctk], and the renderer spawns sys.executable -m waterland.cli instead of re-entering uv mid-job. The running container deliberately stays on 8025366. Its own [ctk] install and UV_NO_SYNC/UV_OFFLINE pins already neutralise both defects, so a rebuild would buy reliability that is already present — and the project is in wind-down. Both guards are kept rather than dropped: the header requirement is a property of this slim image, not of the upstream extra, and the uv pins are now cheap defence-in-depth against any future path that re-enters uv. Also records waterland-dev's confirmation of the unbounded job-store growth and the operator's green-light on their startup-rehydrate fix. That PR merging is the rebuild trigger: one update.sh run lands the rehydrate and 464dfc2 together. Marks the inbox drained.
This commit is contained in:
@@ -6,9 +6,17 @@ a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare
|
||||
`stacks/waterland-studio/`, `restart: unless-stopped`, healthy on
|
||||
irv-ml1:8410. Commits `a2b5b58`, `8189076`.
|
||||
|
||||
Tracking `main` per operator: PR #4 merged and `main` HEAD is exactly the
|
||||
Tracking `main` per operator: PR #4 merged and `main` HEAD was exactly the
|
||||
pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway.
|
||||
|
||||
**`main` has since moved to `464dfc2` (PR #5, 2026-08-19) — the running
|
||||
container is deliberately still on `8025366`.** #5 fixes landmines 1 and 2 at
|
||||
the source (see below). Nothing on the service is broken by staying behind:
|
||||
the image's own `[ctk]` install and env pins already neutralise both, so a
|
||||
rebuild buys reliability that is already present, and the operator has called
|
||||
wind-down on the project. `update.sh` picks up `main` on the next rebuild that
|
||||
has a real reason behind it.
|
||||
|
||||
## Build context lives OUTSIDE the compose dir — on purpose
|
||||
|
||||
`/opt/waterland-studio/src` is the checkout; the Dockerfile is passed
|
||||
@@ -31,6 +39,14 @@ built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the
|
||||
pin ever stops holding the job fails **loudly** instead of quietly rebuilding a
|
||||
slower environment.
|
||||
|
||||
**Fixed upstream in `464dfc2`:** the server now spawns
|
||||
`sys.executable -m waterland.cli` directly — no resolver in the render path at
|
||||
all. **The pins stay anyway.** They cost nothing and are now defence-in-depth:
|
||||
if any future code path re-enters `uv` inside the container, the job fails
|
||||
loudly instead of quietly dropping to the numpy backend. `uv` itself must stay
|
||||
in the image regardless — it performs the build-time `uv sync` /
|
||||
`uv pip install`, and this is a single-stage build.
|
||||
|
||||
## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare
|
||||
|
||||
Every render died 1.7s in with:
|
||||
@@ -53,7 +69,12 @@ hundred MB against ~6 GB for a `-devel` base image. It runs **after**
|
||||
`uv sync`, because sync prunes what it does not know about.
|
||||
|
||||
Reported upstream: it is an undeclared runtime dependency of the `gpu` extra,
|
||||
and anyone running this without a system toolkit hits it.
|
||||
and anyone running this without a system toolkit hits it. **Declared upstream
|
||||
in `464dfc2`** (`gpu` is now `cupy-cuda12x[ctk]>=13`). **The explicit install
|
||||
stays in the Dockerfile**: the header requirement is a property of *this*
|
||||
image — a slim base with no system CUDA toolkit — so it belongs in the file
|
||||
that creates the problem, not inherited from an extra two repos away. It also
|
||||
survives any future restructuring of the `gpu` extra.
|
||||
|
||||
## Landmine 3 — the GPU index inside the container is not the host's
|
||||
|
||||
@@ -96,6 +117,19 @@ startup**. Consequences:
|
||||
Reported to waterland-dev with evidence; **not patched from the infra side** —
|
||||
it is their code. Prune the volume by hand if it bites first.
|
||||
|
||||
**waterland-dev confirmed it (2026-08-19)** — their "bounded ~500 MB" handover
|
||||
claim holds within one process lifetime and nowhere else, which on a
|
||||
`restart: unless-stopped` service is the wrong lifetime to have bounded. They
|
||||
have **surfaced a startup-rehydrate fix to the operator** rather than opening a
|
||||
third PR during wind-down. **Operator green-lit it (2026-08-19)** — a PR is
|
||||
expected. Volume at **61 MB / 16 job dirs** as of 2026-08-19 08:30Z —
|
||||
unchanged since first observed, so there is no clock on it.
|
||||
|
||||
**This is the rebuild trigger.** When the rehydrate PR merges, one `update.sh`
|
||||
run on irv-ml1 moves the container off `8025366` and picks up `464dfc2`'s
|
||||
fixes in the same motion — the "rebuild with a real reason behind it" named
|
||||
above. Nothing else on the infra side needs coordinating.
|
||||
|
||||
## Access
|
||||
|
||||
Repo is not anonymously readable (a bare clone 403s). Operator granted
|
||||
|
||||
@@ -109,11 +109,11 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-08-19 (late) — a long infra session, all of it landed and committed: **fleet `*.internal` DNS is live** (git-sourced, 42 names, three resolvers incl. a NEW colo one), **waterland studio containerised** on irv-ml1, **Homepage cleaned up and themed** with Australis Skyfall + an Arbo-generated background, and **four unmanaged stacks adopted** into `stacks/` (two of which turned out to be quietly broken — SearXNG's healthcheck, SeaFile down 3 months). ⚠ 17 commits unpushed. ⚠ 2 unread althing messages from waterland-dev left deliberately undrained — operator deferred the inbox to after the `/clear`._
|
||||
_As of 2026-08-19 (late) — a long infra session, all of it landed and committed: **fleet `*.internal` DNS is live** (git-sourced, 42 names, three resolvers incl. a NEW colo one), **waterland studio containerised** on irv-ml1, **Homepage cleaned up and themed** with Australis Skyfall + an Arbo-generated background, and **four unmanaged stacks adopted** into `stacks/` (two of which turned out to be quietly broken — SearXNG's healthcheck, SeaFile down 3 months). ⚠ 17 commits unpushed. ~~⚠ 2 unread althing messages from waterland-dev left deliberately undrained~~ — **drained 2026-08-19 post-`/clear`; thread closed, see below.**_
|
||||
|
||||
- **🟢 FLEET `.internal` DNS — LIVE 2026-08-19.** `<host>.<site>.internal`, sites `ana`/`esh`/`nh3`. `dns/internal.yaml` is the source of truth; `scripts/dns-sync.py` reconciles the three AdGuard resolvers (diff → prompt → apply, idempotent). 42 names resolving from all three sites. **Colo got its first resolver ever** (`stacks/adguard-ana/`, API on **8053** not 8080, no blocklists by design) — before this, ana-docker resolved straight against `1.1.1.1`. Auth = a dedicated `infra-ops` AdGuard user, password vaulted `nh3-dev/adguard-infra-ops-password`. **⏳ TWO OPEN, both operator's to schedule:** (a) colo hosts still point at `1.1.1.1` so they do not yet *use* the new resolver — repointing a site's DNS is a separate change; (b) the static-v6 convention (each server at its site's `/64` with low bits echoing the v4 octet, `esh-docker-vm` → `…::45`) is **proposed, not ruled on**. The `v6:` column is empty and correct — no fleet host has a global v6 address yet. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md`
|
||||
|
||||
- **🟢 WATERLAND STUDIO — containerised + live, irv-ml1:8410.** Displaced a bare `nohup` (PID 1283383). Warm 256²+anim render **6.1s**, beating the 7.4s bare-metal figure. `claude-bot` now has **read** on `vh/waterland` (operator-granted); token on irv-ml1 root-owned 0600, repo-scoped helper, nothing in `.git/config`. **⏳ Reported upstream, NOT patched by us:** the on-disk job store grows without bound across restarts (`RETAIN=40` eviction only sees the in-memory dict). → `persistent-memory.d/2026-08-19-waterland-studio-containerised.md`
|
||||
- **🟢 WATERLAND STUDIO — containerised + live, irv-ml1:8410.** Displaced a bare `nohup` (PID 1283383). Warm 256²+anim render **6.1s**, beating the 7.4s bare-metal figure. `claude-bot` now has **read** on `vh/waterland` (operator-granted); token on irv-ml1 root-owned 0600, repo-scoped helper, nothing in `.git/config`. **⏳ Reported upstream, NOT patched by us:** the on-disk job store grows without bound across restarts (`RETAIN=40` eviction only sees the in-memory dict) — waterland-dev **confirmed it as a real defect**, and the **operator green-lit their startup-rehydrate fix** — PR expected (61 MB / 16 dirs, no clock on it). Upstream `main` is now `464dfc2` (PR #5 fixes the uv-prune and CUDA-header landmines at source); **the container deliberately stays on `8025366`** and keeps its `[ctk]` install + `UV_NO_SYNC`/`UV_OFFLINE` pins as defence-in-depth. **⏳ NEXT: when the rehydrate PR merges, one `update.sh` on irv-ml1 lands it and `464dfc2` together** — that merge is the rebuild trigger. Thread closed. → `persistent-memory.d/2026-08-19-waterland-studio-containerised.md`
|
||||
|
||||
- **🟢 HOMEPAGE — cleaned + themed (Australis Skyfall).** Three real defects fixed (UltraSeedbox on all tabs, Uptime Kuma double-rendered, fiction column counts), AI tab reordered by clickability, then themed from the operator's Skyfall handoff bundle with a background generated by **Arbo** (`t2i-ui-background`, job `13f0891f4e42`). ⚠ **After any recreate the tab bar/wallpaper/i18n vanish for up to ~an hour and then return on their own — do not chase it.** ⚠ CSS is served per-request: a theme change needs a **reload**, not a recreate, and candidate CSS can be injected live via Playwright for seconds-long iteration. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md`
|
||||
|
||||
|
||||
Reference in New Issue
Block a user