memory: snapshot — .internal DNS live, waterland containerised, homepage themed
Captures a long infra session: fleet *.internal DNS (git-sourced, 42 names, three resolvers including a new colo one), waterland studio containerised on irv-ml1, Homepage cleaned up and themed with Australis Skyfall over an Arbo-generated background, and four unmanaged stacks adopted into stacks/. Four detail files added. Auto-archived 4 entries to archival-memory.md (Recent decisions 2, Tried and abandoned 2); 5 held back by the open-deferred guard rather than moved. Also records three operator-owned open items: the colo DNS repoint, the static-v6 convention, and the deliberately belayed AI-tab Dormant regrouping.
This commit is contained in:
@@ -0,0 +1,118 @@
|
||||
# `[2026-08-19]` waterland studio containerised on irv-ml1 — three landmines, all measured
|
||||
|
||||
Handover from `waterland-dev` over althing (thread `01M0CDRGEZWAJCEJXXMQWXV80F`):
|
||||
a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare
|
||||
`nohup` (PID 1283383) that would not survive a reboot. Now
|
||||
`stacks/waterland-studio/`, `restart: unless-stopped`, healthy on
|
||||
irv-ml1:8410. Commits `a2b5b58`, `8189076`.
|
||||
|
||||
Tracking `main` per operator: PR #4 merged and `main` HEAD is exactly the
|
||||
pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway.
|
||||
|
||||
## Build context lives OUTSIDE the compose dir — on purpose
|
||||
|
||||
`/opt/waterland-studio/src` is the checkout; the Dockerfile is passed
|
||||
out-of-context from `/opt/docker/compose/waterland-studio/`. **`deploy-stack.sh`
|
||||
rsyncs `stacks/<stack>/` with `--delete`**, so a checkout kept beside
|
||||
`compose.yaml` would be destroyed by the next deploy of this stack. `update.sh`
|
||||
refreshes source → rebuild → recreate → health, and is verified end to end.
|
||||
|
||||
## Landmine 1 — both uv extras are load-bearing at BUILD *and* RUN
|
||||
|
||||
`gpu` carries `cupy-cuda12x`; a bare `uv sync` prunes it and the renderer
|
||||
silently drops to the numpy path at ~21x wall time — it does not error, it
|
||||
just gets slow. waterland-dev warned about the build side.
|
||||
|
||||
The runtime side is worse and was not in the handover: **`studio/jobs.py`
|
||||
shells the renderer out as a literal `uv run waterland ...` with no `--extra`
|
||||
flags** (`cwd=WATERLAND_STUDIO_REPO`). Left alone, uv re-syncs the project
|
||||
mid-job to its default extras and prunes cupy back out from under a correctly
|
||||
built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the
|
||||
pin ever stops holding the job fails **loudly** instead of quietly rebuilding a
|
||||
slower environment.
|
||||
|
||||
## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare
|
||||
|
||||
Every render died 1.7s in with:
|
||||
|
||||
```
|
||||
RuntimeError: Failed to find CUDA headers.
|
||||
```
|
||||
|
||||
printed **through argparse's usage banner**, which makes it read like a CLI
|
||||
argument bug rather than a missing toolkit. That misdirection is the reason
|
||||
this is written down.
|
||||
|
||||
cupy compiles kernels at runtime through NVRTC, which needs the toolkit
|
||||
**headers** — not just the driver and the runtime libs bundled in the
|
||||
`cupy-cuda12x` wheel. irv-ml1 has a CUDA toolkit installed system-wide, so the
|
||||
bare `nohup` process found them **by accident**; a slim image has none.
|
||||
|
||||
Fixed with `uv pip install "cupy-cuda12x[ctk]"` — headers as wheels, a few
|
||||
hundred MB against ~6 GB for a `-devel` base image. It runs **after**
|
||||
`uv sync`, because sync prunes what it does not know about.
|
||||
|
||||
Reported upstream: it is an undeclared runtime dependency of the `gpu` extra,
|
||||
and anyone running this without a system toolkit hits it.
|
||||
|
||||
## Landmine 3 — the GPU index inside the container is not the host's
|
||||
|
||||
The app pins `CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects
|
||||
`CUDA_VISIBLE_DEVICES_TARGET` (default `1`, correct on the host, where
|
||||
`nvidia-smi` shows A6000 at 1). Compose exposes **exactly one** GPU
|
||||
(`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the
|
||||
container that card is index **0** ⇒ `CUDA_VISIBLE_DEVICES_TARGET=0`. Copying
|
||||
the host's value selects a device that does not exist. Host device 0 is the
|
||||
3090, which carries the TTS zoo and must not be touched.
|
||||
|
||||
## Cold start is ~17s of NVRTC compile → `/root/.cupy` is a volume
|
||||
|
||||
| job | wall |
|
||||
|---|---|
|
||||
| 256² + anim, cold container | 23.3 s |
|
||||
| 256² + anim, warm | **6.1 s** |
|
||||
| 256² plate only (`--codec none`) | 3.9 s |
|
||||
| 512² plate only | 6.4 s |
|
||||
|
||||
Warm beats the **7.4 s** recorded against the bare-metal process, so
|
||||
containerising cost nothing. Verified the cache volume properly: recreate
|
||||
(fresh cache → 23.2 s first render) then restart (populated → 6.0 s). Without
|
||||
it every restart makes the next user wait 4x and the service merely *looks*
|
||||
slow.
|
||||
|
||||
## Upstream finding — the on-disk job store grows without bound
|
||||
|
||||
`JobStore._jobs` is a plain dict and **nothing scans `WATERLAND_STUDIO_DATA` at
|
||||
startup**. Consequences:
|
||||
|
||||
1. After a restart `/api/jobs` lists only jobs created since — cosmetic, and
|
||||
how this was spotted: the API reported **1 job** while the volume held all
|
||||
**16 directories, 60.6 MB**. Not data loss.
|
||||
2. The real one: `RETAIN = 40` eviction only ever iterates the in-memory dict,
|
||||
so directories orphaned by a restart are **never reclaimed**. The
|
||||
handover's "bounded around 500 MB" holds within a single process lifetime;
|
||||
across restarts the store grows monotonically at ~12 MB per animated job.
|
||||
|
||||
Reported to waterland-dev with evidence; **not patched from the infra side** —
|
||||
it is their code. Prune the volume by hand if it bites first.
|
||||
|
||||
## Access
|
||||
|
||||
Repo is not anonymously readable (a bare clone 403s). Operator granted
|
||||
**`claude-bot` read on `vh/waterland`** — verified `admin: False, push: False,
|
||||
pull: True`. Token on irv-ml1 at
|
||||
`/root/.config/waterland-studio/git-credentials`, `0600` root-owned, wired as a
|
||||
**repo-scoped** credential helper; `.git/config` carries no token (verified),
|
||||
so the remote stays clean in any diff or backup. The operator's `vh`
|
||||
site-admin token was used only for the initial clone and the grant itself and
|
||||
was **never written to disk on that host** — a site-admin credential on a GPU
|
||||
box is a blast radius nobody needs for a read-only fetch.
|
||||
|
||||
## Constraints honoured as stated (not inferred)
|
||||
|
||||
- **Serial by design — one replica, one card.** A render is 20–45s of near-full
|
||||
GPU with a single worker thread. Two on the same A6000 would OOM or thrash.
|
||||
Throughput is a hardware conversation, not a replica-count one.
|
||||
- **No authentication, arbitrary file uploads** ⇒ stays inside the
|
||||
LAN/WireGuard boundary. Do **not** paper over it with a proxy password;
|
||||
waterland-dev offered to add a real auth layer if wider reach is ever needed.
|
||||
Reference in New Issue
Block a user