memory: snapshot — .internal DNS live, waterland containerised, homepage themed
Captures a long infra session: fleet *.internal DNS (git-sourced, 42 names, three resolvers including a new colo one), waterland studio containerised on irv-ml1, Homepage cleaned up and themed with Australis Skyfall over an Arbo-generated background, and four unmanaged stacks adopted into stacks/. Four detail files added. Auto-archived 4 entries to archival-memory.md (Recent decisions 2, Tried and abandoned 2); 5 held back by the open-deferred guard rather than moved. Also records three operator-owned open items: the colo DNS repoint, the static-v6 convention, and the deliberately belayed AI-tab Dormant regrouping.
This commit is contained in:
@@ -1,15 +0,0 @@
|
||||
`[2026-07-31]` **kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap; fix relayed to heid, NOT applied gateway-side.**
|
||||
|
||||
heid reported that `kimi-k3` (the primary route = Kimi Code coding endpoint `openai/k3` @ `api.kimi.com/coding/v1`) silently degraded its cross-frontier panel: on large/reasoning-heavy dispatches, `completion_tokens: 16381` **exactly**, `content` empty, `reasoning_content` ~64KB, `finish_reason: **stop**` (a truncation mislabeled as a clean stop). `max_tokens: 100000` in the request was not honored.
|
||||
|
||||
**Investigation arc (a clean cross-frontier-triage + verify-on-the-wire case):**
|
||||
1. My first read: a flat ~16384 OUTPUT cap; fix = a LiteLLM `stop→length` relabel callback (heid's fallback ask). Confirmed the cap isn't in our LiteLLM config (no `max_tokens` clamp on the route).
|
||||
2. Operator routed a fix-research pass to **dvalin-smithy-dev + bil-smithy-dev** (independent). Both CONVERGED (docs-based): `max_tokens` is a deprecated alias on Kimi/Moonshot; the canonical field is `max_completion_tokens` (default 131072, max 1M); the coding endpoint defaults output to 16384; fix = send `max_completion_tokens` + `reasoning_effort` via `extra_body` (drop_params-safe).
|
||||
3. **heid's live data REFUTED the docs hypothesis:** a later dispatch hit `completion_tokens: 18455` (ABOVE 16384) cleanly, with `reasoning_tokens: 16198` (just under 16384) and content present. So COMPLETION is uncapped; the bound is on **REASONING at ~16384**. When a hard task's thinking exhausts that budget, nothing's left for content → empty answer under `stop`.
|
||||
4. **I proved it on the wire** — ran heid's real 500KB failing bundle direct at both endpoints (bypassing LiteLLM so `reasoning_effort` isn't dropped): default effort → 504/timeout (the failure); **`reasoning_effort: low` → reasoning ~12–13.5k (under the ceiling), content returns (6–7.6k chars)**, on BOTH coding AND general endpoints. So re-routing to the general endpoint buys nothing — the fix is the effort param, and it works on the wire.
|
||||
|
||||
**THE FIX (caller-side, no shared-gateway change/restart):** send `reasoning_effort` via **`extra_body`** on kimi-k3 dispatches (`low` for large bundles). LiteLLM `drop_params: true` strips the top-level `reasoning_effort` — which is exactly why heid's earlier `reasoning_effort: low` was a no-op. `extra_body` survives drop_params (the house GLM-thinking pattern). Tradeoff: low effort = shallower reasoning, but a complete answer beats today's empty one.
|
||||
|
||||
**Relayed to heid to validate on a real round** (the one unconfirmed hop is whether `extra_body` survives OUR LiteLLM). **Backstop if it doesn't:** add `allowed_openai_params: ["reasoning_effort"]` to the `kimi-k3` route in the gateway config — that IS a shared-gateway change + a ~10s restart (blips all consumers), so it needs a heads-up.
|
||||
|
||||
Gateway = LiteLLM on ana-docker `10.250.50.70:4000`; kimi-k3 config in `stacks/litellm/conf/config.yaml` (see Recent-decisions `[2026-07-25]` Kimi K3 wiring). No gateway change was made this session. Failing dispatch on record: `01KYTASKTY3T` (jackdaw-dev bug-hunt).
|
||||
@@ -0,0 +1,119 @@
|
||||
# `[2026-08-19]` Fleet `.internal` DNS — git-sourced, agent-managed, three resolvers
|
||||
|
||||
Operator: *"with ipv6 i can't memorize the IP addresses anymore. need a way to
|
||||
keep track of local .internal dns names that can be agent managed and is
|
||||
lightweight."* Built and live in one session; commit `b8003c7`.
|
||||
|
||||
## Shape
|
||||
|
||||
```
|
||||
dns/internal.yaml source of truth — 38 hosts + 4 service aliases
|
||||
scripts/dns-sync.py reconciles AdGuard resolvers against it
|
||||
stacks/adguard-ana/ the colo's resolver, which did not exist
|
||||
dns/README.md workflow, naming, the IPv6 caveat
|
||||
```
|
||||
|
||||
Deliberately the same posture as `deploy-stack.sh`: the file is intent, the
|
||||
resolvers are derived state, you see a diff before anything changes.
|
||||
`--dry-run` / `--yes` / `--site <s>`. Verified idempotent — a second run prints
|
||||
`nothing to do`.
|
||||
|
||||
Naming is `<host>.<site>.internal` with sites **`ana` / `esh` / `nh3`**
|
||||
(operator's call). `.internal` is ICANN-reserved for private use since 2024;
|
||||
`.local` is reserved for mDNS, which is why the pre-existing
|
||||
`searxng.pfi.local` was a standards collision that merely happened to work.
|
||||
|
||||
Every name is published to **every** resolver — the site label says where a
|
||||
host *is*, not which resolver knows about it.
|
||||
|
||||
## The framing correction that mattered most
|
||||
|
||||
The ask reads as "I can't memorise v6 addresses", but the deeper problem is
|
||||
that **v6 addresses are derived, not assigned**, so they cannot reliably be
|
||||
*written down once* either. SLAAC gives EUI-64 (MAC-coupled) or
|
||||
privacy-extension (rotating) addresses, and UniFi has **no v6 equivalent of a
|
||||
DHCP reservation** — so a hand-maintained v6 table rots on its own.
|
||||
|
||||
⇒ The fix has two halves and only the second is DNS: (1) pin static v6 on
|
||||
server-class hosts, (2) then the name table is just a file. Surfaced to the
|
||||
operator before building.
|
||||
|
||||
**Verified 2026-08-19: no fleet host has a global v6 address at all yet** —
|
||||
ESH's `/56` is live only on `esh-cameras`, NH3's LANs are back to
|
||||
`ipv6_interface_type: none`, the colo has none. So the `v6:` column ships
|
||||
EMPTY and correct, and the naming layer was built first rather than blocking
|
||||
on v6. Names established now need no renaming when addresses land.
|
||||
|
||||
Suggested convention when they do (awaiting operator): each server static at
|
||||
its site's `/64` with low-order bits echoing the v4 host octet —
|
||||
`esh-docker-vm` at `…::45` — so addresses are declarable *and* semi-memorable.
|
||||
|
||||
## Two properties not to break
|
||||
|
||||
**Authority is scoped to the ZONE, not the resolver.** Only rewrites ending in
|
||||
`.internal` are managed. ESH's resolver turned out to carry three hand-made
|
||||
`esteban.net` rewrites (`eshnas`, `brotherprinter`, `eshhome`) — **my first
|
||||
read of the config missed them**, because an `awk` range on `rewrites:` matched
|
||||
an empty-looking block. A resolver-wide authoritative sync would have silently
|
||||
deleted all three on first run. Verified intact after sync.
|
||||
|
||||
**Within `.internal` it IS authoritative** — names added by hand in the AdGuard
|
||||
UI get deleted by the next sync. That is the point: one place to look.
|
||||
|
||||
## The colo had no resolver at all
|
||||
|
||||
ESH and NH3 each ran AdGuard; **ana-docker resolved straight against
|
||||
`1.1.1.1`**, so the colo had no way to answer for internal names. Closed with
|
||||
`stacks/adguard-ana/`.
|
||||
|
||||
⚠️ Its API is on **8053**, not 8080 — `:8080` and `:3000` were already taken on
|
||||
that busy host. The port is therefore carried **per-site in the yaml**, not
|
||||
assumed by the script, so the odd one out cannot be forgotten.
|
||||
|
||||
⚠️ It ships with **no blocklists**, deliberately. The other two filter ads for
|
||||
human browsing; this one resolves for a rack of servers, where a blocklist
|
||||
false-positive breaks service-to-service calls at 3am for no upside.
|
||||
|
||||
First boot uses a **seed config** (`conf/AdGuardHome.seed.yaml`) copied into
|
||||
the conf volume before first start, so the container comes up configured
|
||||
instead of sitting in the setup wizard.
|
||||
|
||||
## Credential — service account, not the operator's
|
||||
|
||||
Added a dedicated **`infra-ops`** AdGuard user to all three resolvers rather
|
||||
than asking for the `lkraven` password (per the standing migrate-off-operator-
|
||||
creds directive). Password vaulted at
|
||||
`nh3-dev/adguard-infra-ops-password`; `lkraven` untouched; pre-change configs
|
||||
backed up on each host as `AdGuardHome.yaml.bak-preinfraops-*`. Both existing
|
||||
resolvers kept answering across the restart.
|
||||
|
||||
Two landmines worth keeping:
|
||||
|
||||
- **Go's bcrypt rejects `htpasswd`'s `$2y$` prefix.** Same algorithm, different
|
||||
marker; `golang.org/x/crypto/bcrypt` accepts only `$2a$`/`$2b$`. Normalise
|
||||
the prefix, and self-verify the hash with `htpasswd -vb` BEFORE installing it
|
||||
on a live resolver.
|
||||
- **The vault appends a trailing newline on `get`.** A password carrying a
|
||||
stray `\n` fails auth in a way that looks exactly like a wrong password.
|
||||
`dns-sync.py` strips it.
|
||||
|
||||
## `pfi.local` migration — and the one that must NOT move
|
||||
|
||||
`searxng.pfi.local` → `searxng.ana.internal`, with the **old `Host()` kept
|
||||
alongside** in the Traefik rule so nothing breaks mid-migration; both return
|
||||
200. Drop the fallback once the access log shows the old name unused.
|
||||
|
||||
**`matrix.pfi.local` deliberately NOT migrated.** A Matrix `server_name` is
|
||||
baked into every user ID, room ID and signing key, and federation identity
|
||||
derives from it — renaming it is not a DNS change, it is rebuilding the
|
||||
homeserver's identity and invalidating its history. The operator approved
|
||||
"migrate pfi.local" generally; this was surfaced as a deliberate exclusion
|
||||
rather than executed blindly.
|
||||
|
||||
## Still open
|
||||
|
||||
Colo hosts still point at `1.1.1.1`, so they do not yet *use* the new resolver
|
||||
— it only answers what asks it directly. Repointing a whole site's DNS is a
|
||||
bigger change than standing the service up, and is the operator's to schedule.
|
||||
|
||||
See also [[2026-08-17-fleet-ipv6-mesh]].
|
||||
@@ -0,0 +1,145 @@
|
||||
# `[2026-08-19]` Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background
|
||||
|
||||
Commits `9d92c4b`, `c3de7db`, `45c1995`, `f38cf69`, `df68dd2`.
|
||||
|
||||
## The cleanup (three real defects)
|
||||
|
||||
- **UltraSeedbox rendered on all four tabs.** The bookmark group had no entry in
|
||||
`settings.yaml`'s `layout:` block at all, and Homepage's documented behaviour
|
||||
is that a group with no `tab:` is shown on **every** tab. Pinned to Main.
|
||||
⚠️ This will happen again to the next group added without a `tab:` — the rule
|
||||
is now written at the top of the layout block.
|
||||
- **Uptime Kuma rendered twice** — a manual `services.yaml` entry under
|
||||
Monitoring *and* `homepage.group=Apps` on the container. Exactly the
|
||||
"never list a labelled container manually" failure the stack README warns
|
||||
about; it survived the previous day's audit because a duplicate reads as two
|
||||
plausible cards rather than as an error. Manual block deleted, label moved to
|
||||
`Monitoring`, `homepage.siteMonitor` added.
|
||||
- **Column counts were fiction** — several groups declared more columns than
|
||||
they had members, so the last row of each was dead space (Notes: 1 card in a
|
||||
4-wide row). Columns now track member counts; `GET /api/services` prints the
|
||||
live per-group counts and is the check.
|
||||
|
||||
Later, on operator instruction, the **AI tab was reordered by clickability**:
|
||||
Gateways & Chat → Image & Media → Audio Tools on top, then the vLLM `/docs`
|
||||
seats and TTS endpoints. Reasoning written into the config so it survives:
|
||||
order by "would I click this?", not by how central the service is.
|
||||
|
||||
## ⚠️ The expensive red herring — the tab bar after a recreate
|
||||
|
||||
After a recreate the client render comes up with **no tab bar, no wallpaper and
|
||||
no i18n** (search box shows the raw key `search.search`), groups falling back to
|
||||
side-by-side columns. **It restores itself with no intervention.**
|
||||
|
||||
Timing, measured rather than assumed: a fresh container was still tab-less at
|
||||
**4m30s, twice**; it was healthy again after roughly an hour. `docker ps`
|
||||
reporting `healthy` says nothing about it — the container is serving, the page
|
||||
is just wrong.
|
||||
|
||||
An hour went into ruling out four causes that were never the cause:
|
||||
|
||||
1. **Not the config** — restoring `settings.yaml` *and* `services.yaml` to
|
||||
their committed versions reproduces it, as does the pre-adoption backup in
|
||||
`/opt/docker-bu/conf/homepage/`.
|
||||
2. **Not the v2.0.0 release** — a throwaway container on `v1.13.2` shows
|
||||
identical symptoms, and the image never changed anyway (working and broken
|
||||
both report `v2.0.0` / rev `17456f2`).
|
||||
3. **Not `PUID`/`PGID`**, and not Docker discovery — tested both, and with the
|
||||
socket unmounted entirely.
|
||||
4. **Not server-side** — the server-rendered HTML still contains the tab
|
||||
markup, the background URL and `useEqualHeights`; `GET /api/validate`
|
||||
returns `[]`. The loss is client-side, with no page error, no failed chunk
|
||||
and no non-200.
|
||||
|
||||
Every throwaway container in that list was judged within ~30s of starting, so
|
||||
they were all inside the same window — and that consistency **read as a
|
||||
reproduction when it was the same measurement mistake five times over.**
|
||||
|
||||
**Operative rule: recreate, walk away, re-check later. Do not chase it.**
|
||||
|
||||
## ⚠️ The iteration loop that would have prevented the overcook
|
||||
|
||||
`custom.css` is served **per request** from `/api/config/custom.css`, so a CSS
|
||||
change needs a **browser reload** — not a container recreate, and it never owed
|
||||
the layout warm-up above. Conflating the two costs ~10 operator-visible minutes
|
||||
per attempt (operator called this out directly).
|
||||
|
||||
Faster still, and how the final pass was done: **inject candidate CSS into the
|
||||
running page and screenshot it** —
|
||||
`await p.addStyleTag({content: css})` in Playwright against the live
|
||||
dashboard. Seconds per iteration, no deploy. Build + deploy only once the
|
||||
render looks right.
|
||||
|
||||
## The theme — Australis Skyfall
|
||||
|
||||
Operator supplied a Claude Design handoff bundle via the Booth (`26-copper`).
|
||||
Skyfall is a dual-theme OKLCH system: one lightness law across every chromatic
|
||||
family (deep 0.48 / base 0.66 / bright 0.80), all hues cooler than neutral, a
|
||||
Sea neutral ramp drifting ice-blue→ocean-green as it brightens, and a
|
||||
"calm depth" language of **hairline + two-layer shadow on every elevated
|
||||
surface, never one without the other**.
|
||||
|
||||
```
|
||||
theme/colors.css layout.css typography.css vendored VERBATIM from the bundle
|
||||
theme/fonts/Supreme-{400,500,700}.woff2 the body/UI face
|
||||
theme/skyfall.css.in the Homepage bindings (ours)
|
||||
theme/build.py → conf/custom.css (generated — do not hand-edit)
|
||||
```
|
||||
|
||||
The build step exists for one reason: **Homepage serves only `custom.css` and
|
||||
`custom.js` out of its config dir**, with no static route beside them, so a
|
||||
`@font-face` pointing at a vendored `.woff2` would 404 — the face must arrive
|
||||
as a data URI. The background image takes the other road, because
|
||||
`/app/public/images` **is** a real static route (mounted read-only in
|
||||
`compose.yaml`).
|
||||
|
||||
Only Supreme is embedded: a link dashboard has no display type, and Victor
|
||||
Mono ships as 2.4 MB TTF statics per cut — 30x the whole stylesheet for a
|
||||
handful of latency figures.
|
||||
|
||||
## The background is generated, not stock
|
||||
|
||||
**Arbo as an image-gen engine** (the operator's actual ask, which I first
|
||||
misread as "use Arbo's palette" and had to redo). Arbo's `t2i-ui-background`
|
||||
workflow is purpose-built: *"abstract full-bleed backgrounds, no subject"*.
|
||||
Job `13f0891f4e42`, seed 26, flux2-klein-9b, 2048×1152, 1.6 MB PNG → **22 KB
|
||||
WebP** (smooth gradients compress absurdly well).
|
||||
|
||||
⚠️ Arbo API gotcha: `prompt` is a **discriminated union, not a string** — a
|
||||
bare string 422s. `{"kind":"raw","text":…,"negative":…}` is the shape.
|
||||
|
||||
## Two documented deviations from the design system
|
||||
|
||||
1. **Skyfall forbids this background.** Its rule is "flat semantic surfaces; no
|
||||
photography, no textures", with one permitted motif — a subtle aurora
|
||||
gradient on hero/empty-state areas only, *"never behind body text blocks"*.
|
||||
A dashboard is a body-text block. Present on the operator's explicit
|
||||
instruction, mitigated rather than excused: abstract, no subject, strictly
|
||||
cool temperature, held at **`opacity: 30`**. That number is load-bearing —
|
||||
at 14 the aurora was invisible, and turning it up makes the cards fight the
|
||||
ribbon.
|
||||
2. **Service icons stay full-colour vendor logos.** Desaturating them from CSS
|
||||
only makes them illegible.
|
||||
|
||||
## Overcorrection, and the colour pass
|
||||
|
||||
First stat-well pass went from `font-thin` 13px straight to **bold 22px in
|
||||
heading white** — operator: *"went from subtle to BASH YOU OVER THE HEAD."*
|
||||
The principle missed: a stat only has to out-rank **its own label**, not the
|
||||
service name above it. Now `--text-md` medium in cyan.
|
||||
|
||||
Colour was then lifted **from inside the system**: Skyfall names Aurora (blue,
|
||||
cyan, green) the *primary* families, "used generously, in that order", while
|
||||
Dawn (amber/red/violet) is semantic-only. So group markers cycle
|
||||
blue→cyan→green down the page (icons full strength, names at 0.72), service
|
||||
icons take a single cool wash, latency tags move to the info family so
|
||||
"how fast" stops looking like "is it alive". **No Dawn colour is used
|
||||
decoratively anywhere.**
|
||||
|
||||
Two DOM findings that made it possible:
|
||||
|
||||
- **Homepage renders mdi icons as a gradient behind an SVG mask** — recolour
|
||||
via `background`, not `color`.
|
||||
- **Homepage emits `docker-status-<state>`, not `status-<state>`.** The
|
||||
original selectors matched nothing, so every green pill up to that point was
|
||||
stock colouring rather than the theme. Both forms are now matched.
|
||||
@@ -0,0 +1,77 @@
|
||||
# `[2026-08-19]` Four unmanaged stacks found on live hosts — and two of them were quietly broken
|
||||
|
||||
Commits `42c594c`, `dc3e47b`, plus `uptimekuma` in `9d92c4b`.
|
||||
|
||||
## The pattern worth remembering
|
||||
|
||||
Chasing two bad-looking cards on the dashboard turned up **four stacks running
|
||||
on fleet hosts with no canonical copy anywhere**: `uptimekuma` and (already
|
||||
known) the two AdGuards on esh-docker-vm, `searxng` and `seafile` on
|
||||
ana-docker, and `heretic2-charrp-reasoning` on ana-ml2 (untracked in git).
|
||||
|
||||
⇒ **A dashboard card is a cheap census of what is actually running.** When
|
||||
something on it looks wrong, check whether the stack behind it is even in
|
||||
`stacks/` before debugging the symptom — twice here the answer was "no", and
|
||||
the fix belonged in version control as much as on the host.
|
||||
|
||||
Adopted: `stacks/uptimekuma/`, `stacks/searxng/`, `stacks/seafile/`,
|
||||
`stacks/heretic2-charrp-reasoning/`. ESH/NH3 AdGuard compose files were
|
||||
**deliberately left unmanaged** — adopting three live resolvers while also
|
||||
introducing a new DNS naming system is two risky changes at once.
|
||||
|
||||
## SearXNG — the healthcheck was eating itself
|
||||
|
||||
Card flapped UNHEALTHY; the container was fine the whole time. The compose
|
||||
passed `--tries` and `--spider` as **two separate argv entries**, so wget
|
||||
consumed `--spider` as the *value* of `--tries`. Spider mode never engaged,
|
||||
which means every probe since April **downloaded** the healthz response to a
|
||||
file:
|
||||
|
||||
```
|
||||
295,287 healthz.N files in the container's working directory
|
||||
```
|
||||
|
||||
With that many files, wget's scan for the next free filename is what
|
||||
intermittently blew the 10s timeout. **Self-worsening — every probe made the
|
||||
next one slower.** Restored `--tries=1`; the junk lived in the writable layer
|
||||
so the recreate cleared it. Now `healthy`, `fails=0`, 200 in 0.16s.
|
||||
|
||||
Lesson: an argv list in YAML has no shell to catch a missing `=`. A flag that
|
||||
silently swallows the next argument turns a liveness probe into a workload.
|
||||
|
||||
## SeaFile — not broken, never restarted
|
||||
|
||||
Card showed EXITED for three months. **None of the three services declared a
|
||||
restart policy**, so Docker defaulted them to `no`. On
|
||||
**2026-05-06T21:27:45Z** the daemon stopped all three within 200ms of each
|
||||
other — a daemon restart or host reboot — and nothing brought them back.
|
||||
|
||||
⚠️ **Exit code `255` is a red herring**: it is what a container that ignores
|
||||
SIGTERM reports when the daemon stops it, **not** evidence of a crash. Reading
|
||||
it as one sends you hunting a bug that does not exist. The tell was all three
|
||||
services stopping within 200ms.
|
||||
|
||||
Added `restart: unless-stopped` to all three; brought up; mariadb gated on its
|
||||
healthcheck exactly as the existing `depends_on` comments intended, seahub
|
||||
started without the race, `302` → login page. Data was in local named volumes,
|
||||
not on the ana-nas NFS, so nothing was at risk.
|
||||
|
||||
Three months of silent downtime whose only signal was a card nobody read as an
|
||||
outage — the argument for semantic status colour on the dashboard (see
|
||||
[[2026-08-19-homepage-skyfall-theme]], where amber EXITED pills made six
|
||||
mis-grouped AI seats obvious at a glance).
|
||||
|
||||
## heretic2-charrp-reasoning — tracked, with its shim
|
||||
|
||||
The `char-rp-reasoning` seat (NEO-CODE Heretic2 27B, modelopt NVFP4 + grafted
|
||||
BF16 MTP head, ~77 tok/s via `qwen3_5_mtp` spec-decode) had been running
|
||||
untracked. Now in `stacks/`, including
|
||||
`conf/mtp-workaround/sitecustomize.py`, which is **not optional**: vLLM 0.24.0
|
||||
does not propagate modelopt `exclude_modules` to the spec-decode **draft**
|
||||
model, so the BF16 MTP head gets quantized and the engine dies at load. Both
|
||||
the mount and `PYTHONPATH` are load-bearing.
|
||||
|
||||
Added the two files house convention expects and the directory lacked — a
|
||||
`.env.example` naming every knob (all values are compose defaults; the host
|
||||
overrides only the three VRAM ones) and a README pointing at
|
||||
`docs/runbooks/heretic2-nvfp4-mtp-seat.md` rather than duplicating it.
|
||||
@@ -0,0 +1,118 @@
|
||||
# `[2026-08-19]` waterland studio containerised on irv-ml1 — three landmines, all measured
|
||||
|
||||
Handover from `waterland-dev` over althing (thread `01M0CDRGEZWAJCEJXXMQWXV80F`):
|
||||
a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare
|
||||
`nohup` (PID 1283383) that would not survive a reboot. Now
|
||||
`stacks/waterland-studio/`, `restart: unless-stopped`, healthy on
|
||||
irv-ml1:8410. Commits `a2b5b58`, `8189076`.
|
||||
|
||||
Tracking `main` per operator: PR #4 merged and `main` HEAD is exactly the
|
||||
pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway.
|
||||
|
||||
## Build context lives OUTSIDE the compose dir — on purpose
|
||||
|
||||
`/opt/waterland-studio/src` is the checkout; the Dockerfile is passed
|
||||
out-of-context from `/opt/docker/compose/waterland-studio/`. **`deploy-stack.sh`
|
||||
rsyncs `stacks/<stack>/` with `--delete`**, so a checkout kept beside
|
||||
`compose.yaml` would be destroyed by the next deploy of this stack. `update.sh`
|
||||
refreshes source → rebuild → recreate → health, and is verified end to end.
|
||||
|
||||
## Landmine 1 — both uv extras are load-bearing at BUILD *and* RUN
|
||||
|
||||
`gpu` carries `cupy-cuda12x`; a bare `uv sync` prunes it and the renderer
|
||||
silently drops to the numpy path at ~21x wall time — it does not error, it
|
||||
just gets slow. waterland-dev warned about the build side.
|
||||
|
||||
The runtime side is worse and was not in the handover: **`studio/jobs.py`
|
||||
shells the renderer out as a literal `uv run waterland ...` with no `--extra`
|
||||
flags** (`cwd=WATERLAND_STUDIO_REPO`). Left alone, uv re-syncs the project
|
||||
mid-job to its default extras and prunes cupy back out from under a correctly
|
||||
built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the
|
||||
pin ever stops holding the job fails **loudly** instead of quietly rebuilding a
|
||||
slower environment.
|
||||
|
||||
## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare
|
||||
|
||||
Every render died 1.7s in with:
|
||||
|
||||
```
|
||||
RuntimeError: Failed to find CUDA headers.
|
||||
```
|
||||
|
||||
printed **through argparse's usage banner**, which makes it read like a CLI
|
||||
argument bug rather than a missing toolkit. That misdirection is the reason
|
||||
this is written down.
|
||||
|
||||
cupy compiles kernels at runtime through NVRTC, which needs the toolkit
|
||||
**headers** — not just the driver and the runtime libs bundled in the
|
||||
`cupy-cuda12x` wheel. irv-ml1 has a CUDA toolkit installed system-wide, so the
|
||||
bare `nohup` process found them **by accident**; a slim image has none.
|
||||
|
||||
Fixed with `uv pip install "cupy-cuda12x[ctk]"` — headers as wheels, a few
|
||||
hundred MB against ~6 GB for a `-devel` base image. It runs **after**
|
||||
`uv sync`, because sync prunes what it does not know about.
|
||||
|
||||
Reported upstream: it is an undeclared runtime dependency of the `gpu` extra,
|
||||
and anyone running this without a system toolkit hits it.
|
||||
|
||||
## Landmine 3 — the GPU index inside the container is not the host's
|
||||
|
||||
The app pins `CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects
|
||||
`CUDA_VISIBLE_DEVICES_TARGET` (default `1`, correct on the host, where
|
||||
`nvidia-smi` shows A6000 at 1). Compose exposes **exactly one** GPU
|
||||
(`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the
|
||||
container that card is index **0** ⇒ `CUDA_VISIBLE_DEVICES_TARGET=0`. Copying
|
||||
the host's value selects a device that does not exist. Host device 0 is the
|
||||
3090, which carries the TTS zoo and must not be touched.
|
||||
|
||||
## Cold start is ~17s of NVRTC compile → `/root/.cupy` is a volume
|
||||
|
||||
| job | wall |
|
||||
|---|---|
|
||||
| 256² + anim, cold container | 23.3 s |
|
||||
| 256² + anim, warm | **6.1 s** |
|
||||
| 256² plate only (`--codec none`) | 3.9 s |
|
||||
| 512² plate only | 6.4 s |
|
||||
|
||||
Warm beats the **7.4 s** recorded against the bare-metal process, so
|
||||
containerising cost nothing. Verified the cache volume properly: recreate
|
||||
(fresh cache → 23.2 s first render) then restart (populated → 6.0 s). Without
|
||||
it every restart makes the next user wait 4x and the service merely *looks*
|
||||
slow.
|
||||
|
||||
## Upstream finding — the on-disk job store grows without bound
|
||||
|
||||
`JobStore._jobs` is a plain dict and **nothing scans `WATERLAND_STUDIO_DATA` at
|
||||
startup**. Consequences:
|
||||
|
||||
1. After a restart `/api/jobs` lists only jobs created since — cosmetic, and
|
||||
how this was spotted: the API reported **1 job** while the volume held all
|
||||
**16 directories, 60.6 MB**. Not data loss.
|
||||
2. The real one: `RETAIN = 40` eviction only ever iterates the in-memory dict,
|
||||
so directories orphaned by a restart are **never reclaimed**. The
|
||||
handover's "bounded around 500 MB" holds within a single process lifetime;
|
||||
across restarts the store grows monotonically at ~12 MB per animated job.
|
||||
|
||||
Reported to waterland-dev with evidence; **not patched from the infra side** —
|
||||
it is their code. Prune the volume by hand if it bites first.
|
||||
|
||||
## Access
|
||||
|
||||
Repo is not anonymously readable (a bare clone 403s). Operator granted
|
||||
**`claude-bot` read on `vh/waterland`** — verified `admin: False, push: False,
|
||||
pull: True`. Token on irv-ml1 at
|
||||
`/root/.config/waterland-studio/git-credentials`, `0600` root-owned, wired as a
|
||||
**repo-scoped** credential helper; `.git/config` carries no token (verified),
|
||||
so the remote stays clean in any diff or backup. The operator's `vh`
|
||||
site-admin token was used only for the initial clone and the grant itself and
|
||||
was **never written to disk on that host** — a site-admin credential on a GPU
|
||||
box is a blast radius nobody needs for a read-only fetch.
|
||||
|
||||
## Constraints honoured as stated (not inferred)
|
||||
|
||||
- **Serial by design — one replica, one card.** A render is 20–45s of near-full
|
||||
GPU with a single worker thread. Two on the same A6000 would OOM or thrash.
|
||||
Throughput is a hardware conversation, not a replica-count one.
|
||||
- **No authentication, arbitrary file uploads** ⇒ stays inside the
|
||||
LAN/WireGuard boundary. Do **not** paper over it with a proxy password;
|
||||
waterland-dev offered to add a real auth layer if wider reach is ever needed.
|
||||
Reference in New Issue
Block a user