fc834a8a23d4ffbf3f8a48f6fa36b3c580e9493d
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
e0d1c44137 |
chore(fleet): repoint stale irv-ml1 refs (10.100.79.3 -> irv-ml1.nh3.internal)
The 2026-09-06 headscale cutover retired irv-ml1's wg0 tunnel IP 10.100.79.3 (now 10.6.110.50). Repointed all LIVE canonical refs to the DNS NAME so the next move can't re-break them: homepage.href/siteMonitor labels across 25 stack composes, load-bearing env defaults (asset-engine INFERENCE_HOST, open-webui AUDIO_TTS_OPENAI_API_BASE_URL, skaldsong SKALDSONG_TTS_BASE_URL, zonos-gateway ZONOS_URL, dia), homepage services.yaml manual cards (Voice Design Studio, IRV-ML1), and servers/irv-ml1/ssh-target. Updated the stale 'WG tunnel' comment to the mesh reality. Left as-is: README curl-examples and .env.example comments (docs), and historical mentions in CLAUDE.md/persistent-memory. NOTE: applying the label repoints to the RUNNING irv-ml1 containers needs a recreate per service (labels read at creation); deployed .env values are separate from these canonical defaults. |
||
|
vh
|
39da1d4a97 |
feat(homepage): recategorise on "do I open this?", collapse the API groups
The board mixed tools with endpoints. A vLLM seat whose href is a /docs page
sat in the same band as ComfyUI; the MQTT broker and the RustDesk relay, which
have no page at all, sat in Apps; and `Service Networking` was thirteen members
spanning three AdGuards, five Dockges, two Traefiks and four headless agents.
Every group is now one of two kinds and they never mix. TOOLS are expanded and
sit at the top of their tab. ENDPOINTS — an API, a broker, a background agent,
an href that is /docs or /ping or nothing — carry `initiallyCollapsed: true`
and sit at the bottom. Collapsed is not hidden: the eyebrow and its rule still
render, so the tab still says the thing exists and one click expands it.
A second rule fell out of the same pass and now shapes the group boundaries: a
group's members should all carry a widget or none should. A stat strip makes a
card ~50px taller, so one widget card in a row of plain ones opens a void under
the plain ones. That is why AdGuard and Traefik get their own groups rather
than sharing one with Dockge, and it is most of why the old Service Networking
band looked broken. AdGuard (ANA) was the last short card in its row and now
carries the same query/blocked/latency strip as its two siblings — one
infra-ops AdGuard login authenticates against all three instances, verified
against each; it lives in that stack's .env on the host and is vaulted.
The sixteen GPU-backed model seats were deliberately NOT relabelled.
`homepage.group` is read at container creation, so clearer names for
`AI - Inference` and friends would have cost a recreate on six vLLM seats, four
eval seats and four TTS engines — multi-minute model reloads on endpoints peers
reach through the gateway. Order plus `initiallyCollapsed` buys the same
separation for nothing, so those names stay as they are on purpose.
28 containers that ARE cheap to bounce were relabelled, across five hosts, via
rerunnable elway playbooks. Their label steps are gated on the old value still
being present, so a second run reports skipped rather than churning. Two verify
steps were wrong on first contact and are fixed with the reason recorded: the
traefik check raced its own recreate, and asserting a model seat is "running"
cannot answer "did I bounce it" when a seat may be legitimately stopped —
container age can, and now does.
The canonical stacks/ tree was synced to the deployed labels afterwards, so
intent and reality agree again on all fourteen tracked stacks.
Also documents the real nature of the post-recreate blank dashboard, which cost
~25 minutes here and an hour on 2026-08-19. `initialSettings":{}` in the served
HTML is the catch branch of the page's data loader, not a warm-up and not a
cache — and the error can vanish entirely, because the logger is assigned inside
the same try and the catch only logs if the logger exists. Ruled out by
measurement this time: all four API routes return 200 with correct content while
the page serves {}, and the previous known-good settings.yaml reproduces it
identically. The README now carries the one-command test and the next lead.
Before/after, all four tabs: http://10.100.10.50:8090/b/homepage-relayout/
|
||
|
vh
|
e6907819b0 |
feat(waterland-studio): deploy b72425b — all three upstream findings fixed
One update.sh run on irv-ml1 carried both open upstream PRs, per the operator's green-light on the job-store fix: - #5 (464dfc2) declares cupy-cuda12x[ctk] on the gpu extra and takes uv out of the render path (sys.executable -m waterland.cli), retiring the runtime prune trap at the source. - #6 (b72425b) rehydrates the job index from the data volume at startup, fixing the unbounded store growth reported from this side. Verified after the update rather than assumed: healthy on backend cupy; /api/jobs went 1 -> 16 against 16 directories on disk, so API and volume agree for the first time; nothing wrongly reclaimed, correct since 16 is under RETAIN=40 and adoption only makes them visible; a real 256^2 plate render completes warm, so the kernel-cache volume survived the image swap. A subsequent render took both counts to 17. The image keeps its explicit [ctk] install and UV_NO_SYNC/UV_OFFLINE pins even though both are now redundant. The header requirement is a property of this slim base, not of the upstream extra, and the cost is measured rather than assumed: uv sync satisfies it first, so the line reports "Audited 1 package" and adds 0.3s to the build. The env pins are now cheap defence-in-depth against any future path that re-enters uv. Docs corrected in place: the README's upstream-finding section is now a resolved-finding record, and the two "bounded ~500 MB" claims say which commit made that bound hold across restarts rather than only within a process. Comment-side changes pushed to the live compose dir; no restart was needed for them. |
||
|
vh
|
a2b5b58eee |
feat(waterland-studio): containerise the GPU render service on irv-ml1
Replaces a bare nohup on irv-ml1:8410 that would not have survived a reboot, handed over by waterland-dev. Tracks vh/waterland @ main (PR #4 merged; main HEAD is exactly the pinned 8025366). Build context is a checkout at /opt/waterland-studio/src, deliberately OUTSIDE the compose dir — deploy-stack.sh rsyncs stacks/<stack>/ with --delete and would otherwise eat it. The Dockerfile is passed out-of-context. Three landmines, all measured: 1. Both uv extras are load-bearing at build AND run. jobs.py shells the renderer out as a literal with no --extra flags, so uv would re-sync at runtime and prune cupy — silently dropping to the numpy path at ~21x wall time. UV_NO_SYNC pins it; UV_OFFLINE makes any failure loud instead of quietly slow. 2. cupy needs CUDA HEADERS for its NVRTC compile, not just the driver and the wheel's runtime libs. The host has a system CUDA toolkit so the nohup process found them by accident; a slim image does not, and every render died 1.7s in with 'Failed to find CUDA headers' printed through argparse's usage banner — which reads like a CLI bug, not a missing toolkit. Fixed with cupy-cuda12x[ctk] (hundreds of MB, vs ~6 GB for a -devel base). 3. The A6000 is host device 1 but container device 0, since compose exposes exactly one GPU. CUDA_VISIBLE_DEVICES_TARGET=0 inside; copying the host's value selects a device that does not exist. /root/.cupy is a volume because the NVRTC compile costs ~17s: verified at 23.3s cold vs 6.1s warm, and re-verified across a restart (23.2s on a fresh cache volume, 6.0s once populated). Warm 256^2+anim beats the 7.4s recorded against bare metal, so containerising cost nothing. Job store seeded with the 4 jobs from the displaced instance. Serial by design (one replica, one card) and unauthenticated, so it stays LAN/WireGuard-only. |