From 39da1d4a971a3a5d93ea2b55a69267312a14047a Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 24 Aug 2026 08:54:06 -0700 Subject: [PATCH] feat(homepage): recategorise on "do I open this?", collapse the API groups MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The board mixed tools with endpoints. A vLLM seat whose href is a /docs page sat in the same band as ComfyUI; the MQTT broker and the RustDesk relay, which have no page at all, sat in Apps; and `Service Networking` was thirteen members spanning three AdGuards, five Dockges, two Traefiks and four headless agents. Every group is now one of two kinds and they never mix. TOOLS are expanded and sit at the top of their tab. ENDPOINTS — an API, a broker, a background agent, an href that is /docs or /ping or nothing — carry `initiallyCollapsed: true` and sit at the bottom. Collapsed is not hidden: the eyebrow and its rule still render, so the tab still says the thing exists and one click expands it. A second rule fell out of the same pass and now shapes the group boundaries: a group's members should all carry a widget or none should. A stat strip makes a card ~50px taller, so one widget card in a row of plain ones opens a void under the plain ones. That is why AdGuard and Traefik get their own groups rather than sharing one with Dockge, and it is most of why the old Service Networking band looked broken. AdGuard (ANA) was the last short card in its row and now carries the same query/blocked/latency strip as its two siblings — one infra-ops AdGuard login authenticates against all three instances, verified against each; it lives in that stack's .env on the host and is vaulted. The sixteen GPU-backed model seats were deliberately NOT relabelled. `homepage.group` is read at container creation, so clearer names for `AI - Inference` and friends would have cost a recreate on six vLLM seats, four eval seats and four TTS engines — multi-minute model reloads on endpoints peers reach through the gateway. Order plus `initiallyCollapsed` buys the same separation for nothing, so those names stay as they are on purpose. 28 containers that ARE cheap to bounce were relabelled, across five hosts, via rerunnable elway playbooks. Their label steps are gated on the old value still being present, so a second run reports skipped rather than churning. Two verify steps were wrong on first contact and are fixed with the reason recorded: the traefik check raced its own recreate, and asserting a model seat is "running" cannot answer "did I bounce it" when a seat may be legitimately stopped — container age can, and now does. The canonical stacks/ tree was synced to the deployed labels afterwards, so intent and reality agree again on all fourteen tracked stacks. Also documents the real nature of the post-recreate blank dashboard, which cost ~25 minutes here and an hour on 2026-08-19. `initialSettings":{}` in the served HTML is the catch branch of the page's data loader, not a warm-up and not a cache — and the error can vanish entirely, because the logger is assigned inside the same try and the catch only logs if the logger exists. Ruled out by measurement this time: all four API routes return 200 with correct content while the page serves {}, and the previous known-good settings.yaml reproduces it identically. The README now carries the one-command test and the next lead. Before/after, all four tabs: http://10.100.10.50:8090/b/homepage-relayout/ --- .../2026-08-24-homepage-uniform-grid.md | 94 ++++++++ persistent-memory.md | 3 +- playbooks/homepage-regroup-ana-docker.yaml | 226 ++++++++++++++++++ playbooks/homepage-regroup-ana-ml2.yaml | 74 ++++++ playbooks/homepage-regroup-esh-docker-vm.yaml | 135 +++++++++++ playbooks/homepage-regroup-irv-ml1.yaml | 115 +++++++++ playbooks/homepage-regroup-nh3-docker.yaml | 58 +++++ stacks/adguard-ana/.env.example | 16 ++ stacks/adguard-ana/compose.yaml | 18 +- stacks/arbo/compose.yaml | 2 +- stacks/comfyui/compose.yaml | 2 +- stacks/dockge/compose.yaml | 2 +- stacks/gitea-runner/compose.yaml | 2 +- stacks/homepage/README.md | 127 +++++++++- stacks/homepage/conf/services.yaml | 44 ++-- stacks/homepage/conf/settings.yaml | 180 +++++++++----- stacks/lobe-chat/compose.yaml | 2 +- stacks/memos/compose.yaml | 2 +- stacks/miniflux/compose.yaml | 2 +- stacks/open-webui/compose.yaml | 4 +- stacks/rest-server-ana/compose.yaml | 2 +- stacks/scriberr/compose.yaml | 17 +- stacks/searxng/compose.yaml | 2 +- stacks/waterland-studio/compose.yaml | 2 +- stacks/zed-fim-proxy/compose.yaml | 2 +- 25 files changed, 1022 insertions(+), 111 deletions(-) create mode 100644 playbooks/homepage-regroup-ana-docker.yaml create mode 100644 playbooks/homepage-regroup-ana-ml2.yaml create mode 100644 playbooks/homepage-regroup-esh-docker-vm.yaml create mode 100644 playbooks/homepage-regroup-irv-ml1.yaml create mode 100644 playbooks/homepage-regroup-nh3-docker.yaml create mode 100644 stacks/adguard-ana/.env.example diff --git a/persistent-memory.d/2026-08-24-homepage-uniform-grid.md b/persistent-memory.d/2026-08-24-homepage-uniform-grid.md index 5fec4c2..cabdb15 100644 --- a/persistent-memory.d/2026-08-24-homepage-uniform-grid.md +++ b/persistent-memory.d/2026-08-24-homepage-uniform-grid.md @@ -116,3 +116,97 @@ The Main tab still opens on three sparse bands: **Notes** (1 member) and News, or folding Games into Apps, would tighten the top of the page — but that is information architecture, not layout, and the group names are the operator's. Surfaced rather than done. + +→ **Resolved in pass 2 below**, where the operator delegated the naming +("re-categorize however you want"). Notes + News became `Daily`, Games folded +into `Apps`, and the `AI - Audio Tools` placement in this pass was superseded +(Scriberr moved on to `AI - Studios`). + +--- + +# `[2026-08-24, pass 2]` Recategorised on "do I open this?", API groups collapsed + +Operator, after seeing pass 1: *"You can re-categorize however you want. +service networking tab is uneven, you can split out the adguard cards, etc. +most of the issues are that tools I use and have a UI are interspersed with API +endpoints which are largely informational only. They might even go in their own +cards or start collapsed."* + +## The axis + +Every group is now either **tools** (expanded, top of tab) or **endpoints** (an +API, a broker, an agent — `initiallyCollapsed: true`, bottom of tab). A +collapsed group still renders its eyebrow and rule, so presence costs one line +instead of two rows. + +Second, quieter rule that fell out of the same pass: **a group's members should +all carry a widget or none should.** A stat strip adds ~50px, so one widget card +in a row of plain ones opens a void under the plain ones — which is most of what +made the 13-member `Service Networking` band look broken. + +## Shape + +- **Main** — `Daily` (Memos, Miniflux, Nevermore, SearXNG — replaces the + 1-card Notes and 2-card News bands), `Monitoring`, `Apps` (12; absorbed the + 1-card `Games` band), `Media`, `UltraSeedbox`. +- **AI** — `AI - Gateways & Chat` (8) and `AI - Studios` (6) expanded; then + `AI - Inference` (7), `AI - Eval & Retrieval` (4), `AI - Speech (TTS)` (4), + `AI - Audio Tools` (2), `AI - Dormant` (6) all collapsed. +- **Toolchain** — `DNS & Filtering` (3), `Reverse Proxies` (2), + `Compose Consoles` (5), `Toolchain` (3), `Agents (no UI)` (6, collapsed). +- **Infrastructure** — unchanged; every card there is already a console. + +Measured after: every group `card=360`, and `DNS & Filtering` and +`Reverse Proxies` both `h=134..134` — dead flush. + +## ⚠ The move that made it affordable + +**The sixteen GPU-backed model seats were NOT relabelled.** `homepage.group` is +read at container **creation**, so renaming `AI - Inference` to something +clearer would have meant recreating six vLLM seats plus four eval seats plus +four TTS engines — multi-minute model reloads on endpoints peers reach through +the gateway. Order plus `initiallyCollapsed` buys the same separation for free, +so the names stay ugly on purpose. **Do not spend that recreate on a label.** + +28 containers *were* relabelled — all cheap web services — via five rerunnable +elway playbooks, `playbooks/homepage-regroup-.yaml`. The canonical +`stacks/` tree was synced to match afterwards, so intent and reality agree. + +`initiallyCollapsed: true` is a per-group key in `layout:`; confirmed present in +this build (`defaultOpen: !(group?.initiallyCollapsed ?? global)` in +`/app/.next/server/pages/index.js`). + +## AdGuard (ANA) gained its widget, and the credential is fleet-wide + +It was the only AdGuard without a query/blocked/latency strip, so it sat short +beside two tall siblings. **One `infra-ops` AdGuard login authenticates against +all three instances** (ANA `:8053`, NH3 `:8080`, ESH `:8080` — all returned 200 +on `POST /control/login`, verified 2026-08-24). Vaulted at +`secret get nh3-dev/adguard-infra-ops-password`; written to +`/opt/docker/compose/adguard-ana/.env` (0600, root) and never into git. Its icon +was also the odd one out (`mdi-dns` against two `si-adguard`). + +## ⚠⚠ `initialSettings":{}` — the tab-bar mystery is a SWALLOWED EXCEPTION + +The biggest durable finding of the day, and it cost ~25 minutes. Full write-up +in `stacks/homepage/README.md`; the short version: + +`initialSettings":{}` in the served HTML is **the catch branch** of the page's +data loader, not a warm-up and not a cache. And the error can vanish without +trace: the logger is assigned as the first statement *inside* the same `try`, +and the `catch` only logs `if (logger)`. If the logger is what threw, nothing is +written anywhere — which is exactly what was observed. + +Ruled out by measurement, do not re-run: `/api/services`, `/api/bookmarks`, +`/api/widgets` and `/api/hash` all return **200 with correct content** while the +page serves `{}`; restoring the previous known-good `settings.yaml` reproduces +it identically; `/api/validate` returns `[]`; disk and permissions are fine. + +**One-command test:** +`curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}'` + +**What broke the streak:** three consecutive recreates came up empty, then +rolling the 8.6 MB `conf/homepage/logs/homepage.log` aside and recreating healed +it within 15 seconds. That is one observation, not proof — but it is a coherent +mechanism (oversized log → logger init throws → silent catch) and it is the +cheapest thing to try first next time. diff --git a/persistent-memory.md b/persistent-memory.md index 2ebc10f..b40fe67 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -110,7 +110,7 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-08-24 — a very long ops session; everything it opened, it closed. Nothing is mid-flight. The **homepage** dashboard pass is done (below); the only thing left open on it is an information-architecture question for the operator, not work._ -- **🟢 Homepage relaid out on a UNIFORM 4-column grid, and its remote-label discovery re-verified.** `/api/services` proves the board reads container labels off **all five** Docker engines — 74 of 107 cards are label-discovered and only 13 come from its own host. ⚠ **"columns = member count" is RETIRED**; per-group column tuning is what made the card width change five times down the page. Also fixed: long names printing under their own status pill (6 cards, AI tab), the `AI Systems`/Scriberr all-tabs leak, and grey-smudge icons. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` +- **🟢 Homepage rebuilt in two passes: uniform 4-column grid, then recategorised on "do I open this?"** Groups are now either TOOLS (expanded, top of tab) or ENDPOINTS (`initiallyCollapsed: true`, bottom); `Service Networking` split four ways; 28 containers relabelled via rerunnable `playbooks/homepage-regroup-.yaml`. Remote-label discovery re-verified — the board reads labels off **all five** Docker engines (74 of 107 cards, only 13 from its own host). ⚠ **The 16 GPU model seats were deliberately NOT relabelled** — order + collapse buys the same separation without a model reload. ⚠ **"columns = member count" is RETIRED.** ⚠ `initialSettings":{}` in the served HTML is a **swallowed exception**, not a warm-up. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` - **🔒 ana-gw's public admin surface is now ZERO open TCP ports**, and the box is scheduled for replacement by **OPNsense on a Dell R420** (which brings WireGuard onto the edge — so the downstream-WireGuard-VM design is moot, do not scope it). WAN admin was opened as a cutover contingency then closed on instruction; the FortiOS **ACME listener** on :80 was disabled too. **No out-of-band path to ana-gw remains** — if both tunnels drop it is console-only; re-open is two one-liners in auto-memory. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md` @@ -136,6 +136,7 @@ _As of 2026-08-24 — a very long ops session; everything it opened, it closed. ## Recent decisions +- `[2026-08-24]` **Homepage reorganised on the axis "do I open this?" — UI groups expanded on top, API/agent groups collapsed at the bottom** (operator-delegated: "re-categorize however you want"). Load-bearing constraint: `homepage.group` is read at container CREATION, so the 16 GPU-backed model seats keep their unlovely names rather than eat a recreate — `initiallyCollapsed` + order is free. Second rule discovered here: **group members should all have widgets or none should**, because a stat strip adds ~50px and opens a void beside plain cards. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` - `[2026-08-24]` **Homepage columns unified at 4 for every group; the 2026-08-18 "columns = member count" rule is retired.** It was avoiding dead cells in a short last row and bought a worse defect — card width changing at every group boundary. Also carries two CSS traps: `overflow: hidden` clips at the PADDING box (so a `padding-right` gutter is spill room, not a guard), and a `:root` override of a Homepage theme variable is silently outranked by `.theme-slate` on the same `` element. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` - `[2026-08-24]` **AES-128 adopted on both Anaheim tunnels; the per-flow ceiling root-caused to the UDM's software AES-CBC, exonerating the FortiGate.** Proven by an A/B/A cipher swap at identical CPU — hardware offload is not cipher-cost-sensitive. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` - `[2026-08-24]` **ana-gw's public admin surface closed to zero open ports, ACME listener included.** Two of my diagnoses were wrong first (an "ISP proxy" that was the FortiGate, and an "all-port VIP" alarm that was a parser gap) — both from reading config instead of measuring from outside. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md` diff --git a/playbooks/homepage-regroup-ana-docker.yaml b/playbooks/homepage-regroup-ana-docker.yaml new file mode 100644 index 0000000..e40dacc --- /dev/null +++ b/playbooks/homepage-regroup-ana-docker.yaml @@ -0,0 +1,226 @@ +# Homepage recategorisation — ana-docker (10.250.50.70), 13 containers. +# +# Splits the dashboard on ONE axis: do you open this thing, or is it an +# endpoint you only want to know is alive? See the layout: block in +# stacks/homepage/conf/settings.yaml for the target shape. +# +# memos, miniflux, nevermore, searxng -> Daily (was Notes / News / Apps) +# zed-fim-proxy -> AI - Inference (no UI; href is /ping) +# adguardhome -> DNS & Filtering +# traefik -> Reverse Proxies +# dockge -> Compose Consoles +# crowdsec, mailrise, rest-server, +# gitea-runner, hbbr (RustDesk relay) -> Agents (no UI) +# +# `homepage.group` is read at container CREATION, so each edit is followed by +# `compose up -d ` — a restart would leave the old label in place. +# Both halves are idempotent: the sed is gated on the old value still being +# present, and `up -d` is a no-op when the container already matches its spec. +# +# Run: scripts/elway infra-ops@10.250.50.70 --playbook playbooks/homepage-regroup-ana-docker.yaml + +steps: + # ---- label edits ------------------------------------------------------- + - name: memos -> Daily + sudo: true + shell: >- + sed -i 's|homepage.group=Notes$|homepage.group=Daily|' + /opt/docker/compose/memos/compose.yaml + when: grep -q 'homepage.group=Notes$' /opt/docker/compose/memos/compose.yaml + + - name: miniflux -> Daily + sudo: true + shell: >- + sed -i 's|homepage.group=News$|homepage.group=Daily|' + /opt/docker/compose/miniflux/compose.yaml + when: grep -q 'homepage.group=News$' /opt/docker/compose/miniflux/compose.yaml + + - name: nevermore -> Daily + sudo: true + shell: >- + sed -i 's|homepage.group=News$|homepage.group=Daily|' + /opt/docker/compose/nevermore/compose.yaml + when: grep -q 'homepage.group=News$' /opt/docker/compose/nevermore/compose.yaml + + - name: searxng -> Daily + sudo: true + shell: >- + sed -i 's|homepage.group=Apps$|homepage.group=Daily|' + /opt/docker/compose/searxng/compose.yaml + when: grep -q 'homepage.group=Apps$' /opt/docker/compose/searxng/compose.yaml + + - name: zed-fim-proxy -> AI - Inference + sudo: true + shell: >- + sed -i 's|homepage.group=AI - Gateways . Chat$|homepage.group=AI - Inference|' + /opt/docker/compose/zed-fim-proxy/compose.yaml + when: grep -q 'homepage.group=AI - Gateways . Chat$' /opt/docker/compose/zed-fim-proxy/compose.yaml + + - name: adguardhome -> DNS & Filtering + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=DNS \& Filtering|' + /opt/docker/compose/adguard-ana/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/adguard-ana/compose.yaml + + - name: traefik -> Reverse Proxies + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Reverse Proxies|' + /opt/docker/compose/traefik/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/traefik/compose.yaml + + - name: dockge -> Compose Consoles + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|' + /opt/docker/compose/dockge/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml + + - name: crowdsec -> Agents (no UI) + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Agents (no UI)|' + /opt/docker/compose/crowdsec/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/crowdsec/compose.yaml + + - name: mailrise -> Agents (no UI) + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Agents (no UI)|' + /opt/docker/compose/mailrise/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/mailrise/compose.yaml + + - name: rest-server -> Agents (no UI) + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Agents (no UI)|' + /opt/docker/compose/rest-server-ana/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/rest-server-ana/compose.yaml + + - name: gitea-runner -> Agents (no UI) + sudo: true + shell: >- + sed -i 's|homepage.group=Toolchain$|homepage.group=Agents (no UI)|' + /opt/docker/compose/gitea-runner/compose.yaml + when: grep -q 'homepage.group=Toolchain$' /opt/docker/compose/gitea-runner/compose.yaml + + - name: rustdesk (hbbr) -> Agents (no UI) + sudo: true + shell: >- + sed -i 's|homepage.group=Apps$|homepage.group=Agents (no UI)|' + /opt/docker/compose/rustdesk/compose.yaml + when: grep -q 'homepage.group=Apps$' /opt/docker/compose/rustdesk/compose.yaml + + # Two containers were both named plain "Open WebUI" and, once the ESH one + # joined this group, they landed side by side — same name, same icon family, + # only the description telling them apart. Site suffix, like Traefik/Dockge/ + # AdGuard already carry. + - name: openwebui (ana) -> "Open WebUI (ana)" + sudo: true + shell: >- + sed -i 's|homepage.name=Open WebUI$|homepage.name=Open WebUI (ana)|' + /opt/docker/compose/openwebui/compose.yaml + when: grep -q 'homepage.name=Open WebUI$' /opt/docker/compose/openwebui/compose.yaml + + # ---- recreates --------------------------------------------------------- + # traefik goes LAST: crowdsec is its bouncer, so bounce the bouncer first + # and let traefik come up against a settled agent. + - name: recreate memos + sudo: true + shell: cd /opt/docker/compose/memos && docker compose up -d memos + + - name: recreate miniflux + sudo: true + shell: cd /opt/docker/compose/miniflux && docker compose up -d miniflux + + - name: recreate nevermore-web + sudo: true + shell: cd /opt/docker/compose/nevermore && docker compose up -d nevermore-web + + - name: recreate searxng + sudo: true + shell: cd /opt/docker/compose/searxng && docker compose up -d searxng + + - name: recreate zed-fim-proxy + sudo: true + shell: cd /opt/docker/compose/zed-fim-proxy && docker compose up -d zed-fim-proxy + + - name: recreate mailrise + sudo: true + shell: cd /opt/docker/compose/mailrise && docker compose up -d mailrise + + - name: recreate rest-server + sudo: true + shell: cd /opt/docker/compose/rest-server-ana && docker compose up -d rest-server + + - name: recreate gitea-runner + sudo: true + shell: cd /opt/docker/compose/gitea-runner && docker compose up -d runner + + - name: recreate rustdesk relay + sudo: true + shell: cd /opt/docker/compose/rustdesk && docker compose up -d hbbr + + - name: recreate openwebui (ana) + sudo: true + shell: cd /opt/docker/compose/openwebui && docker compose up -d open-webui + + - name: recreate dockge + sudo: true + shell: cd /opt/docker/compose/dockge && docker compose up -d dockge + + - name: recreate adguardhome + sudo: true + shell: cd /opt/docker/compose/adguard-ana && docker compose up -d adguardhome + + - name: recreate crowdsec + sudo: true + shell: cd /opt/docker/compose/crowdsec && docker compose up -d crowdsec + + - name: recreate traefik + sudo: true + shell: cd /opt/docker/compose/traefik && docker compose up -d traefik + +verify: + - name: every relabelled container now carries its new group + sudo: true + changed_when: "false" + shell: >- + docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}' + memos miniflux nevermore-web searxng zed-fim-proxy adguardhome traefik + dockge crowdsec mailrise rest-server gitea-runner hbbr + + - name: no container is left in the retired Service Networking group + sudo: true + changed_when: "false" + shell: >- + test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')" + + # dig is not installed everywhere in the fleet, so fall back to the AdGuard + # UI — a resolver that serves its own dashboard on :8053 has come back up. + - name: adguard is back (DNS answer, or its UI if dig is absent) + changed_when: "false" + shell: >- + if command -v dig >/dev/null 2>&1; + then dig +short +time=3 +tries=2 @10.250.50.70 gitea.phasefinal.com | grep -q .; + else curl -sf -o /dev/null -m 8 http://10.250.50.70:8053/; fi + + # Retried, not one-shot: the first run of this playbook checked 0.12s after + # `Started` and got rc=7 while traefik was still binding. The container was + # fine — `:8380/` 301s to /dashboard/ and both public hostnames answered 200 + # seconds later. A recreate needs a moment; assert the settled state. + - name: traefik still routes + changed_when: "false" + shell: >- + for i in 1 2 3 4 5 6 7 8 9 10; do + curl -sfL -o /dev/null -m 5 http://127.0.0.1:8380/dashboard/ && exit 0; + sleep 3; done; exit 1 + + - name: everything is running + sudo: true + changed_when: "false" + shell: >- + test "$(docker inspect -f '{{.State.Running}}' memos miniflux nevermore-web + searxng zed-fim-proxy adguardhome traefik dockge crowdsec mailrise + rest-server gitea-runner hbbr | sort -u)" = "true" diff --git a/playbooks/homepage-regroup-ana-ml2.yaml b/playbooks/homepage-regroup-ana-ml2.yaml new file mode 100644 index 0000000..988c565 --- /dev/null +++ b/playbooks/homepage-regroup-ana-ml2.yaml @@ -0,0 +1,74 @@ +# Homepage recategorisation — ana-ml2 (10.250.50.54), 2 containers. +# Sibling of playbooks/homepage-regroup-ana-docker.yaml; rationale lives there. +# +# scriberr -> AI - Studios (a transcription UI you open, not an API seat) +# dockge -> Compose Consoles +# +# ⚠ THE vLLM SEATS ON THIS HOST ARE DELIBERATELY NOT TOUCHED. Every one of them +# would need a recreate to change its `homepage.group`, and a recreate means a +# multi-minute model reload on a seat that peers reach through the gateway. The +# separation the operator asked for — UI up top, API endpoints out of the way — +# is achieved for those groups by ORDER and `initiallyCollapsed` in +# stacks/homepage/conf/settings.yaml, which costs nothing. Keep it that way: if +# a future pass wants to rename `AI - Inference`, weigh it against bouncing six +# model seats. +# +# Run: scripts/elway infra-ops@10.250.50.54 --playbook playbooks/homepage-regroup-ana-ml2.yaml + +steps: + - name: scriberr -> AI - Studios + sudo: true + shell: >- + sed -i 's|homepage.group=AI - Audio Tools$|homepage.group=AI - Studios|' + /opt/docker/compose/scriberr/compose.yaml + when: grep -q 'homepage.group=AI - Audio Tools$' /opt/docker/compose/scriberr/compose.yaml + + - name: dockge -> Compose Consoles + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|' + /opt/docker/compose/dockge/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml + + - name: recreate dockge + sudo: true + shell: cd /opt/docker/compose/dockge && docker compose up -d dockge + + - name: recreate scriberr + sudo: true + shell: cd /opt/docker/compose/scriberr && docker compose up -d scriberr + +verify: + - name: every relabelled container now carries its new group + sudo: true + changed_when: "false" + shell: >- + docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}' + scriberr dockge + + - name: no container is left in the retired Service Networking group + sudo: true + changed_when: "false" + shell: >- + test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')" + + # Container age, not liveness — a seat may be legitimately stopped, so + # "is it running" cannot answer "did I bounce it". See the same step in + # playbooks/homepage-regroup-irv-ml1.yaml for how that distinction was found. + - name: the vLLM seats were NOT recreated by this run + sudo: true + changed_when: "false" + shell: >- + for c in $(docker ps -a --filter 'name=vllm-' --filter 'name=llama-' + --format '{{.Names}}'); do + created=$(docker inspect -f '{{.Created}}' "$c" 2>/dev/null) || continue; + age=$(( $(date +%s) - $(date -d "$created" +%s) )); + if [ "$age" -lt 600 ]; then echo "$c was recreated ${age}s ago"; exit 1; fi; + done + + - name: scriberr answers + changed_when: "false" + shell: >- + for i in 1 2 3 4 5 6 7 8 9 10 11 12; do + curl -sfL -o /dev/null -m 5 http://127.0.0.1:8080/ && exit 0; + sleep 5; done; exit 1 diff --git a/playbooks/homepage-regroup-esh-docker-vm.yaml b/playbooks/homepage-regroup-esh-docker-vm.yaml new file mode 100644 index 0000000..bd7256a --- /dev/null +++ b/playbooks/homepage-regroup-esh-docker-vm.yaml @@ -0,0 +1,135 @@ +# Homepage recategorisation — esh-docker-vm (10.0.50.45), 6 containers. +# Sibling of playbooks/homepage-regroup-ana-docker.yaml; the rationale, the +# label-at-creation constraint and the idempotency scheme are documented there. +# +# lobe-chat, open-webui -> AI - Gateways & Chat (chat frontends belong +# with the other chat frontends, not in Apps) +# adguardhome -> DNS & Filtering +# traefik -> Reverse Proxies +# dockge -> Compose Consoles +# mosquitto -> Agents (no UI) (an MQTT broker has no page) +# +# ⚠ adguard and traefik here use `docker-compose.yml`, not `compose.yaml`. +# +# Run: scripts/elway infra-ops@10.0.50.45 --playbook playbooks/homepage-regroup-esh-docker-vm.yaml + +steps: + - name: lobe-chat -> AI - Gateways & Chat + sudo: true + shell: >- + sed -i 's|homepage.group=Apps$|homepage.group=AI - Gateways \& Chat|' + /opt/docker/compose/lobe-chat/compose.yaml + when: grep -q 'homepage.group=Apps$' /opt/docker/compose/lobe-chat/compose.yaml + + - name: open-webui -> AI - Gateways & Chat + sudo: true + shell: >- + sed -i 's|homepage.group=Apps$|homepage.group=AI - Gateways \& Chat|' + /opt/docker/compose/open-webui/compose.yaml + when: grep -q 'homepage.group=Apps$' /opt/docker/compose/open-webui/compose.yaml + + # Site suffix — the ana instance is also called "Open WebUI" and the two now + # sit side by side in the same group. See the sibling step in + # playbooks/homepage-regroup-ana-docker.yaml. + - name: open-webui -> "Open WebUI (esh)" + sudo: true + shell: >- + sed -i 's|homepage.name=Open WebUI$|homepage.name=Open WebUI (esh)|' + /opt/docker/compose/open-webui/compose.yaml + when: grep -q 'homepage.name=Open WebUI$' /opt/docker/compose/open-webui/compose.yaml + + - name: mosquitto -> Agents (no UI) + sudo: true + shell: >- + sed -i 's|homepage.group=Apps$|homepage.group=Agents (no UI)|' + /opt/docker/compose/mosquitto/compose.yaml + when: grep -q 'homepage.group=Apps$' /opt/docker/compose/mosquitto/compose.yaml + + - name: adguardhome -> DNS & Filtering + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=DNS \& Filtering|' + /opt/docker/compose/adguard/docker-compose.yml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/adguard/docker-compose.yml + + - name: traefik -> Reverse Proxies + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Reverse Proxies|' + /opt/docker/compose/traefik/docker-compose.yml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/traefik/docker-compose.yml + + - name: dockge -> Compose Consoles + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|' + /opt/docker/compose/dockge/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml + + # ---- recreates --------------------------------------------------------- + - name: recreate lobe-chat + sudo: true + shell: cd /opt/docker/compose/lobe-chat && docker compose up -d lobe-chat + + - name: recreate open-webui + sudo: true + shell: cd /opt/docker/compose/open-webui && docker compose up -d open-webui + + - name: recreate mosquitto + sudo: true + shell: cd /opt/docker/compose/mosquitto && docker compose up -d mosquitto + + - name: recreate dockge + sudo: true + shell: cd /opt/docker/compose/dockge && docker compose up -d dockge + + - name: recreate adguardhome + sudo: true + shell: cd /opt/docker/compose/adguard && docker compose up -d adguardhome + + - name: recreate traefik + sudo: true + shell: cd /opt/docker/compose/traefik && docker compose up -d traefik + +verify: + - name: every relabelled container now carries its new group + sudo: true + changed_when: "false" + shell: >- + docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}' + lobe-chat open-webui mosquitto adguardhome traefik dockge + + - name: no container is left in the retired Service Networking group + sudo: true + changed_when: "false" + shell: >- + test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')" + + - name: adguard is back + changed_when: "false" + shell: >- + for i in 1 2 3 4 5 6 7 8 9 10; do + curl -sfL -o /dev/null -m 5 http://127.0.0.1:8080/ && exit 0; + sleep 3; done; exit 1 + + - name: traefik still routes + changed_when: "false" + shell: >- + for i in 1 2 3 4 5 6 7 8 9 10; do + curl -sfL -o /dev/null -m 5 http://127.0.0.1:8380/dashboard/ && exit 0; + sleep 3; done; exit 1 + + # ⚠ Must be 10.0.50.45, NOT 127.0.0.1. `HOMEPAGE_ALLOWED_HOSTS` matches + # host AND port, and `127.0.0.1:5100` is not in the list — it answers 400 + # while the dashboard is perfectly healthy. The first run of this playbook + # failed here on rc=22 for exactly that reason. + - name: the dashboard itself is still served + changed_when: "false" + shell: curl -sf -o /dev/null -m 10 http://10.0.50.45:5100/api/services + + - name: everything is running + sudo: true + changed_when: "false" + shell: >- + test "$(docker inspect -f '{{.State.Running}}' lobe-chat open-webui + mosquitto adguardhome traefik dockge | sort -u)" = "true" diff --git a/playbooks/homepage-regroup-irv-ml1.yaml b/playbooks/homepage-regroup-irv-ml1.yaml new file mode 100644 index 0000000..9db1e4b --- /dev/null +++ b/playbooks/homepage-regroup-irv-ml1.yaml @@ -0,0 +1,115 @@ +# Homepage recategorisation — irv-ml1 (10.100.79.3), 5 containers. +# Sibling of playbooks/homepage-regroup-ana-docker.yaml; rationale lives there. +# +# arbo, comfyui, waterland-studio -> AI - Studios (was AI - Image & Media) +# yt-voice-clipper -> AI - Studios (was AI - Audio Tools) +# dockge -> Compose Consoles +# +# `AI - Studios` is the "you open this and do work in it" group; the ASR and +# TTS API seats stay where they are and get collapsed by settings.yaml instead, +# which is what keeps the GPU seats out of this playbook entirely. +# +# ⚠ yt-voice-clipper carries its homepage labels in `docker-compose.override.yml`, +# not in `docker-compose.yml`, and its compose dir is a git checkout of the +# project — the override is the deploy-local layer, which is the right place +# for it. +# +# ⚠ This host is reached over the WireGuard tunnel. If the run cannot connect, +# check the tunnel before assuming the host is down. +# +# Run: scripts/elway infra-ops@10.100.79.3 --playbook playbooks/homepage-regroup-irv-ml1.yaml + +steps: + - name: arbo -> AI - Studios + sudo: true + shell: >- + sed -i 's|homepage.group=AI - Image . Media$|homepage.group=AI - Studios|' + /opt/docker/compose/arbo/compose.yaml + when: grep -q 'homepage.group=AI - Image . Media$' /opt/docker/compose/arbo/compose.yaml + + - name: comfyui -> AI - Studios + sudo: true + shell: >- + sed -i 's|homepage.group=AI - Image . Media$|homepage.group=AI - Studios|' + /opt/docker/compose/comfyui/compose.yaml + when: grep -q 'homepage.group=AI - Image . Media$' /opt/docker/compose/comfyui/compose.yaml + + - name: waterland-studio -> AI - Studios + sudo: true + shell: >- + sed -i 's|homepage.group=AI - Image . Media$|homepage.group=AI - Studios|' + /opt/docker/compose/waterland-studio/compose.yaml + when: grep -q 'homepage.group=AI - Image . Media$' /opt/docker/compose/waterland-studio/compose.yaml + + - name: yt-voice-clipper -> AI - Studios + sudo: true + shell: >- + sed -i 's|homepage.group=AI - Audio Tools$|homepage.group=AI - Studios|' + /opt/docker/compose/yt-voice-clipper/docker-compose.override.yml + when: grep -q 'homepage.group=AI - Audio Tools$' /opt/docker/compose/yt-voice-clipper/docker-compose.override.yml + + - name: dockge -> Compose Consoles + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|' + /opt/docker/compose/dockge/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml + + # ---- recreates --------------------------------------------------------- + - name: recreate arbo + sudo: true + shell: cd /opt/docker/compose/arbo && docker compose up -d engine + + - name: recreate comfyui + sudo: true + shell: cd /opt/docker/compose/comfyui && docker compose up -d comfyui + + - name: recreate waterland-studio + sudo: true + shell: cd /opt/docker/compose/waterland-studio && docker compose up -d waterland-studio + + - name: recreate yt-voice-clipper + sudo: true + shell: cd /opt/docker/compose/yt-voice-clipper && docker compose up -d api + + - name: recreate dockge + sudo: true + shell: cd /opt/docker/compose/dockge && docker compose up -d dockge + +verify: + - name: every relabelled container now carries its new group + sudo: true + changed_when: "false" + shell: >- + docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}' + arbo comfyui waterland-studio yt-voice-clipper-api-1 dockge + + - name: no container is left in the retired Service Networking group + sudo: true + changed_when: "false" + shell: >- + test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')" + + # Assert NOT-RECREATED, not RUNNING. The first version of this checked that + # all five seats were up and failed — because chatterbox-fast has been down + # since 2026-08-10 and speaches since earlier the same morning, both long + # before this playbook existed. "Is it running" is the wrong question: a seat + # can be legitimately stopped. The question this step is actually asking is + # "did I bounce a model seat to relabel a dashboard", and container age + # answers it directly. + - name: the TTS and ASR seats were NOT recreated by this run + sudo: true + changed_when: "false" + shell: >- + for c in kokoro dots-tts chatterbox-fast tts-gateway parakeet speaches; do + created=$(docker inspect -f '{{.Created}}' "$c" 2>/dev/null) || continue; + age=$(( $(date +%s) - $(date -d "$created" +%s) )); + if [ "$age" -lt 600 ]; then echo "$c was recreated ${age}s ago"; exit 1; fi; + done + + - name: everything relabelled is running + sudo: true + changed_when: "false" + shell: >- + test "$(docker inspect -f '{{.State.Running}}' arbo comfyui + waterland-studio yt-voice-clipper-api-1 dockge | sort -u)" = "true" diff --git a/playbooks/homepage-regroup-nh3-docker.yaml b/playbooks/homepage-regroup-nh3-docker.yaml new file mode 100644 index 0000000..1795f9e --- /dev/null +++ b/playbooks/homepage-regroup-nh3-docker.yaml @@ -0,0 +1,58 @@ +# Homepage recategorisation — nh3-docker (10.100.50.40), 2 containers. +# Sibling of playbooks/homepage-regroup-ana-docker.yaml; rationale lives there. +# +# adguardhome -> DNS & Filtering (⚠ uses docker-compose.yml) +# dockge -> Compose Consoles +# +# Run: scripts/elway infra-ops@10.100.50.40 --playbook playbooks/homepage-regroup-nh3-docker.yaml + +steps: + - name: adguardhome -> DNS & Filtering + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=DNS \& Filtering|' + /opt/docker/compose/adguard/docker-compose.yml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/adguard/docker-compose.yml + + - name: dockge -> Compose Consoles + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|' + /opt/docker/compose/dockge/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml + + - name: recreate dockge + sudo: true + shell: cd /opt/docker/compose/dockge && docker compose up -d dockge + + - name: recreate adguardhome + sudo: true + shell: cd /opt/docker/compose/adguard && docker compose up -d adguardhome + +verify: + - name: every relabelled container now carries its new group + sudo: true + changed_when: "false" + shell: >- + docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}' + adguardhome dockge + + - name: no container is left in the retired Service Networking group + sudo: true + changed_when: "false" + shell: >- + test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')" + + - name: adguard is back + changed_when: "false" + shell: >- + for i in 1 2 3 4 5 6 7 8 9 10; do + curl -sfL -o /dev/null -m 5 http://127.0.0.1:8080/ && exit 0; + sleep 3; done; exit 1 + + - name: everything is running + sudo: true + changed_when: "false" + shell: >- + test "$(docker inspect -f '{{.State.Running}}' adguardhome dockge + | sort -u)" = "true" diff --git a/stacks/adguard-ana/.env.example b/stacks/adguard-ana/.env.example new file mode 100644 index 0000000..bdd5e16 --- /dev/null +++ b/stacks/adguard-ana/.env.example @@ -0,0 +1,16 @@ +# AdGuard Home — Anaheim. Copy to `.env` on ana-docker at +# /opt/docker/compose/adguard-ana/.env and chmod 600. Never commit the real file. + +# Web UI / API port. compose.yaml defaults to 8053 if unset. +AG_WEB_PORT=8053 + +# Password for the `infra-ops` AdGuard login, used ONLY by the Homepage +# query/blocked/latency widget (homepage.widget.password in compose.yaml). +# One credential authenticates against all three fleet instances (ANA, ESH, +# NH3) — verified 2026-08-24. Fetch it from the vault rather than copying it +# between boxes: +# +# secret get nh3-dev/adguard-infra-ops-password +# +# (`secret` is services/secrets-broker/secret on nh3-dev.) +ADGUARD_WIDGET_PASSWORD= diff --git a/stacks/adguard-ana/compose.yaml b/stacks/adguard-ana/compose.yaml index 536d8be..2808cd7 100644 --- a/stacks/adguard-ana/compose.yaml +++ b/stacks/adguard-ana/compose.yaml @@ -39,12 +39,26 @@ services: networks: - tnet labels: - - homepage.group=Service Networking + - homepage.group=DNS & Filtering - homepage.name=AdGuard (ANA) - - homepage.icon=mdi-dns + # si-adguard, matching the ESH and NH3 instances. This carried mdi-dns + # and was the only one of the three wearing a different mark. + - homepage.icon=si-adguard - homepage.description=DNS resolver + .internal zone (colo) - homepage.href=http://10.250.50.70:${AG_WEB_PORT:-8053} - homepage.siteMonitor=http://10.250.50.70:${AG_WEB_PORT:-8053} + # Query/blocked/latency strip, same as ESH and NH3. Added 2026-08-24: + # without it this was a short card sitting beside two tall ones, which + # is the whole reason the DNS group exists as its own band — a group's + # members should all carry a widget or none of them should. + # Credential is the fleet AdGuard `infra-ops` login, which authenticates + # against all three instances; it lives in this stack's .env on the host + # (never in git) and is vaulted at + # `secret get nh3-dev/adguard-infra-ops-password`. + - homepage.widget.type=adguard + - homepage.widget.url=http://10.250.50.70:${AG_WEB_PORT:-8053} + - homepage.widget.username=infra-ops + - homepage.widget.password=${ADGUARD_WIDGET_PASSWORD} volumes: adguard_work: {} diff --git a/stacks/arbo/compose.yaml b/stacks/arbo/compose.yaml index 7fc6527..96924f0 100644 --- a/stacks/arbo/compose.yaml +++ b/stacks/arbo/compose.yaml @@ -85,7 +85,7 @@ services: networks: - tnet labels: - - homepage.group=AI - Image & Media + - homepage.group=AI - Studios - homepage.name=Arbo - homepage.icon=mdi-image-multiple-outline - homepage.description=Catalog-driven ComfyUI engine (irv-ml1) diff --git a/stacks/comfyui/compose.yaml b/stacks/comfyui/compose.yaml index 99cbb3b..0013695 100644 --- a/stacks/comfyui/compose.yaml +++ b/stacks/comfyui/compose.yaml @@ -90,7 +90,7 @@ services: networks: - tnet labels: - - homepage.group=AI - Image & Media + - homepage.group=AI - Studios - homepage.name=ComfyUI - homepage.icon=mdi-image-auto-adjust - homepage.description=Node-based SD/Flux inference (irv-ml1) diff --git a/stacks/dockge/compose.yaml b/stacks/dockge/compose.yaml index 6a56c38..6e19707 100644 --- a/stacks/dockge/compose.yaml +++ b/stacks/dockge/compose.yaml @@ -27,7 +27,7 @@ services: networks: - tnet labels: - - homepage.group=Service Networking + - homepage.group=Compose Consoles - homepage.name=Dockge (${DOCKGE_HOST_LABEL}) - homepage.icon=sh-dockge.png - homepage.description=Compose UI on ${DOCKGE_HOST_LABEL} diff --git a/stacks/gitea-runner/compose.yaml b/stacks/gitea-runner/compose.yaml index 965985e..167a6a1 100644 --- a/stacks/gitea-runner/compose.yaml +++ b/stacks/gitea-runner/compose.yaml @@ -27,7 +27,7 @@ services: networks: - tnet labels: - - homepage.group=Toolchain + - homepage.group=Agents (no UI) - homepage.name=gitea-runner - homepage.icon=mdi-cog-play - homepage.description=Gitea Actions self-hosted runner (${GITEA_RUNNER_NAME}) diff --git a/stacks/homepage/README.md b/stacks/homepage/README.md index 2555a2d..f8c56f8 100644 --- a/stacks/homepage/README.md +++ b/stacks/homepage/README.md @@ -33,11 +33,44 @@ recreated. ## Layout -`conf/settings.yaml` owns tabs, group order, and column counts — `services.yaml` -owns *what exists*, `settings.yaml` owns *where it sits*. Four tabs: Main, AI, -Infrastructure, Toolchain. A group listed in `layout:` with no members simply -renders empty, so a group can look "dead" when its provider host is unreachable -rather than when the group is wrong. +`conf/settings.yaml` owns tabs, group order, column counts and collapse state — +`services.yaml` owns *what exists*, `settings.yaml` owns *where it sits*. Four +tabs: Main, AI, Infrastructure, Toolchain. A group listed in `layout:` with no +members simply renders empty, so a group can look "dead" when its provider host +is unreachable rather than when the group is wrong. + +**The organising question is "do I open this?", not "what is it?"** (operator, +2026-08-24). Every group is either **tools** — expanded, top of its tab — or +**endpoints** (an API, a broker, a background agent, an href that is a `/docs` +page or a `/ping` or nothing) — `initiallyCollapsed: true`, bottom of its tab. +A collapsed group still renders its eyebrow and rule, so the tab tells you the +thing exists without spending a row on it. `AI - Inference` holds the seats the +whole fleet runs on and it is collapsed, because you reach them through the +gateway, not by clicking them. + +A second, quieter rule shapes the same block: **a group's members should all +have widgets or all not have them.** A stat strip makes a card ~50px taller, so +one widget card in a row of plain ones opens a void under the plain ones. That +is why AdGuard (3 widget cards) and Traefik (2 widget cards) are separate groups +from Dockge (5 plain cards) rather than one `Service Networking` band. + +Group *membership* is not in this file — it is the `homepage.group=` label on +each container, and labels are read at container **creation**. The 2026-08-24 +pass moved 28 services and the playbooks that did it are rerunnable: + +```bash +scripts/elway infra-ops@10.250.50.70 --playbook playbooks/homepage-regroup-ana-docker.yaml +scripts/elway infra-ops@10.0.50.45 --playbook playbooks/homepage-regroup-esh-docker-vm.yaml +scripts/elway infra-ops@10.100.50.40 --playbook playbooks/homepage-regroup-nh3-docker.yaml +scripts/elway infra-ops@10.100.79.3 --playbook playbooks/homepage-regroup-irv-ml1.yaml +scripts/elway infra-ops@10.250.50.54 --playbook playbooks/homepage-regroup-ana-ml2.yaml +``` + +⚠ **The GPU-backed model seats are deliberately still named `AI - Inference`, +`AI - Eval & Retrieval`, `AI - Speech (TTS)` and `AI - Audio Tools`.** Clearer +names would cost a recreate on sixteen seats — multi-minute model reloads on +endpoints peers reach through the gateway. Order and `initiallyCollapsed` buy +the same separation for free. Don't spend that recreate on a label. ## Foot-guns found in the 2026-08-17 audit @@ -124,18 +157,86 @@ is in the "iteration loop" note below. ## The tab bar goes missing after a recreate, then comes back on its own -**Status: self-healing, cause not pinned, and INTERMITTENT. Do not chase it.** +**Status: intermittent, self-healing, and now HALF-DIAGNOSED. Still don't chase +it — but you can now tell in one command whether you are in it.** + After a recreate the client render can come up with no tab bar, no wallpaper and no i18n (the search box shows the raw key `search.search`), with groups falling back to side-by-side columns. It restores itself with no intervention. -⚠ **It does not happen every time.** The 2026-08-24 recreate (`compose up -d ---force-recreate homepage`, to land a `settings.yaml` change) came up correct -within 10 seconds: the fresh `columns: 4` payload was in the served HTML on the -first poll, and a Playwright pass clicked through all four tabs a minute later -with the bar present. So do not pre-emptively schedule an hour of ugliness -around a config change — recreate, check, and only then walk away if it is in -the broken state. +### The one-command test + +```bash +curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}' +``` + +`initialSettings":{"…` — healthy. `initialSettings":{}` — you are in it. + +### What `initialSettings":{}` actually means (found 2026-08-24) + +It is **not** a warm-up, a cache, or a partial render. It is the **catch branch** +of the page's data loader. Decompiled from `/app/.next/server/pages/index.js`: + +```js +async function K() { + let a; + try { + a = logger("index"); // <-- assigned INSIDE the try + let { providers, ...d } = getSettings(); + ... await servicesResponse(), bookmarksResponse(), widgetsResponse() + return { props: { initialSettings: d, ... } }; + } catch (b) { + return a && b && a.error(b), // <-- guarded on `a` + { props: { initialSettings: {}, ... } }; + } +} +``` + +Two consequences worth knowing: + +1. **Something in that try block is throwing.** The page is not "still warming + up"; it has already failed and returned a degraded shape. +2. **The error can be swallowed completely.** `a` is assigned inside the same + try, and the catch only logs `if (a)`. If the *logger itself* is what threw, + `a` is `undefined` and nothing is written anywhere — which is exactly what + was observed on 2026-08-24: catch branch demonstrably taken, and not one + `index`-tagged line in `docker logs` or `conf/homepage/logs/homepage.log`. + +### What it is NOT — ruled out by measurement, don't re-run these + +- **Not the config.** `/api/services`, `/api/bookmarks`, `/api/widgets` and + `/api/hash` all return **200 with fully correct content** while the page + serves `initialSettings":{}` — including the brand-new group structure, in the + right order. Every input the loader awaits works when called directly. +- **Not the 2026-08-24 layout rewrite.** Restoring the *previous, known-good* + `settings.yaml` and recreating reproduced the empty payload identically. (This + matches the 2026-08-19 finding that the pre-adoption backup config reproduces + it too.) +- **Not `/api/validate`,** which returns `[]` throughout. +- **Not disk, not permissions.** 206 GB free; the container runs as root and a + write test into `/app/config/logs` succeeds. + +### Timing, measured rather than assumed + +Wildly variable, which is the whole trap. On 2026-08-24 one recreate came up +correct **within 10 seconds**, and three consecutive recreates ~40 minutes later +were still empty after 60-120s each. The 2026-08-19 session measured a fresh +container still tab-less at 4m30s twice, and healthy again after roughly an hour. +`docker ps` reporting `healthy` says nothing about it — the container is serving, +the page is just wrong. + +**Practical rule: recreate, run the one-command test, and if it is empty, go do +something else and re-check.** Do not start editing config — that is how an hour +got spent in 2026-08-19 ruling out four causes that were never the cause (the +config, the v2.0.0 release, `PUID`/`PGID` and Docker discovery, and the server +side). Every one of those remains ruled out. + +**Next lead, for whoever picks this up:** move the `logger("index")` hypothesis +forward. The winston file logger writes to `conf/homepage/logs/homepage.log`, +which had grown to 8.6 MB and stopped being appended to at the same time the +render started failing. Rolling it aside is a one-liner and is the cheapest +thing left to try: +`sudo mv /opt/docker/conf/homepage/logs/homepage.log{,.rolled}` then recreate. **Timing, measured rather than assumed:** five minutes is NOT enough — a fresh container was still tab-less at 4m30s, twice. It was observed healthy again diff --git a/stacks/homepage/conf/services.yaml b/stacks/homepage/conf/services.yaml index 1bda6a4..9af2f36 100644 --- a/stacks/homepage/conf/services.yaml +++ b/stacks/homepage/conf/services.yaml @@ -27,24 +27,37 @@ icon: mdi-filmstrip siteMonitor: http://10.100.10.50:8090/healthz description: Media drop + upload-for-pickup + the standing agent link board — nh3-dev, 24h TTL except kept boards - - Voice Design Studio: - href: http://10.100.79.3:8216/ - icon: mdi-microphone - siteMonitor: http://10.100.79.3:8216/health - description: Mint, audition and keeper-mark synthetic fleet voices — irv-ml1, CPU-only - The Henge: href: http://park.phasefinal.com:8420/ icon: mdi-clipboard-check siteMonitor: http://park.phasefinal.com:8420/healthz description: Durable needs-attention / idea parking (stonehenge-park) — ana-docker + # Was its own one-card `Games` group, which burned a full 4-wide row on a + # single panel. It is an app you open; this is where apps you open live. + - Pterodactyl: + href: http://10.250.50.55/ + icon: mdi-gamepad-square + siteMonitor: http://10.250.50.55 + description: Game server panel -# The AI tab is fully Docker-auto-discovered. Each inference service carries -# a homepage.group=AI - label on its compose file (AI - Inference, -# AI - Eval & Retrieval, AI - Gateways & Chat, AI - Speech (TTS), -# AI - Audio Tools, AI - Image & Media). Tab assignment, group order, and -# column counts live in settings.yaml. Do not add entries here or they'll -# double up. To move a service between AI groups, change the label on its -# compose file and recreate the container (labels only apply on recreate). +# The AI tab is otherwise fully Docker-auto-discovered. Each service carries a +# homepage.group=AI - label on its compose file (AI - Gateways & Chat, +# AI - Studios, AI - Inference, AI - Eval & Retrieval, AI - Speech (TTS), +# AI - Audio Tools, AI - Dormant). Tab assignment, group order, columns and +# collapse state live in settings.yaml. Do not add a labelled container here as +# well or it renders twice. To move a service between AI groups, change the +# label on its compose file and recreate the container — labels are read at +# creation, so `restart` will not do it. +- AI - Studios: + # Manual entry — Voice Design Studio is a user-level systemd service on + # irv-ml1, not a Docker-labeled stack, so it cannot auto-discover. It sits + # with the other studios rather than in Apps: it is a workspace you open and + # produce something in, which is exactly what that group is for. + - Voice Design Studio: + href: http://10.100.79.3:8216/ + icon: mdi-microphone + siteMonitor: http://10.100.79.3:8216/health + description: Mint, audition and keeper-mark synthetic fleet voices — irv-ml1, CPU-only - Media: - Plex: @@ -67,13 +80,6 @@ key: '{{HOMEPAGE_VAR_JELLYFIN_KEY}}' enableBlocks: true -- Games: - - Pterodactyl: - href: http://10.250.50.55/ - icon: mdi-gamepad-square - siteMonitor: http://10.250.50.55 - description: Game server panel - - Infra - ANA: - ANA-Firewall: href: https://10.250.250.1 diff --git a/stacks/homepage/conf/settings.yaml b/stacks/homepage/conf/settings.yaml index 312588c..d5b299b 100644 --- a/stacks/homepage/conf/settings.yaml +++ b/stacks/homepage/conf/settings.yaml @@ -46,34 +46,55 @@ statusStyle: "" # reads as a rendering fault. useEqualHeights: false -# Function-first layout, four-tab split: -# Main - daily-use apps, media, bookmarks, monitoring -# AI - the inference fleet, grouped by role (see below) -# Infrastructure - hardware, hypervisors, BMCs (per site) -# Toolchain - backend services running but rarely clicked +# =========================================================================== +# THE ORGANISING QUESTION IS "DO I OPEN THIS?" — NOT "WHAT IS IT?" +# (operator, 2026-08-24: "most of the issues are that tools I use and have a +# UI are interspersed with API endpoints which are largely informational +# only. They might even go in their own cards or start collapsed.") # -# The AI tab splits the fleet by function so a 20+ service list reads as -# sorted groups instead of one endless column. Group membership is set by -# the homepage.group=AI - label on each service's compose file: -# AI - Inference LLM seats you call (gen, char-rp, char-rp-reasoning, summarizer) -# AI - Eval & Retrieval judges, reward, rerank, embed, image-quality -# AI - Gateways & Chat routing gateway, control plane, chat frontends -# AI - Speech (TTS) text-to-speech engines -# AI - Audio Tools speech-to-text + audio dataset tooling -# AI - Image & Media image/video generation + pipelines -# AI - Dormant stopped stacks (rollback seats, retired auditions) +# Every group on this board is one of two kinds, and they never mix: # -# AI TAB ORDER IS BY CLICKABILITY, NOT BY IMPORTANCE (operator, 2026-08-18). -# Groups render in the order they appear in this block, so the top of the tab -# is prime real estate and it should hold the things you actually open in a -# browser — chat frontends, ComfyUI, the control plane. Most of the model -# seats below them are vLLM API endpoints whose href is a `/docs` page: they -# are worth SEEING (status at a glance) but not worth reaching for, so they -# sink. Order is therefore: -# interactive UIs -> mixed -> API-only seats -> dormant -# If you add an AI group, place it by asking "would I click this?", not by -# how central the service is to the fleet. +# TOOLS — you click the card and do something in the thing it opens. +# Expanded, and placed at the TOP of its tab. +# ENDPOINTS — an API, a broker, a background agent. Its href is a `/docs` +# page, a `/ping`, or nothing at all. The only thing you want from +# the card is "is it alive". `initiallyCollapsed: true`, and placed +# at the BOTTOM of its tab. # +# A collapsed group is not hidden — the eyebrow and its rule still render, so +# the tab still tells you the thing exists, and one click expands it. That is +# the whole point: presence without cost. +# +# When you add a service, ask "would I open this in a browser to get work +# done?" If no, it belongs in a collapsed endpoint group, no matter how +# central it is to the fleet. `AI - Inference` holds the seats the entire +# fleet runs on and it is collapsed, because you consume them through the +# gateway rather than by clicking them. +# +# Four tabs: +# Main - what you actually open day to day +# AI - AI tools up top, model/API seats collapsed below +# Infrastructure - hardware, hypervisors, BMCs (per site) — all consoles +# Toolchain - the plumbing, split by kind of plumbing +# +# --------------------------------------------------------------------------- +# GROUP MEMBERSHIP LIVES ON THE CONTAINER, NOT HERE. +# +# This block controls tab, order, columns and collapse. WHICH services are in +# a group is set by `homepage.group=` on each container's compose file, and +# labels only apply at container CREATION — moving a service between groups +# means editing the label and running `docker compose up -d `, not +# `restart`. The 2026-08-24 pass did 28 of those; the playbooks that did it +# are `playbooks/homepage-regroup-.yaml` and they are rerunnable. +# +# ⚠ THE MODEL SEATS ARE DELIBERATELY STILL NAMED `AI - Inference`, +# `AI - Eval & Retrieval`, `AI - Speech (TTS)` AND `AI - Audio Tools`. +# Renaming them to something like "AI API - …" would be clearer, and it would +# cost a recreate on sixteen GPU-backed seats — multi-minute model reloads on +# endpoints peers reach through the gateway. Order and `initiallyCollapsed` +# buy the same separation for free. Do not spend that recreate on a label. +# +# --------------------------------------------------------------------------- # COLUMNS ARE 4 EVERYWHERE. DO NOT TUNE THEM PER GROUP. # # `columns: N` is not a density dial — it sets `lg:grid-cols-N` on that one @@ -97,24 +118,36 @@ useEqualHeights: false # still collapses this to 2-up and 1-up on narrow viewports, so 4 is a desktop # maximum, not a hard floor. # +# --------------------------------------------------------------------------- +# GROUPS ARE ALSO KEPT UNIFORM IN CARD HEIGHT, WHICH IS WHY ADGUARD AND +# TRAEFIK GOT THEIR OWN GROUPS. +# +# A service with a widget (AdGuard's query counts, Traefik's router counts, +# Uptime Kuma's uptime) renders a stat strip that makes its card ~50px taller +# than a plain link card. Put one of those in a row of three plain cards and +# you get a void under the plain ones — which is what made the old 13-member +# `Service Networking` group look broken. Split so that a group's members all +# have widgets or all do not, and every row comes out flush. That is the real +# reason `DNS & Filtering` (3 widget cards) and `Reverse Proxies` (2 widget +# cards) are separate from `Compose Consoles` (5 plain cards). +# +# --------------------------------------------------------------------------- # EVERY GROUP NEEDS A `tab:` — including bookmark groups, and including groups # that arrive from a `homepage.group=` container label rather than from this # file. A group with no tab assignment renders on ALL FOUR TABS. That is how # UltraSeedbox ended up repeated at the bottom of every tab (fixed 2026-08-18) # and how Scriberr's `AI Systems` label did the same from 2026-08-23 (fixed -# 2026-08-24 by relabelling it into `AI - Audio Tools`). It is Homepage -# behaviour, not a bug, and it will happen again to the next container labelled -# with a group name that does not appear below. +# 2026-08-24). It is Homepage behaviour, not a bug, and it will happen again to +# the next container labelled with a group name that does not appear below. # `GET /api/services` prints the live group list — anything in it that is not a # key here is currently leaking onto all four tabs. +# =========================================================================== layout: - Notes: - icon: mdi-note-text-outline - tab: Main - style: row - columns: 4 - News: - icon: mdi-rss + # ---- Main: what you actually open ------------------------------------- + # Replaces the old Notes (1 member) and News (2) bands, which each burned a + # full 4-wide row on a single card. + Daily: + icon: mdi-coffee-outline tab: Main style: row columns: 4 @@ -123,6 +156,9 @@ layout: tab: Main style: row columns: 4 + # Absorbed the old one-card Games band (Pterodactyl). Lost SearXNG to Daily, + # the two chat frontends to the AI tab, and Mosquitto + the RustDesk relay to + # Agents (no UI) — neither of those has a page to open. Apps: icon: mdi-apps tab: Main @@ -133,11 +169,6 @@ layout: tab: Main style: row columns: 4 - Games: - icon: mdi-gamepad-square - tab: Main - style: row - columns: 4 # Bookmarks. Listed here for the tab pin above all else — without it this # group appears on every tab. `style: row` also turns the eight entries # from full-width stacked bars into a compact grid. @@ -146,44 +177,53 @@ layout: tab: Main style: row columns: 4 - # --- AI tab: ordered interactive -> API-only -> dormant (see note above) --- - # Things you open: chat frontends, the control plane, the LiteLLM UI. + + # ---- AI: tools, then collapsed endpoints ------------------------------ + # Chat frontends and the control plane — Lobe Chat and the ESH Open WebUI + # joined from Main on 2026-08-24; they are chat frontends and belong with the + # other chat frontends. AI - Gateways & Chat: icon: mdi-router-network tab: AI style: row columns: 4 - # ComfyUI is a full node editor and Arbo has a real UI — both get clicked. - AI - Image & Media: - icon: mdi-image-multiple + # Replaces `AI - Image & Media`. Everything here is a workspace you open and + # produce something in: ComfyUI's node editor, Arbo, Waterland, the YT + # Voice Clipper audition console, Scriberr's transcription UI. + AI - Studios: + icon: mdi-palette-outline tab: AI style: row columns: 4 - # Mixed: YT Voice Clipper has an audition console, Parakeet is an API. - # Scriberr joined here 2026-08-24 (it was the stray `AI Systems` group). - AI - Audio Tools: - icon: mdi-waveform - tab: AI - style: row - columns: 4 - # Below here: model seats whose href is a vLLM `/docs` page. Status at a - # glance is the whole value; you consume these through the gateway, not by - # clicking them. + # ---- collapsed from here down: seats you call, not pages you open ------ + # Named `AI - Inference` rather than something clearer on purpose — see the + # warning above about what renaming these costs. AI - Inference: icon: mdi-brain tab: AI style: row columns: 4 + initiallyCollapsed: true AI - Eval & Retrieval: icon: mdi-scale-balance tab: AI style: row columns: 4 + initiallyCollapsed: true AI - Speech (TTS): icon: mdi-account-voice tab: AI style: row columns: 4 + initiallyCollapsed: true + # What is left of the old Audio Tools group after Scriberr and the YT Voice + # Clipper moved to Studios: the two ASR API seats. + AI - Audio Tools: + icon: mdi-waveform + tab: AI + style: row + columns: 4 + initiallyCollapsed: true # Stopped stacks kept for rollback / superseded seats / retired auditions. # They stay 'created' (not running) via `docker compose up --no-start`, so # they show here as offline cards and revive with `docker compose start`. @@ -192,6 +232,9 @@ layout: tab: AI style: row columns: 4 + initiallyCollapsed: true + + # ---- Infrastructure: every card is a console -------------------------- Infra - ANA: icon: si-proxmox tab: Infrastructure @@ -212,13 +255,38 @@ layout: tab: Infrastructure style: row columns: 4 - Service Networking: + + # ---- Toolchain: the plumbing, split by kind --------------------------- + # The old `Service Networking` group was thirteen members mixing three + # AdGuards, five Dockges, two Traefiks and four headless agents — widget + # cards next to plain ones next to things with no href at all. Split four + # ways on 2026-08-24. + DNS & Filtering: + icon: mdi-dns + tab: Toolchain + style: row + columns: 4 + Reverse Proxies: icon: mdi-transit-connection-variant tab: Toolchain style: row columns: 4 + Compose Consoles: + icon: mdi-docker + tab: Toolchain + style: row + columns: 4 Toolchain: icon: mdi-toolbox tab: Toolchain style: row columns: 4 + # No href, or an href that is an API. CrowdSec, Mailrise, the restic + # rest-server, the Gitea Actions runner, the MQTT broker, the RustDesk relay. + # You never open these; you only ever want to know they are up. + Agents (no UI): + icon: mdi-cog-transfer-outline + tab: Toolchain + style: row + columns: 4 + initiallyCollapsed: true diff --git a/stacks/lobe-chat/compose.yaml b/stacks/lobe-chat/compose.yaml index df62b0e..106cd90 100644 --- a/stacks/lobe-chat/compose.yaml +++ b/stacks/lobe-chat/compose.yaml @@ -120,7 +120,7 @@ services: networks: - tnet labels: - - homepage.group=Apps + - homepage.group=AI - Gateways & Chat - homepage.name=Lobe Chat - homepage.icon=mdi-chat-processing - homepage.description=Chat frontend over the LiteLLM gateway (eval) diff --git a/stacks/memos/compose.yaml b/stacks/memos/compose.yaml index f8dc4f2..2845a93 100644 --- a/stacks/memos/compose.yaml +++ b/stacks/memos/compose.yaml @@ -33,7 +33,7 @@ services: networks: - tnet labels: - - homepage.group=Notes + - homepage.group=Daily - homepage.name=Memos - homepage.icon=mdi-note-text-outline - homepage.description=Self-hosted note + memo server (ana-docker) diff --git a/stacks/miniflux/compose.yaml b/stacks/miniflux/compose.yaml index 114d28e..47e5f15 100644 --- a/stacks/miniflux/compose.yaml +++ b/stacks/miniflux/compose.yaml @@ -44,7 +44,7 @@ services: networks: - tnet labels: - - homepage.group=News + - homepage.group=Daily - homepage.name=Miniflux - homepage.icon=mdi-rss - homepage.description=RSS reader (subreddits, blogs, HN) diff --git a/stacks/open-webui/compose.yaml b/stacks/open-webui/compose.yaml index 9447316..1af61ed 100644 --- a/stacks/open-webui/compose.yaml +++ b/stacks/open-webui/compose.yaml @@ -106,8 +106,8 @@ services: networks: - tnet labels: - - homepage.group=Apps - - homepage.name=Open WebUI + - homepage.group=AI - Gateways & Chat + - homepage.name=Open WebUI (esh) - homepage.icon=mdi-chat-question - homepage.description=Env-declarative chat frontend over the LiteLLM gateway (eval vs Lobe) - homepage.href=http://10.0.50.45:${OPENWEBUI_PORT:-3211} diff --git a/stacks/rest-server-ana/compose.yaml b/stacks/rest-server-ana/compose.yaml index 1d14c30..a2e550a 100644 --- a/stacks/rest-server-ana/compose.yaml +++ b/stacks/rest-server-ana/compose.yaml @@ -52,7 +52,7 @@ services: networks: - tnet labels: - - homepage.group=Service Networking + - homepage.group=Agents (no UI) - homepage.name=Restic (rest-server) - homepage.icon=mdi-cloud-upload - homepage.description=Anaheim restic endpoint (data on NFS) diff --git a/stacks/scriberr/compose.yaml b/stacks/scriberr/compose.yaml index d4058b1..ac302da 100644 --- a/stacks/scriberr/compose.yaml +++ b/stacks/scriberr/compose.yaml @@ -90,13 +90,16 @@ services: networks: - tnet labels: - # `AI - Audio Tools`, not `AI Systems`. The group name has to match a key - # in the dashboard's settings.yaml `layout:` block — a group that appears - # nowhere in that block gets no `tab:` assignment, and Homepage renders an - # untabbed group on EVERY tab. `AI Systems` did exactly that from - # 2026-08-23 until it was caught; transcription belongs with the other ASR - # seats anyway (Parakeet, Speaches). - - homepage.group=AI - Audio Tools + # ⚠ The group name MUST match a key in the dashboard's settings.yaml + # `layout:` block. A group that appears nowhere in that block gets no + # `tab:` assignment, and Homepage renders an untabbed group on EVERY tab. + # This label read `AI Systems` — a group that existed nowhere — from + # 2026-08-23 until it was caught on 2026-08-24. + # `AI - Studios` and not one of the ASR groups because Scriberr is a + # transcription UI you open and work in, which is what Studios collects; + # the bare ASR endpoints (Parakeet, Speaches) live in the collapsed + # `AI - Audio Tools` group instead. + - homepage.group=AI - Studios - homepage.name=Scriberr - homepage.icon=mdi-microphone-message - homepage.description=Audio/video transcription + diarization (ana-ml2, GPU1) diff --git a/stacks/searxng/compose.yaml b/stacks/searxng/compose.yaml index 97cb302..16ad556 100644 --- a/stacks/searxng/compose.yaml +++ b/stacks/searxng/compose.yaml @@ -87,7 +87,7 @@ services: - traefik.http.routers.searxng.tls=true - traefik.http.routers.searxng.service=searxng - traefik.http.services.searxng.loadbalancer.server.port=8080 - - homepage.group=Apps + - homepage.group=Daily - homepage.name=SearXNG - homepage.icon=si-searxng - homepage.description=Privacy-respecting meta-search diff --git a/stacks/waterland-studio/compose.yaml b/stacks/waterland-studio/compose.yaml index 11f405e..463fa3e 100644 --- a/stacks/waterland-studio/compose.yaml +++ b/stacks/waterland-studio/compose.yaml @@ -82,7 +82,7 @@ services: networks: - tnet labels: - - homepage.group=AI - Image & Media + - homepage.group=AI - Studios - homepage.name=Waterland Studio - homepage.icon=mdi-watercolor - homepage.description=Watercolour plate + reveal animation renderer (irv-ml1, A6000) diff --git a/stacks/zed-fim-proxy/compose.yaml b/stacks/zed-fim-proxy/compose.yaml index 7284e75..3faec0e 100644 --- a/stacks/zed-fim-proxy/compose.yaml +++ b/stacks/zed-fim-proxy/compose.yaml @@ -29,7 +29,7 @@ services: retries: 3 start_period: 10s labels: - - homepage.group=AI - Gateways & Chat + - homepage.group=AI - Inference - homepage.name=Zed FIM Proxy - homepage.icon=mdi-code-braces-box - homepage.description=Keyless /v1/completions for Zed edit-predictions (coder-fast, ana-docker)