diff --git a/persistent-memory.d/2026-08-24-homepage-uniform-grid.md b/persistent-memory.d/2026-08-24-homepage-uniform-grid.md index 5fec4c2..cabdb15 100644 --- a/persistent-memory.d/2026-08-24-homepage-uniform-grid.md +++ b/persistent-memory.d/2026-08-24-homepage-uniform-grid.md @@ -116,3 +116,97 @@ The Main tab still opens on three sparse bands: **Notes** (1 member) and News, or folding Games into Apps, would tighten the top of the page — but that is information architecture, not layout, and the group names are the operator's. Surfaced rather than done. + +→ **Resolved in pass 2 below**, where the operator delegated the naming +("re-categorize however you want"). Notes + News became `Daily`, Games folded +into `Apps`, and the `AI - Audio Tools` placement in this pass was superseded +(Scriberr moved on to `AI - Studios`). + +--- + +# `[2026-08-24, pass 2]` Recategorised on "do I open this?", API groups collapsed + +Operator, after seeing pass 1: *"You can re-categorize however you want. +service networking tab is uneven, you can split out the adguard cards, etc. +most of the issues are that tools I use and have a UI are interspersed with API +endpoints which are largely informational only. They might even go in their own +cards or start collapsed."* + +## The axis + +Every group is now either **tools** (expanded, top of tab) or **endpoints** (an +API, a broker, an agent — `initiallyCollapsed: true`, bottom of tab). A +collapsed group still renders its eyebrow and rule, so presence costs one line +instead of two rows. + +Second, quieter rule that fell out of the same pass: **a group's members should +all carry a widget or none should.** A stat strip adds ~50px, so one widget card +in a row of plain ones opens a void under the plain ones — which is most of what +made the 13-member `Service Networking` band look broken. + +## Shape + +- **Main** — `Daily` (Memos, Miniflux, Nevermore, SearXNG — replaces the + 1-card Notes and 2-card News bands), `Monitoring`, `Apps` (12; absorbed the + 1-card `Games` band), `Media`, `UltraSeedbox`. +- **AI** — `AI - Gateways & Chat` (8) and `AI - Studios` (6) expanded; then + `AI - Inference` (7), `AI - Eval & Retrieval` (4), `AI - Speech (TTS)` (4), + `AI - Audio Tools` (2), `AI - Dormant` (6) all collapsed. +- **Toolchain** — `DNS & Filtering` (3), `Reverse Proxies` (2), + `Compose Consoles` (5), `Toolchain` (3), `Agents (no UI)` (6, collapsed). +- **Infrastructure** — unchanged; every card there is already a console. + +Measured after: every group `card=360`, and `DNS & Filtering` and +`Reverse Proxies` both `h=134..134` — dead flush. + +## ⚠ The move that made it affordable + +**The sixteen GPU-backed model seats were NOT relabelled.** `homepage.group` is +read at container **creation**, so renaming `AI - Inference` to something +clearer would have meant recreating six vLLM seats plus four eval seats plus +four TTS engines — multi-minute model reloads on endpoints peers reach through +the gateway. Order plus `initiallyCollapsed` buys the same separation for free, +so the names stay ugly on purpose. **Do not spend that recreate on a label.** + +28 containers *were* relabelled — all cheap web services — via five rerunnable +elway playbooks, `playbooks/homepage-regroup-.yaml`. The canonical +`stacks/` tree was synced to match afterwards, so intent and reality agree. + +`initiallyCollapsed: true` is a per-group key in `layout:`; confirmed present in +this build (`defaultOpen: !(group?.initiallyCollapsed ?? global)` in +`/app/.next/server/pages/index.js`). + +## AdGuard (ANA) gained its widget, and the credential is fleet-wide + +It was the only AdGuard without a query/blocked/latency strip, so it sat short +beside two tall siblings. **One `infra-ops` AdGuard login authenticates against +all three instances** (ANA `:8053`, NH3 `:8080`, ESH `:8080` — all returned 200 +on `POST /control/login`, verified 2026-08-24). Vaulted at +`secret get nh3-dev/adguard-infra-ops-password`; written to +`/opt/docker/compose/adguard-ana/.env` (0600, root) and never into git. Its icon +was also the odd one out (`mdi-dns` against two `si-adguard`). + +## ⚠⚠ `initialSettings":{}` — the tab-bar mystery is a SWALLOWED EXCEPTION + +The biggest durable finding of the day, and it cost ~25 minutes. Full write-up +in `stacks/homepage/README.md`; the short version: + +`initialSettings":{}` in the served HTML is **the catch branch** of the page's +data loader, not a warm-up and not a cache. And the error can vanish without +trace: the logger is assigned as the first statement *inside* the same `try`, +and the `catch` only logs `if (logger)`. If the logger is what threw, nothing is +written anywhere — which is exactly what was observed. + +Ruled out by measurement, do not re-run: `/api/services`, `/api/bookmarks`, +`/api/widgets` and `/api/hash` all return **200 with correct content** while the +page serves `{}`; restoring the previous known-good `settings.yaml` reproduces +it identically; `/api/validate` returns `[]`; disk and permissions are fine. + +**One-command test:** +`curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}'` + +**What broke the streak:** three consecutive recreates came up empty, then +rolling the 8.6 MB `conf/homepage/logs/homepage.log` aside and recreating healed +it within 15 seconds. That is one observation, not proof — but it is a coherent +mechanism (oversized log → logger init throws → silent catch) and it is the +cheapest thing to try first next time. diff --git a/persistent-memory.md b/persistent-memory.md index 2ebc10f..b40fe67 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -110,7 +110,7 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-08-24 — a very long ops session; everything it opened, it closed. Nothing is mid-flight. The **homepage** dashboard pass is done (below); the only thing left open on it is an information-architecture question for the operator, not work._ -- **🟢 Homepage relaid out on a UNIFORM 4-column grid, and its remote-label discovery re-verified.** `/api/services` proves the board reads container labels off **all five** Docker engines — 74 of 107 cards are label-discovered and only 13 come from its own host. ⚠ **"columns = member count" is RETIRED**; per-group column tuning is what made the card width change five times down the page. Also fixed: long names printing under their own status pill (6 cards, AI tab), the `AI Systems`/Scriberr all-tabs leak, and grey-smudge icons. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` +- **🟢 Homepage rebuilt in two passes: uniform 4-column grid, then recategorised on "do I open this?"** Groups are now either TOOLS (expanded, top of tab) or ENDPOINTS (`initiallyCollapsed: true`, bottom); `Service Networking` split four ways; 28 containers relabelled via rerunnable `playbooks/homepage-regroup-.yaml`. Remote-label discovery re-verified — the board reads labels off **all five** Docker engines (74 of 107 cards, only 13 from its own host). ⚠ **The 16 GPU model seats were deliberately NOT relabelled** — order + collapse buys the same separation without a model reload. ⚠ **"columns = member count" is RETIRED.** ⚠ `initialSettings":{}` in the served HTML is a **swallowed exception**, not a warm-up. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` - **🔒 ana-gw's public admin surface is now ZERO open TCP ports**, and the box is scheduled for replacement by **OPNsense on a Dell R420** (which brings WireGuard onto the edge — so the downstream-WireGuard-VM design is moot, do not scope it). WAN admin was opened as a cutover contingency then closed on instruction; the FortiOS **ACME listener** on :80 was disabled too. **No out-of-band path to ana-gw remains** — if both tunnels drop it is console-only; re-open is two one-liners in auto-memory. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md` @@ -136,6 +136,7 @@ _As of 2026-08-24 — a very long ops session; everything it opened, it closed. ## Recent decisions +- `[2026-08-24]` **Homepage reorganised on the axis "do I open this?" — UI groups expanded on top, API/agent groups collapsed at the bottom** (operator-delegated: "re-categorize however you want"). Load-bearing constraint: `homepage.group` is read at container CREATION, so the 16 GPU-backed model seats keep their unlovely names rather than eat a recreate — `initiallyCollapsed` + order is free. Second rule discovered here: **group members should all have widgets or none should**, because a stat strip adds ~50px and opens a void beside plain cards. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` - `[2026-08-24]` **Homepage columns unified at 4 for every group; the 2026-08-18 "columns = member count" rule is retired.** It was avoiding dead cells in a short last row and bought a worse defect — card width changing at every group boundary. Also carries two CSS traps: `overflow: hidden` clips at the PADDING box (so a `padding-right` gutter is spill room, not a guard), and a `:root` override of a Homepage theme variable is silently outranked by `.theme-slate` on the same `` element. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md` - `[2026-08-24]` **AES-128 adopted on both Anaheim tunnels; the per-flow ceiling root-caused to the UDM's software AES-CBC, exonerating the FortiGate.** Proven by an A/B/A cipher swap at identical CPU — hardware offload is not cipher-cost-sensitive. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md` - `[2026-08-24]` **ana-gw's public admin surface closed to zero open ports, ACME listener included.** Two of my diagnoses were wrong first (an "ISP proxy" that was the FortiGate, and an "all-port VIP" alarm that was a parser gap) — both from reading config instead of measuring from outside. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md` diff --git a/playbooks/homepage-regroup-ana-docker.yaml b/playbooks/homepage-regroup-ana-docker.yaml new file mode 100644 index 0000000..e40dacc --- /dev/null +++ b/playbooks/homepage-regroup-ana-docker.yaml @@ -0,0 +1,226 @@ +# Homepage recategorisation — ana-docker (10.250.50.70), 13 containers. +# +# Splits the dashboard on ONE axis: do you open this thing, or is it an +# endpoint you only want to know is alive? See the layout: block in +# stacks/homepage/conf/settings.yaml for the target shape. +# +# memos, miniflux, nevermore, searxng -> Daily (was Notes / News / Apps) +# zed-fim-proxy -> AI - Inference (no UI; href is /ping) +# adguardhome -> DNS & Filtering +# traefik -> Reverse Proxies +# dockge -> Compose Consoles +# crowdsec, mailrise, rest-server, +# gitea-runner, hbbr (RustDesk relay) -> Agents (no UI) +# +# `homepage.group` is read at container CREATION, so each edit is followed by +# `compose up -d ` — a restart would leave the old label in place. +# Both halves are idempotent: the sed is gated on the old value still being +# present, and `up -d` is a no-op when the container already matches its spec. +# +# Run: scripts/elway infra-ops@10.250.50.70 --playbook playbooks/homepage-regroup-ana-docker.yaml + +steps: + # ---- label edits ------------------------------------------------------- + - name: memos -> Daily + sudo: true + shell: >- + sed -i 's|homepage.group=Notes$|homepage.group=Daily|' + /opt/docker/compose/memos/compose.yaml + when: grep -q 'homepage.group=Notes$' /opt/docker/compose/memos/compose.yaml + + - name: miniflux -> Daily + sudo: true + shell: >- + sed -i 's|homepage.group=News$|homepage.group=Daily|' + /opt/docker/compose/miniflux/compose.yaml + when: grep -q 'homepage.group=News$' /opt/docker/compose/miniflux/compose.yaml + + - name: nevermore -> Daily + sudo: true + shell: >- + sed -i 's|homepage.group=News$|homepage.group=Daily|' + /opt/docker/compose/nevermore/compose.yaml + when: grep -q 'homepage.group=News$' /opt/docker/compose/nevermore/compose.yaml + + - name: searxng -> Daily + sudo: true + shell: >- + sed -i 's|homepage.group=Apps$|homepage.group=Daily|' + /opt/docker/compose/searxng/compose.yaml + when: grep -q 'homepage.group=Apps$' /opt/docker/compose/searxng/compose.yaml + + - name: zed-fim-proxy -> AI - Inference + sudo: true + shell: >- + sed -i 's|homepage.group=AI - Gateways . Chat$|homepage.group=AI - Inference|' + /opt/docker/compose/zed-fim-proxy/compose.yaml + when: grep -q 'homepage.group=AI - Gateways . Chat$' /opt/docker/compose/zed-fim-proxy/compose.yaml + + - name: adguardhome -> DNS & Filtering + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=DNS \& Filtering|' + /opt/docker/compose/adguard-ana/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/adguard-ana/compose.yaml + + - name: traefik -> Reverse Proxies + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Reverse Proxies|' + /opt/docker/compose/traefik/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/traefik/compose.yaml + + - name: dockge -> Compose Consoles + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|' + /opt/docker/compose/dockge/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml + + - name: crowdsec -> Agents (no UI) + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Agents (no UI)|' + /opt/docker/compose/crowdsec/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/crowdsec/compose.yaml + + - name: mailrise -> Agents (no UI) + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Agents (no UI)|' + /opt/docker/compose/mailrise/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/mailrise/compose.yaml + + - name: rest-server -> Agents (no UI) + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Agents (no UI)|' + /opt/docker/compose/rest-server-ana/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/rest-server-ana/compose.yaml + + - name: gitea-runner -> Agents (no UI) + sudo: true + shell: >- + sed -i 's|homepage.group=Toolchain$|homepage.group=Agents (no UI)|' + /opt/docker/compose/gitea-runner/compose.yaml + when: grep -q 'homepage.group=Toolchain$' /opt/docker/compose/gitea-runner/compose.yaml + + - name: rustdesk (hbbr) -> Agents (no UI) + sudo: true + shell: >- + sed -i 's|homepage.group=Apps$|homepage.group=Agents (no UI)|' + /opt/docker/compose/rustdesk/compose.yaml + when: grep -q 'homepage.group=Apps$' /opt/docker/compose/rustdesk/compose.yaml + + # Two containers were both named plain "Open WebUI" and, once the ESH one + # joined this group, they landed side by side — same name, same icon family, + # only the description telling them apart. Site suffix, like Traefik/Dockge/ + # AdGuard already carry. + - name: openwebui (ana) -> "Open WebUI (ana)" + sudo: true + shell: >- + sed -i 's|homepage.name=Open WebUI$|homepage.name=Open WebUI (ana)|' + /opt/docker/compose/openwebui/compose.yaml + when: grep -q 'homepage.name=Open WebUI$' /opt/docker/compose/openwebui/compose.yaml + + # ---- recreates --------------------------------------------------------- + # traefik goes LAST: crowdsec is its bouncer, so bounce the bouncer first + # and let traefik come up against a settled agent. + - name: recreate memos + sudo: true + shell: cd /opt/docker/compose/memos && docker compose up -d memos + + - name: recreate miniflux + sudo: true + shell: cd /opt/docker/compose/miniflux && docker compose up -d miniflux + + - name: recreate nevermore-web + sudo: true + shell: cd /opt/docker/compose/nevermore && docker compose up -d nevermore-web + + - name: recreate searxng + sudo: true + shell: cd /opt/docker/compose/searxng && docker compose up -d searxng + + - name: recreate zed-fim-proxy + sudo: true + shell: cd /opt/docker/compose/zed-fim-proxy && docker compose up -d zed-fim-proxy + + - name: recreate mailrise + sudo: true + shell: cd /opt/docker/compose/mailrise && docker compose up -d mailrise + + - name: recreate rest-server + sudo: true + shell: cd /opt/docker/compose/rest-server-ana && docker compose up -d rest-server + + - name: recreate gitea-runner + sudo: true + shell: cd /opt/docker/compose/gitea-runner && docker compose up -d runner + + - name: recreate rustdesk relay + sudo: true + shell: cd /opt/docker/compose/rustdesk && docker compose up -d hbbr + + - name: recreate openwebui (ana) + sudo: true + shell: cd /opt/docker/compose/openwebui && docker compose up -d open-webui + + - name: recreate dockge + sudo: true + shell: cd /opt/docker/compose/dockge && docker compose up -d dockge + + - name: recreate adguardhome + sudo: true + shell: cd /opt/docker/compose/adguard-ana && docker compose up -d adguardhome + + - name: recreate crowdsec + sudo: true + shell: cd /opt/docker/compose/crowdsec && docker compose up -d crowdsec + + - name: recreate traefik + sudo: true + shell: cd /opt/docker/compose/traefik && docker compose up -d traefik + +verify: + - name: every relabelled container now carries its new group + sudo: true + changed_when: "false" + shell: >- + docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}' + memos miniflux nevermore-web searxng zed-fim-proxy adguardhome traefik + dockge crowdsec mailrise rest-server gitea-runner hbbr + + - name: no container is left in the retired Service Networking group + sudo: true + changed_when: "false" + shell: >- + test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')" + + # dig is not installed everywhere in the fleet, so fall back to the AdGuard + # UI — a resolver that serves its own dashboard on :8053 has come back up. + - name: adguard is back (DNS answer, or its UI if dig is absent) + changed_when: "false" + shell: >- + if command -v dig >/dev/null 2>&1; + then dig +short +time=3 +tries=2 @10.250.50.70 gitea.phasefinal.com | grep -q .; + else curl -sf -o /dev/null -m 8 http://10.250.50.70:8053/; fi + + # Retried, not one-shot: the first run of this playbook checked 0.12s after + # `Started` and got rc=7 while traefik was still binding. The container was + # fine — `:8380/` 301s to /dashboard/ and both public hostnames answered 200 + # seconds later. A recreate needs a moment; assert the settled state. + - name: traefik still routes + changed_when: "false" + shell: >- + for i in 1 2 3 4 5 6 7 8 9 10; do + curl -sfL -o /dev/null -m 5 http://127.0.0.1:8380/dashboard/ && exit 0; + sleep 3; done; exit 1 + + - name: everything is running + sudo: true + changed_when: "false" + shell: >- + test "$(docker inspect -f '{{.State.Running}}' memos miniflux nevermore-web + searxng zed-fim-proxy adguardhome traefik dockge crowdsec mailrise + rest-server gitea-runner hbbr | sort -u)" = "true" diff --git a/playbooks/homepage-regroup-ana-ml2.yaml b/playbooks/homepage-regroup-ana-ml2.yaml new file mode 100644 index 0000000..988c565 --- /dev/null +++ b/playbooks/homepage-regroup-ana-ml2.yaml @@ -0,0 +1,74 @@ +# Homepage recategorisation — ana-ml2 (10.250.50.54), 2 containers. +# Sibling of playbooks/homepage-regroup-ana-docker.yaml; rationale lives there. +# +# scriberr -> AI - Studios (a transcription UI you open, not an API seat) +# dockge -> Compose Consoles +# +# ⚠ THE vLLM SEATS ON THIS HOST ARE DELIBERATELY NOT TOUCHED. Every one of them +# would need a recreate to change its `homepage.group`, and a recreate means a +# multi-minute model reload on a seat that peers reach through the gateway. The +# separation the operator asked for — UI up top, API endpoints out of the way — +# is achieved for those groups by ORDER and `initiallyCollapsed` in +# stacks/homepage/conf/settings.yaml, which costs nothing. Keep it that way: if +# a future pass wants to rename `AI - Inference`, weigh it against bouncing six +# model seats. +# +# Run: scripts/elway infra-ops@10.250.50.54 --playbook playbooks/homepage-regroup-ana-ml2.yaml + +steps: + - name: scriberr -> AI - Studios + sudo: true + shell: >- + sed -i 's|homepage.group=AI - Audio Tools$|homepage.group=AI - Studios|' + /opt/docker/compose/scriberr/compose.yaml + when: grep -q 'homepage.group=AI - Audio Tools$' /opt/docker/compose/scriberr/compose.yaml + + - name: dockge -> Compose Consoles + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|' + /opt/docker/compose/dockge/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml + + - name: recreate dockge + sudo: true + shell: cd /opt/docker/compose/dockge && docker compose up -d dockge + + - name: recreate scriberr + sudo: true + shell: cd /opt/docker/compose/scriberr && docker compose up -d scriberr + +verify: + - name: every relabelled container now carries its new group + sudo: true + changed_when: "false" + shell: >- + docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}' + scriberr dockge + + - name: no container is left in the retired Service Networking group + sudo: true + changed_when: "false" + shell: >- + test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')" + + # Container age, not liveness — a seat may be legitimately stopped, so + # "is it running" cannot answer "did I bounce it". See the same step in + # playbooks/homepage-regroup-irv-ml1.yaml for how that distinction was found. + - name: the vLLM seats were NOT recreated by this run + sudo: true + changed_when: "false" + shell: >- + for c in $(docker ps -a --filter 'name=vllm-' --filter 'name=llama-' + --format '{{.Names}}'); do + created=$(docker inspect -f '{{.Created}}' "$c" 2>/dev/null) || continue; + age=$(( $(date +%s) - $(date -d "$created" +%s) )); + if [ "$age" -lt 600 ]; then echo "$c was recreated ${age}s ago"; exit 1; fi; + done + + - name: scriberr answers + changed_when: "false" + shell: >- + for i in 1 2 3 4 5 6 7 8 9 10 11 12; do + curl -sfL -o /dev/null -m 5 http://127.0.0.1:8080/ && exit 0; + sleep 5; done; exit 1 diff --git a/playbooks/homepage-regroup-esh-docker-vm.yaml b/playbooks/homepage-regroup-esh-docker-vm.yaml new file mode 100644 index 0000000..bd7256a --- /dev/null +++ b/playbooks/homepage-regroup-esh-docker-vm.yaml @@ -0,0 +1,135 @@ +# Homepage recategorisation — esh-docker-vm (10.0.50.45), 6 containers. +# Sibling of playbooks/homepage-regroup-ana-docker.yaml; the rationale, the +# label-at-creation constraint and the idempotency scheme are documented there. +# +# lobe-chat, open-webui -> AI - Gateways & Chat (chat frontends belong +# with the other chat frontends, not in Apps) +# adguardhome -> DNS & Filtering +# traefik -> Reverse Proxies +# dockge -> Compose Consoles +# mosquitto -> Agents (no UI) (an MQTT broker has no page) +# +# ⚠ adguard and traefik here use `docker-compose.yml`, not `compose.yaml`. +# +# Run: scripts/elway infra-ops@10.0.50.45 --playbook playbooks/homepage-regroup-esh-docker-vm.yaml + +steps: + - name: lobe-chat -> AI - Gateways & Chat + sudo: true + shell: >- + sed -i 's|homepage.group=Apps$|homepage.group=AI - Gateways \& Chat|' + /opt/docker/compose/lobe-chat/compose.yaml + when: grep -q 'homepage.group=Apps$' /opt/docker/compose/lobe-chat/compose.yaml + + - name: open-webui -> AI - Gateways & Chat + sudo: true + shell: >- + sed -i 's|homepage.group=Apps$|homepage.group=AI - Gateways \& Chat|' + /opt/docker/compose/open-webui/compose.yaml + when: grep -q 'homepage.group=Apps$' /opt/docker/compose/open-webui/compose.yaml + + # Site suffix — the ana instance is also called "Open WebUI" and the two now + # sit side by side in the same group. See the sibling step in + # playbooks/homepage-regroup-ana-docker.yaml. + - name: open-webui -> "Open WebUI (esh)" + sudo: true + shell: >- + sed -i 's|homepage.name=Open WebUI$|homepage.name=Open WebUI (esh)|' + /opt/docker/compose/open-webui/compose.yaml + when: grep -q 'homepage.name=Open WebUI$' /opt/docker/compose/open-webui/compose.yaml + + - name: mosquitto -> Agents (no UI) + sudo: true + shell: >- + sed -i 's|homepage.group=Apps$|homepage.group=Agents (no UI)|' + /opt/docker/compose/mosquitto/compose.yaml + when: grep -q 'homepage.group=Apps$' /opt/docker/compose/mosquitto/compose.yaml + + - name: adguardhome -> DNS & Filtering + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=DNS \& Filtering|' + /opt/docker/compose/adguard/docker-compose.yml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/adguard/docker-compose.yml + + - name: traefik -> Reverse Proxies + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Reverse Proxies|' + /opt/docker/compose/traefik/docker-compose.yml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/traefik/docker-compose.yml + + - name: dockge -> Compose Consoles + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|' + /opt/docker/compose/dockge/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml + + # ---- recreates --------------------------------------------------------- + - name: recreate lobe-chat + sudo: true + shell: cd /opt/docker/compose/lobe-chat && docker compose up -d lobe-chat + + - name: recreate open-webui + sudo: true + shell: cd /opt/docker/compose/open-webui && docker compose up -d open-webui + + - name: recreate mosquitto + sudo: true + shell: cd /opt/docker/compose/mosquitto && docker compose up -d mosquitto + + - name: recreate dockge + sudo: true + shell: cd /opt/docker/compose/dockge && docker compose up -d dockge + + - name: recreate adguardhome + sudo: true + shell: cd /opt/docker/compose/adguard && docker compose up -d adguardhome + + - name: recreate traefik + sudo: true + shell: cd /opt/docker/compose/traefik && docker compose up -d traefik + +verify: + - name: every relabelled container now carries its new group + sudo: true + changed_when: "false" + shell: >- + docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}' + lobe-chat open-webui mosquitto adguardhome traefik dockge + + - name: no container is left in the retired Service Networking group + sudo: true + changed_when: "false" + shell: >- + test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')" + + - name: adguard is back + changed_when: "false" + shell: >- + for i in 1 2 3 4 5 6 7 8 9 10; do + curl -sfL -o /dev/null -m 5 http://127.0.0.1:8080/ && exit 0; + sleep 3; done; exit 1 + + - name: traefik still routes + changed_when: "false" + shell: >- + for i in 1 2 3 4 5 6 7 8 9 10; do + curl -sfL -o /dev/null -m 5 http://127.0.0.1:8380/dashboard/ && exit 0; + sleep 3; done; exit 1 + + # ⚠ Must be 10.0.50.45, NOT 127.0.0.1. `HOMEPAGE_ALLOWED_HOSTS` matches + # host AND port, and `127.0.0.1:5100` is not in the list — it answers 400 + # while the dashboard is perfectly healthy. The first run of this playbook + # failed here on rc=22 for exactly that reason. + - name: the dashboard itself is still served + changed_when: "false" + shell: curl -sf -o /dev/null -m 10 http://10.0.50.45:5100/api/services + + - name: everything is running + sudo: true + changed_when: "false" + shell: >- + test "$(docker inspect -f '{{.State.Running}}' lobe-chat open-webui + mosquitto adguardhome traefik dockge | sort -u)" = "true" diff --git a/playbooks/homepage-regroup-irv-ml1.yaml b/playbooks/homepage-regroup-irv-ml1.yaml new file mode 100644 index 0000000..9db1e4b --- /dev/null +++ b/playbooks/homepage-regroup-irv-ml1.yaml @@ -0,0 +1,115 @@ +# Homepage recategorisation — irv-ml1 (10.100.79.3), 5 containers. +# Sibling of playbooks/homepage-regroup-ana-docker.yaml; rationale lives there. +# +# arbo, comfyui, waterland-studio -> AI - Studios (was AI - Image & Media) +# yt-voice-clipper -> AI - Studios (was AI - Audio Tools) +# dockge -> Compose Consoles +# +# `AI - Studios` is the "you open this and do work in it" group; the ASR and +# TTS API seats stay where they are and get collapsed by settings.yaml instead, +# which is what keeps the GPU seats out of this playbook entirely. +# +# ⚠ yt-voice-clipper carries its homepage labels in `docker-compose.override.yml`, +# not in `docker-compose.yml`, and its compose dir is a git checkout of the +# project — the override is the deploy-local layer, which is the right place +# for it. +# +# ⚠ This host is reached over the WireGuard tunnel. If the run cannot connect, +# check the tunnel before assuming the host is down. +# +# Run: scripts/elway infra-ops@10.100.79.3 --playbook playbooks/homepage-regroup-irv-ml1.yaml + +steps: + - name: arbo -> AI - Studios + sudo: true + shell: >- + sed -i 's|homepage.group=AI - Image . Media$|homepage.group=AI - Studios|' + /opt/docker/compose/arbo/compose.yaml + when: grep -q 'homepage.group=AI - Image . Media$' /opt/docker/compose/arbo/compose.yaml + + - name: comfyui -> AI - Studios + sudo: true + shell: >- + sed -i 's|homepage.group=AI - Image . Media$|homepage.group=AI - Studios|' + /opt/docker/compose/comfyui/compose.yaml + when: grep -q 'homepage.group=AI - Image . Media$' /opt/docker/compose/comfyui/compose.yaml + + - name: waterland-studio -> AI - Studios + sudo: true + shell: >- + sed -i 's|homepage.group=AI - Image . Media$|homepage.group=AI - Studios|' + /opt/docker/compose/waterland-studio/compose.yaml + when: grep -q 'homepage.group=AI - Image . Media$' /opt/docker/compose/waterland-studio/compose.yaml + + - name: yt-voice-clipper -> AI - Studios + sudo: true + shell: >- + sed -i 's|homepage.group=AI - Audio Tools$|homepage.group=AI - Studios|' + /opt/docker/compose/yt-voice-clipper/docker-compose.override.yml + when: grep -q 'homepage.group=AI - Audio Tools$' /opt/docker/compose/yt-voice-clipper/docker-compose.override.yml + + - name: dockge -> Compose Consoles + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|' + /opt/docker/compose/dockge/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml + + # ---- recreates --------------------------------------------------------- + - name: recreate arbo + sudo: true + shell: cd /opt/docker/compose/arbo && docker compose up -d engine + + - name: recreate comfyui + sudo: true + shell: cd /opt/docker/compose/comfyui && docker compose up -d comfyui + + - name: recreate waterland-studio + sudo: true + shell: cd /opt/docker/compose/waterland-studio && docker compose up -d waterland-studio + + - name: recreate yt-voice-clipper + sudo: true + shell: cd /opt/docker/compose/yt-voice-clipper && docker compose up -d api + + - name: recreate dockge + sudo: true + shell: cd /opt/docker/compose/dockge && docker compose up -d dockge + +verify: + - name: every relabelled container now carries its new group + sudo: true + changed_when: "false" + shell: >- + docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}' + arbo comfyui waterland-studio yt-voice-clipper-api-1 dockge + + - name: no container is left in the retired Service Networking group + sudo: true + changed_when: "false" + shell: >- + test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')" + + # Assert NOT-RECREATED, not RUNNING. The first version of this checked that + # all five seats were up and failed — because chatterbox-fast has been down + # since 2026-08-10 and speaches since earlier the same morning, both long + # before this playbook existed. "Is it running" is the wrong question: a seat + # can be legitimately stopped. The question this step is actually asking is + # "did I bounce a model seat to relabel a dashboard", and container age + # answers it directly. + - name: the TTS and ASR seats were NOT recreated by this run + sudo: true + changed_when: "false" + shell: >- + for c in kokoro dots-tts chatterbox-fast tts-gateway parakeet speaches; do + created=$(docker inspect -f '{{.Created}}' "$c" 2>/dev/null) || continue; + age=$(( $(date +%s) - $(date -d "$created" +%s) )); + if [ "$age" -lt 600 ]; then echo "$c was recreated ${age}s ago"; exit 1; fi; + done + + - name: everything relabelled is running + sudo: true + changed_when: "false" + shell: >- + test "$(docker inspect -f '{{.State.Running}}' arbo comfyui + waterland-studio yt-voice-clipper-api-1 dockge | sort -u)" = "true" diff --git a/playbooks/homepage-regroup-nh3-docker.yaml b/playbooks/homepage-regroup-nh3-docker.yaml new file mode 100644 index 0000000..1795f9e --- /dev/null +++ b/playbooks/homepage-regroup-nh3-docker.yaml @@ -0,0 +1,58 @@ +# Homepage recategorisation — nh3-docker (10.100.50.40), 2 containers. +# Sibling of playbooks/homepage-regroup-ana-docker.yaml; rationale lives there. +# +# adguardhome -> DNS & Filtering (⚠ uses docker-compose.yml) +# dockge -> Compose Consoles +# +# Run: scripts/elway infra-ops@10.100.50.40 --playbook playbooks/homepage-regroup-nh3-docker.yaml + +steps: + - name: adguardhome -> DNS & Filtering + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=DNS \& Filtering|' + /opt/docker/compose/adguard/docker-compose.yml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/adguard/docker-compose.yml + + - name: dockge -> Compose Consoles + sudo: true + shell: >- + sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|' + /opt/docker/compose/dockge/compose.yaml + when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml + + - name: recreate dockge + sudo: true + shell: cd /opt/docker/compose/dockge && docker compose up -d dockge + + - name: recreate adguardhome + sudo: true + shell: cd /opt/docker/compose/adguard && docker compose up -d adguardhome + +verify: + - name: every relabelled container now carries its new group + sudo: true + changed_when: "false" + shell: >- + docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}' + adguardhome dockge + + - name: no container is left in the retired Service Networking group + sudo: true + changed_when: "false" + shell: >- + test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')" + + - name: adguard is back + changed_when: "false" + shell: >- + for i in 1 2 3 4 5 6 7 8 9 10; do + curl -sfL -o /dev/null -m 5 http://127.0.0.1:8080/ && exit 0; + sleep 3; done; exit 1 + + - name: everything is running + sudo: true + changed_when: "false" + shell: >- + test "$(docker inspect -f '{{.State.Running}}' adguardhome dockge + | sort -u)" = "true" diff --git a/stacks/adguard-ana/.env.example b/stacks/adguard-ana/.env.example new file mode 100644 index 0000000..bdd5e16 --- /dev/null +++ b/stacks/adguard-ana/.env.example @@ -0,0 +1,16 @@ +# AdGuard Home — Anaheim. Copy to `.env` on ana-docker at +# /opt/docker/compose/adguard-ana/.env and chmod 600. Never commit the real file. + +# Web UI / API port. compose.yaml defaults to 8053 if unset. +AG_WEB_PORT=8053 + +# Password for the `infra-ops` AdGuard login, used ONLY by the Homepage +# query/blocked/latency widget (homepage.widget.password in compose.yaml). +# One credential authenticates against all three fleet instances (ANA, ESH, +# NH3) — verified 2026-08-24. Fetch it from the vault rather than copying it +# between boxes: +# +# secret get nh3-dev/adguard-infra-ops-password +# +# (`secret` is services/secrets-broker/secret on nh3-dev.) +ADGUARD_WIDGET_PASSWORD= diff --git a/stacks/adguard-ana/compose.yaml b/stacks/adguard-ana/compose.yaml index 536d8be..2808cd7 100644 --- a/stacks/adguard-ana/compose.yaml +++ b/stacks/adguard-ana/compose.yaml @@ -39,12 +39,26 @@ services: networks: - tnet labels: - - homepage.group=Service Networking + - homepage.group=DNS & Filtering - homepage.name=AdGuard (ANA) - - homepage.icon=mdi-dns + # si-adguard, matching the ESH and NH3 instances. This carried mdi-dns + # and was the only one of the three wearing a different mark. + - homepage.icon=si-adguard - homepage.description=DNS resolver + .internal zone (colo) - homepage.href=http://10.250.50.70:${AG_WEB_PORT:-8053} - homepage.siteMonitor=http://10.250.50.70:${AG_WEB_PORT:-8053} + # Query/blocked/latency strip, same as ESH and NH3. Added 2026-08-24: + # without it this was a short card sitting beside two tall ones, which + # is the whole reason the DNS group exists as its own band — a group's + # members should all carry a widget or none of them should. + # Credential is the fleet AdGuard `infra-ops` login, which authenticates + # against all three instances; it lives in this stack's .env on the host + # (never in git) and is vaulted at + # `secret get nh3-dev/adguard-infra-ops-password`. + - homepage.widget.type=adguard + - homepage.widget.url=http://10.250.50.70:${AG_WEB_PORT:-8053} + - homepage.widget.username=infra-ops + - homepage.widget.password=${ADGUARD_WIDGET_PASSWORD} volumes: adguard_work: {} diff --git a/stacks/arbo/compose.yaml b/stacks/arbo/compose.yaml index 7fc6527..96924f0 100644 --- a/stacks/arbo/compose.yaml +++ b/stacks/arbo/compose.yaml @@ -85,7 +85,7 @@ services: networks: - tnet labels: - - homepage.group=AI - Image & Media + - homepage.group=AI - Studios - homepage.name=Arbo - homepage.icon=mdi-image-multiple-outline - homepage.description=Catalog-driven ComfyUI engine (irv-ml1) diff --git a/stacks/comfyui/compose.yaml b/stacks/comfyui/compose.yaml index 99cbb3b..0013695 100644 --- a/stacks/comfyui/compose.yaml +++ b/stacks/comfyui/compose.yaml @@ -90,7 +90,7 @@ services: networks: - tnet labels: - - homepage.group=AI - Image & Media + - homepage.group=AI - Studios - homepage.name=ComfyUI - homepage.icon=mdi-image-auto-adjust - homepage.description=Node-based SD/Flux inference (irv-ml1) diff --git a/stacks/dockge/compose.yaml b/stacks/dockge/compose.yaml index 6a56c38..6e19707 100644 --- a/stacks/dockge/compose.yaml +++ b/stacks/dockge/compose.yaml @@ -27,7 +27,7 @@ services: networks: - tnet labels: - - homepage.group=Service Networking + - homepage.group=Compose Consoles - homepage.name=Dockge (${DOCKGE_HOST_LABEL}) - homepage.icon=sh-dockge.png - homepage.description=Compose UI on ${DOCKGE_HOST_LABEL} diff --git a/stacks/gitea-runner/compose.yaml b/stacks/gitea-runner/compose.yaml index 965985e..167a6a1 100644 --- a/stacks/gitea-runner/compose.yaml +++ b/stacks/gitea-runner/compose.yaml @@ -27,7 +27,7 @@ services: networks: - tnet labels: - - homepage.group=Toolchain + - homepage.group=Agents (no UI) - homepage.name=gitea-runner - homepage.icon=mdi-cog-play - homepage.description=Gitea Actions self-hosted runner (${GITEA_RUNNER_NAME}) diff --git a/stacks/homepage/README.md b/stacks/homepage/README.md index 2555a2d..f8c56f8 100644 --- a/stacks/homepage/README.md +++ b/stacks/homepage/README.md @@ -33,11 +33,44 @@ recreated. ## Layout -`conf/settings.yaml` owns tabs, group order, and column counts — `services.yaml` -owns *what exists*, `settings.yaml` owns *where it sits*. Four tabs: Main, AI, -Infrastructure, Toolchain. A group listed in `layout:` with no members simply -renders empty, so a group can look "dead" when its provider host is unreachable -rather than when the group is wrong. +`conf/settings.yaml` owns tabs, group order, column counts and collapse state — +`services.yaml` owns *what exists*, `settings.yaml` owns *where it sits*. Four +tabs: Main, AI, Infrastructure, Toolchain. A group listed in `layout:` with no +members simply renders empty, so a group can look "dead" when its provider host +is unreachable rather than when the group is wrong. + +**The organising question is "do I open this?", not "what is it?"** (operator, +2026-08-24). Every group is either **tools** — expanded, top of its tab — or +**endpoints** (an API, a broker, a background agent, an href that is a `/docs` +page or a `/ping` or nothing) — `initiallyCollapsed: true`, bottom of its tab. +A collapsed group still renders its eyebrow and rule, so the tab tells you the +thing exists without spending a row on it. `AI - Inference` holds the seats the +whole fleet runs on and it is collapsed, because you reach them through the +gateway, not by clicking them. + +A second, quieter rule shapes the same block: **a group's members should all +have widgets or all not have them.** A stat strip makes a card ~50px taller, so +one widget card in a row of plain ones opens a void under the plain ones. That +is why AdGuard (3 widget cards) and Traefik (2 widget cards) are separate groups +from Dockge (5 plain cards) rather than one `Service Networking` band. + +Group *membership* is not in this file — it is the `homepage.group=` label on +each container, and labels are read at container **creation**. The 2026-08-24 +pass moved 28 services and the playbooks that did it are rerunnable: + +```bash +scripts/elway infra-ops@10.250.50.70 --playbook playbooks/homepage-regroup-ana-docker.yaml +scripts/elway infra-ops@10.0.50.45 --playbook playbooks/homepage-regroup-esh-docker-vm.yaml +scripts/elway infra-ops@10.100.50.40 --playbook playbooks/homepage-regroup-nh3-docker.yaml +scripts/elway infra-ops@10.100.79.3 --playbook playbooks/homepage-regroup-irv-ml1.yaml +scripts/elway infra-ops@10.250.50.54 --playbook playbooks/homepage-regroup-ana-ml2.yaml +``` + +⚠ **The GPU-backed model seats are deliberately still named `AI - Inference`, +`AI - Eval & Retrieval`, `AI - Speech (TTS)` and `AI - Audio Tools`.** Clearer +names would cost a recreate on sixteen seats — multi-minute model reloads on +endpoints peers reach through the gateway. Order and `initiallyCollapsed` buy +the same separation for free. Don't spend that recreate on a label. ## Foot-guns found in the 2026-08-17 audit @@ -124,18 +157,86 @@ is in the "iteration loop" note below. ## The tab bar goes missing after a recreate, then comes back on its own -**Status: self-healing, cause not pinned, and INTERMITTENT. Do not chase it.** +**Status: intermittent, self-healing, and now HALF-DIAGNOSED. Still don't chase +it — but you can now tell in one command whether you are in it.** + After a recreate the client render can come up with no tab bar, no wallpaper and no i18n (the search box shows the raw key `search.search`), with groups falling back to side-by-side columns. It restores itself with no intervention. -⚠ **It does not happen every time.** The 2026-08-24 recreate (`compose up -d ---force-recreate homepage`, to land a `settings.yaml` change) came up correct -within 10 seconds: the fresh `columns: 4` payload was in the served HTML on the -first poll, and a Playwright pass clicked through all four tabs a minute later -with the bar present. So do not pre-emptively schedule an hour of ugliness -around a config change — recreate, check, and only then walk away if it is in -the broken state. +### The one-command test + +```bash +curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}' +``` + +`initialSettings":{"…` — healthy. `initialSettings":{}` — you are in it. + +### What `initialSettings":{}` actually means (found 2026-08-24) + +It is **not** a warm-up, a cache, or a partial render. It is the **catch branch** +of the page's data loader. Decompiled from `/app/.next/server/pages/index.js`: + +```js +async function K() { + let a; + try { + a = logger("index"); // <-- assigned INSIDE the try + let { providers, ...d } = getSettings(); + ... await servicesResponse(), bookmarksResponse(), widgetsResponse() + return { props: { initialSettings: d, ... } }; + } catch (b) { + return a && b && a.error(b), // <-- guarded on `a` + { props: { initialSettings: {}, ... } }; + } +} +``` + +Two consequences worth knowing: + +1. **Something in that try block is throwing.** The page is not "still warming + up"; it has already failed and returned a degraded shape. +2. **The error can be swallowed completely.** `a` is assigned inside the same + try, and the catch only logs `if (a)`. If the *logger itself* is what threw, + `a` is `undefined` and nothing is written anywhere — which is exactly what + was observed on 2026-08-24: catch branch demonstrably taken, and not one + `index`-tagged line in `docker logs` or `conf/homepage/logs/homepage.log`. + +### What it is NOT — ruled out by measurement, don't re-run these + +- **Not the config.** `/api/services`, `/api/bookmarks`, `/api/widgets` and + `/api/hash` all return **200 with fully correct content** while the page + serves `initialSettings":{}` — including the brand-new group structure, in the + right order. Every input the loader awaits works when called directly. +- **Not the 2026-08-24 layout rewrite.** Restoring the *previous, known-good* + `settings.yaml` and recreating reproduced the empty payload identically. (This + matches the 2026-08-19 finding that the pre-adoption backup config reproduces + it too.) +- **Not `/api/validate`,** which returns `[]` throughout. +- **Not disk, not permissions.** 206 GB free; the container runs as root and a + write test into `/app/config/logs` succeeds. + +### Timing, measured rather than assumed + +Wildly variable, which is the whole trap. On 2026-08-24 one recreate came up +correct **within 10 seconds**, and three consecutive recreates ~40 minutes later +were still empty after 60-120s each. The 2026-08-19 session measured a fresh +container still tab-less at 4m30s twice, and healthy again after roughly an hour. +`docker ps` reporting `healthy` says nothing about it — the container is serving, +the page is just wrong. + +**Practical rule: recreate, run the one-command test, and if it is empty, go do +something else and re-check.** Do not start editing config — that is how an hour +got spent in 2026-08-19 ruling out four causes that were never the cause (the +config, the v2.0.0 release, `PUID`/`PGID` and Docker discovery, and the server +side). Every one of those remains ruled out. + +**Next lead, for whoever picks this up:** move the `logger("index")` hypothesis +forward. The winston file logger writes to `conf/homepage/logs/homepage.log`, +which had grown to 8.6 MB and stopped being appended to at the same time the +render started failing. Rolling it aside is a one-liner and is the cheapest +thing left to try: +`sudo mv /opt/docker/conf/homepage/logs/homepage.log{,.rolled}` then recreate. **Timing, measured rather than assumed:** five minutes is NOT enough — a fresh container was still tab-less at 4m30s, twice. It was observed healthy again diff --git a/stacks/homepage/conf/services.yaml b/stacks/homepage/conf/services.yaml index 1bda6a4..9af2f36 100644 --- a/stacks/homepage/conf/services.yaml +++ b/stacks/homepage/conf/services.yaml @@ -27,24 +27,37 @@ icon: mdi-filmstrip siteMonitor: http://10.100.10.50:8090/healthz description: Media drop + upload-for-pickup + the standing agent link board — nh3-dev, 24h TTL except kept boards - - Voice Design Studio: - href: http://10.100.79.3:8216/ - icon: mdi-microphone - siteMonitor: http://10.100.79.3:8216/health - description: Mint, audition and keeper-mark synthetic fleet voices — irv-ml1, CPU-only - The Henge: href: http://park.phasefinal.com:8420/ icon: mdi-clipboard-check siteMonitor: http://park.phasefinal.com:8420/healthz description: Durable needs-attention / idea parking (stonehenge-park) — ana-docker + # Was its own one-card `Games` group, which burned a full 4-wide row on a + # single panel. It is an app you open; this is where apps you open live. + - Pterodactyl: + href: http://10.250.50.55/ + icon: mdi-gamepad-square + siteMonitor: http://10.250.50.55 + description: Game server panel -# The AI tab is fully Docker-auto-discovered. Each inference service carries -# a homepage.group=AI - label on its compose file (AI - Inference, -# AI - Eval & Retrieval, AI - Gateways & Chat, AI - Speech (TTS), -# AI - Audio Tools, AI - Image & Media). Tab assignment, group order, and -# column counts live in settings.yaml. Do not add entries here or they'll -# double up. To move a service between AI groups, change the label on its -# compose file and recreate the container (labels only apply on recreate). +# The AI tab is otherwise fully Docker-auto-discovered. Each service carries a +# homepage.group=AI - label on its compose file (AI - Gateways & Chat, +# AI - Studios, AI - Inference, AI - Eval & Retrieval, AI - Speech (TTS), +# AI - Audio Tools, AI - Dormant). Tab assignment, group order, columns and +# collapse state live in settings.yaml. Do not add a labelled container here as +# well or it renders twice. To move a service between AI groups, change the +# label on its compose file and recreate the container — labels are read at +# creation, so `restart` will not do it. +- AI - Studios: + # Manual entry — Voice Design Studio is a user-level systemd service on + # irv-ml1, not a Docker-labeled stack, so it cannot auto-discover. It sits + # with the other studios rather than in Apps: it is a workspace you open and + # produce something in, which is exactly what that group is for. + - Voice Design Studio: + href: http://10.100.79.3:8216/ + icon: mdi-microphone + siteMonitor: http://10.100.79.3:8216/health + description: Mint, audition and keeper-mark synthetic fleet voices — irv-ml1, CPU-only - Media: - Plex: @@ -67,13 +80,6 @@ key: '{{HOMEPAGE_VAR_JELLYFIN_KEY}}' enableBlocks: true -- Games: - - Pterodactyl: - href: http://10.250.50.55/ - icon: mdi-gamepad-square - siteMonitor: http://10.250.50.55 - description: Game server panel - - Infra - ANA: - ANA-Firewall: href: https://10.250.250.1 diff --git a/stacks/homepage/conf/settings.yaml b/stacks/homepage/conf/settings.yaml index 312588c..d5b299b 100644 --- a/stacks/homepage/conf/settings.yaml +++ b/stacks/homepage/conf/settings.yaml @@ -46,34 +46,55 @@ statusStyle: "" # reads as a rendering fault. useEqualHeights: false -# Function-first layout, four-tab split: -# Main - daily-use apps, media, bookmarks, monitoring -# AI - the inference fleet, grouped by role (see below) -# Infrastructure - hardware, hypervisors, BMCs (per site) -# Toolchain - backend services running but rarely clicked +# =========================================================================== +# THE ORGANISING QUESTION IS "DO I OPEN THIS?" — NOT "WHAT IS IT?" +# (operator, 2026-08-24: "most of the issues are that tools I use and have a +# UI are interspersed with API endpoints which are largely informational +# only. They might even go in their own cards or start collapsed.") # -# The AI tab splits the fleet by function so a 20+ service list reads as -# sorted groups instead of one endless column. Group membership is set by -# the homepage.group=AI - label on each service's compose file: -# AI - Inference LLM seats you call (gen, char-rp, char-rp-reasoning, summarizer) -# AI - Eval & Retrieval judges, reward, rerank, embed, image-quality -# AI - Gateways & Chat routing gateway, control plane, chat frontends -# AI - Speech (TTS) text-to-speech engines -# AI - Audio Tools speech-to-text + audio dataset tooling -# AI - Image & Media image/video generation + pipelines -# AI - Dormant stopped stacks (rollback seats, retired auditions) +# Every group on this board is one of two kinds, and they never mix: # -# AI TAB ORDER IS BY CLICKABILITY, NOT BY IMPORTANCE (operator, 2026-08-18). -# Groups render in the order they appear in this block, so the top of the tab -# is prime real estate and it should hold the things you actually open in a -# browser — chat frontends, ComfyUI, the control plane. Most of the model -# seats below them are vLLM API endpoints whose href is a `/docs` page: they -# are worth SEEING (status at a glance) but not worth reaching for, so they -# sink. Order is therefore: -# interactive UIs -> mixed -> API-only seats -> dormant -# If you add an AI group, place it by asking "would I click this?", not by -# how central the service is to the fleet. +# TOOLS — you click the card and do something in the thing it opens. +# Expanded, and placed at the TOP of its tab. +# ENDPOINTS — an API, a broker, a background agent. Its href is a `/docs` +# page, a `/ping`, or nothing at all. The only thing you want from +# the card is "is it alive". `initiallyCollapsed: true`, and placed +# at the BOTTOM of its tab. # +# A collapsed group is not hidden — the eyebrow and its rule still render, so +# the tab still tells you the thing exists, and one click expands it. That is +# the whole point: presence without cost. +# +# When you add a service, ask "would I open this in a browser to get work +# done?" If no, it belongs in a collapsed endpoint group, no matter how +# central it is to the fleet. `AI - Inference` holds the seats the entire +# fleet runs on and it is collapsed, because you consume them through the +# gateway rather than by clicking them. +# +# Four tabs: +# Main - what you actually open day to day +# AI - AI tools up top, model/API seats collapsed below +# Infrastructure - hardware, hypervisors, BMCs (per site) — all consoles +# Toolchain - the plumbing, split by kind of plumbing +# +# --------------------------------------------------------------------------- +# GROUP MEMBERSHIP LIVES ON THE CONTAINER, NOT HERE. +# +# This block controls tab, order, columns and collapse. WHICH services are in +# a group is set by `homepage.group=` on each container's compose file, and +# labels only apply at container CREATION — moving a service between groups +# means editing the label and running `docker compose up -d `, not +# `restart`. The 2026-08-24 pass did 28 of those; the playbooks that did it +# are `playbooks/homepage-regroup-.yaml` and they are rerunnable. +# +# ⚠ THE MODEL SEATS ARE DELIBERATELY STILL NAMED `AI - Inference`, +# `AI - Eval & Retrieval`, `AI - Speech (TTS)` AND `AI - Audio Tools`. +# Renaming them to something like "AI API - …" would be clearer, and it would +# cost a recreate on sixteen GPU-backed seats — multi-minute model reloads on +# endpoints peers reach through the gateway. Order and `initiallyCollapsed` +# buy the same separation for free. Do not spend that recreate on a label. +# +# --------------------------------------------------------------------------- # COLUMNS ARE 4 EVERYWHERE. DO NOT TUNE THEM PER GROUP. # # `columns: N` is not a density dial — it sets `lg:grid-cols-N` on that one @@ -97,24 +118,36 @@ useEqualHeights: false # still collapses this to 2-up and 1-up on narrow viewports, so 4 is a desktop # maximum, not a hard floor. # +# --------------------------------------------------------------------------- +# GROUPS ARE ALSO KEPT UNIFORM IN CARD HEIGHT, WHICH IS WHY ADGUARD AND +# TRAEFIK GOT THEIR OWN GROUPS. +# +# A service with a widget (AdGuard's query counts, Traefik's router counts, +# Uptime Kuma's uptime) renders a stat strip that makes its card ~50px taller +# than a plain link card. Put one of those in a row of three plain cards and +# you get a void under the plain ones — which is what made the old 13-member +# `Service Networking` group look broken. Split so that a group's members all +# have widgets or all do not, and every row comes out flush. That is the real +# reason `DNS & Filtering` (3 widget cards) and `Reverse Proxies` (2 widget +# cards) are separate from `Compose Consoles` (5 plain cards). +# +# --------------------------------------------------------------------------- # EVERY GROUP NEEDS A `tab:` — including bookmark groups, and including groups # that arrive from a `homepage.group=` container label rather than from this # file. A group with no tab assignment renders on ALL FOUR TABS. That is how # UltraSeedbox ended up repeated at the bottom of every tab (fixed 2026-08-18) # and how Scriberr's `AI Systems` label did the same from 2026-08-23 (fixed -# 2026-08-24 by relabelling it into `AI - Audio Tools`). It is Homepage -# behaviour, not a bug, and it will happen again to the next container labelled -# with a group name that does not appear below. +# 2026-08-24). It is Homepage behaviour, not a bug, and it will happen again to +# the next container labelled with a group name that does not appear below. # `GET /api/services` prints the live group list — anything in it that is not a # key here is currently leaking onto all four tabs. +# =========================================================================== layout: - Notes: - icon: mdi-note-text-outline - tab: Main - style: row - columns: 4 - News: - icon: mdi-rss + # ---- Main: what you actually open ------------------------------------- + # Replaces the old Notes (1 member) and News (2) bands, which each burned a + # full 4-wide row on a single card. + Daily: + icon: mdi-coffee-outline tab: Main style: row columns: 4 @@ -123,6 +156,9 @@ layout: tab: Main style: row columns: 4 + # Absorbed the old one-card Games band (Pterodactyl). Lost SearXNG to Daily, + # the two chat frontends to the AI tab, and Mosquitto + the RustDesk relay to + # Agents (no UI) — neither of those has a page to open. Apps: icon: mdi-apps tab: Main @@ -133,11 +169,6 @@ layout: tab: Main style: row columns: 4 - Games: - icon: mdi-gamepad-square - tab: Main - style: row - columns: 4 # Bookmarks. Listed here for the tab pin above all else — without it this # group appears on every tab. `style: row` also turns the eight entries # from full-width stacked bars into a compact grid. @@ -146,44 +177,53 @@ layout: tab: Main style: row columns: 4 - # --- AI tab: ordered interactive -> API-only -> dormant (see note above) --- - # Things you open: chat frontends, the control plane, the LiteLLM UI. + + # ---- AI: tools, then collapsed endpoints ------------------------------ + # Chat frontends and the control plane — Lobe Chat and the ESH Open WebUI + # joined from Main on 2026-08-24; they are chat frontends and belong with the + # other chat frontends. AI - Gateways & Chat: icon: mdi-router-network tab: AI style: row columns: 4 - # ComfyUI is a full node editor and Arbo has a real UI — both get clicked. - AI - Image & Media: - icon: mdi-image-multiple + # Replaces `AI - Image & Media`. Everything here is a workspace you open and + # produce something in: ComfyUI's node editor, Arbo, Waterland, the YT + # Voice Clipper audition console, Scriberr's transcription UI. + AI - Studios: + icon: mdi-palette-outline tab: AI style: row columns: 4 - # Mixed: YT Voice Clipper has an audition console, Parakeet is an API. - # Scriberr joined here 2026-08-24 (it was the stray `AI Systems` group). - AI - Audio Tools: - icon: mdi-waveform - tab: AI - style: row - columns: 4 - # Below here: model seats whose href is a vLLM `/docs` page. Status at a - # glance is the whole value; you consume these through the gateway, not by - # clicking them. + # ---- collapsed from here down: seats you call, not pages you open ------ + # Named `AI - Inference` rather than something clearer on purpose — see the + # warning above about what renaming these costs. AI - Inference: icon: mdi-brain tab: AI style: row columns: 4 + initiallyCollapsed: true AI - Eval & Retrieval: icon: mdi-scale-balance tab: AI style: row columns: 4 + initiallyCollapsed: true AI - Speech (TTS): icon: mdi-account-voice tab: AI style: row columns: 4 + initiallyCollapsed: true + # What is left of the old Audio Tools group after Scriberr and the YT Voice + # Clipper moved to Studios: the two ASR API seats. + AI - Audio Tools: + icon: mdi-waveform + tab: AI + style: row + columns: 4 + initiallyCollapsed: true # Stopped stacks kept for rollback / superseded seats / retired auditions. # They stay 'created' (not running) via `docker compose up --no-start`, so # they show here as offline cards and revive with `docker compose start`. @@ -192,6 +232,9 @@ layout: tab: AI style: row columns: 4 + initiallyCollapsed: true + + # ---- Infrastructure: every card is a console -------------------------- Infra - ANA: icon: si-proxmox tab: Infrastructure @@ -212,13 +255,38 @@ layout: tab: Infrastructure style: row columns: 4 - Service Networking: + + # ---- Toolchain: the plumbing, split by kind --------------------------- + # The old `Service Networking` group was thirteen members mixing three + # AdGuards, five Dockges, two Traefiks and four headless agents — widget + # cards next to plain ones next to things with no href at all. Split four + # ways on 2026-08-24. + DNS & Filtering: + icon: mdi-dns + tab: Toolchain + style: row + columns: 4 + Reverse Proxies: icon: mdi-transit-connection-variant tab: Toolchain style: row columns: 4 + Compose Consoles: + icon: mdi-docker + tab: Toolchain + style: row + columns: 4 Toolchain: icon: mdi-toolbox tab: Toolchain style: row columns: 4 + # No href, or an href that is an API. CrowdSec, Mailrise, the restic + # rest-server, the Gitea Actions runner, the MQTT broker, the RustDesk relay. + # You never open these; you only ever want to know they are up. + Agents (no UI): + icon: mdi-cog-transfer-outline + tab: Toolchain + style: row + columns: 4 + initiallyCollapsed: true diff --git a/stacks/lobe-chat/compose.yaml b/stacks/lobe-chat/compose.yaml index df62b0e..106cd90 100644 --- a/stacks/lobe-chat/compose.yaml +++ b/stacks/lobe-chat/compose.yaml @@ -120,7 +120,7 @@ services: networks: - tnet labels: - - homepage.group=Apps + - homepage.group=AI - Gateways & Chat - homepage.name=Lobe Chat - homepage.icon=mdi-chat-processing - homepage.description=Chat frontend over the LiteLLM gateway (eval) diff --git a/stacks/memos/compose.yaml b/stacks/memos/compose.yaml index f8dc4f2..2845a93 100644 --- a/stacks/memos/compose.yaml +++ b/stacks/memos/compose.yaml @@ -33,7 +33,7 @@ services: networks: - tnet labels: - - homepage.group=Notes + - homepage.group=Daily - homepage.name=Memos - homepage.icon=mdi-note-text-outline - homepage.description=Self-hosted note + memo server (ana-docker) diff --git a/stacks/miniflux/compose.yaml b/stacks/miniflux/compose.yaml index 114d28e..47e5f15 100644 --- a/stacks/miniflux/compose.yaml +++ b/stacks/miniflux/compose.yaml @@ -44,7 +44,7 @@ services: networks: - tnet labels: - - homepage.group=News + - homepage.group=Daily - homepage.name=Miniflux - homepage.icon=mdi-rss - homepage.description=RSS reader (subreddits, blogs, HN) diff --git a/stacks/open-webui/compose.yaml b/stacks/open-webui/compose.yaml index 9447316..1af61ed 100644 --- a/stacks/open-webui/compose.yaml +++ b/stacks/open-webui/compose.yaml @@ -106,8 +106,8 @@ services: networks: - tnet labels: - - homepage.group=Apps - - homepage.name=Open WebUI + - homepage.group=AI - Gateways & Chat + - homepage.name=Open WebUI (esh) - homepage.icon=mdi-chat-question - homepage.description=Env-declarative chat frontend over the LiteLLM gateway (eval vs Lobe) - homepage.href=http://10.0.50.45:${OPENWEBUI_PORT:-3211} diff --git a/stacks/rest-server-ana/compose.yaml b/stacks/rest-server-ana/compose.yaml index 1d14c30..a2e550a 100644 --- a/stacks/rest-server-ana/compose.yaml +++ b/stacks/rest-server-ana/compose.yaml @@ -52,7 +52,7 @@ services: networks: - tnet labels: - - homepage.group=Service Networking + - homepage.group=Agents (no UI) - homepage.name=Restic (rest-server) - homepage.icon=mdi-cloud-upload - homepage.description=Anaheim restic endpoint (data on NFS) diff --git a/stacks/scriberr/compose.yaml b/stacks/scriberr/compose.yaml index d4058b1..ac302da 100644 --- a/stacks/scriberr/compose.yaml +++ b/stacks/scriberr/compose.yaml @@ -90,13 +90,16 @@ services: networks: - tnet labels: - # `AI - Audio Tools`, not `AI Systems`. The group name has to match a key - # in the dashboard's settings.yaml `layout:` block — a group that appears - # nowhere in that block gets no `tab:` assignment, and Homepage renders an - # untabbed group on EVERY tab. `AI Systems` did exactly that from - # 2026-08-23 until it was caught; transcription belongs with the other ASR - # seats anyway (Parakeet, Speaches). - - homepage.group=AI - Audio Tools + # ⚠ The group name MUST match a key in the dashboard's settings.yaml + # `layout:` block. A group that appears nowhere in that block gets no + # `tab:` assignment, and Homepage renders an untabbed group on EVERY tab. + # This label read `AI Systems` — a group that existed nowhere — from + # 2026-08-23 until it was caught on 2026-08-24. + # `AI - Studios` and not one of the ASR groups because Scriberr is a + # transcription UI you open and work in, which is what Studios collects; + # the bare ASR endpoints (Parakeet, Speaches) live in the collapsed + # `AI - Audio Tools` group instead. + - homepage.group=AI - Studios - homepage.name=Scriberr - homepage.icon=mdi-microphone-message - homepage.description=Audio/video transcription + diarization (ana-ml2, GPU1) diff --git a/stacks/searxng/compose.yaml b/stacks/searxng/compose.yaml index 97cb302..16ad556 100644 --- a/stacks/searxng/compose.yaml +++ b/stacks/searxng/compose.yaml @@ -87,7 +87,7 @@ services: - traefik.http.routers.searxng.tls=true - traefik.http.routers.searxng.service=searxng - traefik.http.services.searxng.loadbalancer.server.port=8080 - - homepage.group=Apps + - homepage.group=Daily - homepage.name=SearXNG - homepage.icon=si-searxng - homepage.description=Privacy-respecting meta-search diff --git a/stacks/waterland-studio/compose.yaml b/stacks/waterland-studio/compose.yaml index 11f405e..463fa3e 100644 --- a/stacks/waterland-studio/compose.yaml +++ b/stacks/waterland-studio/compose.yaml @@ -82,7 +82,7 @@ services: networks: - tnet labels: - - homepage.group=AI - Image & Media + - homepage.group=AI - Studios - homepage.name=Waterland Studio - homepage.icon=mdi-watercolor - homepage.description=Watercolour plate + reveal animation renderer (irv-ml1, A6000) diff --git a/stacks/zed-fim-proxy/compose.yaml b/stacks/zed-fim-proxy/compose.yaml index 7284e75..3faec0e 100644 --- a/stacks/zed-fim-proxy/compose.yaml +++ b/stacks/zed-fim-proxy/compose.yaml @@ -29,7 +29,7 @@ services: retries: 3 start_period: 10s labels: - - homepage.group=AI - Gateways & Chat + - homepage.group=AI - Inference - homepage.name=Zed FIM Proxy - homepage.icon=mdi-code-braces-box - homepage.description=Keyless /v1/completions for Zed edit-predictions (coder-fast, ana-docker)