esh-pfi-infrastructure/stacks/homepage

homepage — the fleet dashboard

ghcr.io/gethomepage/homepage on esh-docker-vm (10.0.50.45:5100), behind Traefik as eshhome / eshhome.esteban.net. Config is plain YAML — no database, no UI-written state — which is why it belongs in this repo like any other stack.

Brought under version control 2026-08-17. Before that it was edited in place on the host, and had accumulated six hand-rolled services.yaml.bak-* files as its only version history. Those were removed; git is the history now. Edit here, then scripts/deploy-stack.sh esh-docker-vm homepage.

How services get on the dashboard

Two paths, and mixing them is the classic failure:

1. Docker label auto-discovery — the default. A stack carries homepage.group= / homepage.name= / homepage.icon= / homepage.description= / homepage.href= labels and appears automatically. conf/docker.yaml wires five engines over plaintext :2375: esh-docker-vm, ana-docker, nh3-docker, ana-ml2, irv-ml1.
2. Manual entries in conf/services.yaml — for anything that is not a labelled container on one of those five hosts: hardware, BMCs, hypervisors, printers, and user-level systemd services (The Booth, Voice Design Studio).

⚠ Never list a labelled container manually — it renders twice. The comments in services.yaml mark which groups are auto-populated (AI ×7, Service Networking, Monitoring). Respect them.

⚠ Labels only apply on container recreate. Changing homepage.group= on a compose file and running restart does nothing; the container must be recreated.

Layout

conf/settings.yaml owns tabs, group order, column counts and collapse state — services.yaml owns what exists, settings.yaml owns where it sits. Four tabs: Main, AI, Infrastructure, Toolchain. A group listed in layout: with no members simply renders empty, so a group can look "dead" when its provider host is unreachable rather than when the group is wrong.

The organising question is "do I open this?", not "what is it?" (operator, 2026-08-24). Every group is either tools — expanded, top of its tab — or endpoints (an API, a broker, a background agent, an href that is a /docs page or a /ping or nothing) — initiallyCollapsed: true, bottom of its tab. A collapsed group still renders its eyebrow and rule, so the tab tells you the thing exists without spending a row on it. AI - Inference holds the seats the whole fleet runs on and it is collapsed, because you reach them through the gateway, not by clicking them.

A second, quieter rule shapes the same block: a group's members should all have widgets or all not have them. A stat strip makes a card ~50px taller, so one widget card in a row of plain ones opens a void under the plain ones. That is why AdGuard (3 widget cards) and Traefik (2 widget cards) are separate groups from Dockge (5 plain cards) rather than one Service Networking band.

Group membership is not in this file — it is the homepage.group= label on each container, and labels are read at container creation. The 2026-08-24 pass moved 28 services and the playbooks that did it are rerunnable:

scripts/elway infra-ops@10.250.50.70 --playbook playbooks/homepage-regroup-ana-docker.yaml
scripts/elway infra-ops@10.0.50.45   --playbook playbooks/homepage-regroup-esh-docker-vm.yaml
scripts/elway infra-ops@10.100.50.40 --playbook playbooks/homepage-regroup-nh3-docker.yaml
scripts/elway infra-ops@10.100.79.3  --playbook playbooks/homepage-regroup-irv-ml1.yaml
scripts/elway infra-ops@10.250.50.54 --playbook playbooks/homepage-regroup-ana-ml2.yaml

⚠ The GPU-backed model seats are deliberately still named AI - Inference, AI - Eval & Retrieval, AI - Speech (TTS) and AI - Audio Tools. Clearer names would cost a recreate on sixteen seats — multi-minute model reloads on endpoints peers reach through the gateway. Order and initiallyCollapsed buy the same separation for free. Don't spend that recreate on a label.

Foot-guns found in the 2026-08-17 audit

• HOMEPAGE_ALLOWED_HOSTS matches host and port. The entry 10.0.50.45 does not cover http://10.0.50.45:5100/ — that combination was being rejected with Host validation failed in the container log while the Traefik hostnames worked fine. Fixed 2026-08-17; every host:port the dashboard is reached by needs its own entry. See .env.example.
• The .env was mode 644 with the Plex and Jellyfin API keys in it. Now 600. It is root-owned, so it needs the infra-ops identity to edit — lkraven on this host has password-sudo only.
• Homepage renders client-side. Grepping the served HTML to verify a config change gives false readings — first a stale prerender, then an empty page. GET /api/services is the honest check; it returns the resolved group/service tree. Config edits also need a container recreate, not a restart: a restart keeps the writable layer and its cached render.
• :2375 is plaintext and unauthenticated on all five engines. Fine on a trusted LAN, and unchanged by this commit, but it is real exposure: anything that can reach those ports has full Docker control of that host. docker.yaml carries a commented TLS example for when that stops being acceptable.
• ping: cards can only be judged from esh-docker-vm. Probing them from another box gives false FAILs — ICMP is filtered across some site links. All 34 entries were verified reachable from the dashboard host on 2026-08-17.

2026-08-18 cleanup

Three fixes, all in this stack's config except where noted:

• UltraSeedbox appeared on all four tabs. The bookmark group had no entry in settings.yaml's layout: block at all, and Homepage's documented behaviour is that "if a group has no tab specified (and tabs are set on other groups), services and bookmarks will be shown on all tabs." It now carries tab: Main plus style: row / columns: 4, which also turns eight full-width bars into a compact grid. Any group added without a tab: will do this again — the rule is now written at the top of the layout block.
• Uptime Kuma rendered twice. It was listed manually under Monitoring in services.yaml and labelled homepage.group=Apps on its container. The manual block is gone; the container's label now says Monitoring and carries homepage.siteMonitor. The container was adopted into this repo at stacks/uptimekuma/ in the same commit — it had been running unmanaged.
• Column counts were fiction. Several groups declared more columns than they had members, so the last row of each was mostly dead space (Notes: 1 card in a 4-wide row). Columns now track member counts; see the rule in settings.yaml. Check with GET /api/services, which prints live per-group counts. ⚠ Superseded 2026-08-24 — see below.

2026-08-24 layout + theme pass

• Columns are 4 everywhere now; "columns = member count" is retired. columns: sets lg:grid-cols-N for one group, so tuning it per group fixes that group's card width — and the width then changed at every group boundary. Measured on the live board: Notes rendered a single 1464px card, News and Media 728px, the AI tab alternated 360 / 728 / 286 down the page. The old rule was avoiding dead cells in a short last row and bought a worse defect. A short last row is what a grid looks like; a card wider than its neighbours is what a mistake looks like. Rule and reasoning are at the top of settings.yaml's layout: block.
• Scriberr's AI Systems group was on all four tabs — the same untabbed- group behaviour as UltraSeedbox in 2026-08-18, arriving this time from a container label rather than from this file. Relabelled to AI - Audio Tools in stacks/scriberr/compose.yaml (its real home, alongside Parakeet and Speaches) and the container recreated. The homepage.group=AI Systems sample in the repo-root CLAUDE.md was the source and now carries the constraint.
• Long service names printed under their own status pill. .service-name reserved a 78px right gutter with padding-right and relied on overflow: hidden to hold it — but overflow clips at the padding box, so the gutter was spill room, not a guard. Every long name on the AI tab collided. It holds by wrapping now; see the comment on the rule.
• Descriptions are clamped to three lines (floor of two, unchanged). Four- and five-line descriptions were dragging their whole row 30-50px taller than the cards beside them.
• Icons were grey smudges. Homepage masks every mdi glyph over --color-logo-start/stop, stock slate-400 → slate-700, and the dark stop sank the bottom of each glyph into the card fill. Overridden to ice → aurora blue. ⚠ The override must sit on html[class], not :root — Homepage sets the same variables on .theme-slate, which is on <html>, and a class beats :root on the same element.
• Bookmark groups and Jellyfin's trailing stream rows were the last two un-themed components; both now use the card/eyebrow vocabulary.

Verified with Playwright against the live board — per-group card width, card height spread, and a geometric title-vs-status collision check. Script pattern is in the "iteration loop" note below.

The tab bar goes missing after a recreate, then comes back on its own

Status: intermittent, self-healing, and now HALF-DIAGNOSED. Still don't chase it — but you can now tell in one command whether you are in it.

After a recreate the client render can come up with no tab bar, no wallpaper and no i18n (the search box shows the raw key search.search), with groups falling back to side-by-side columns. It restores itself with no intervention.

The one-command test

curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}'

initialSettings":{"… — healthy. initialSettings":{} — you are in it.

What initialSettings":{} actually means (found 2026-08-24)

It is not a warm-up, a cache, or a partial render. It is the catch branch of the page's data loader. Decompiled from /app/.next/server/pages/index.js:

async function K() {
  let a;
  try {
    a = logger("index");                    // <-- assigned INSIDE the try
    let { providers, ...d } = getSettings();
    ... await servicesResponse(), bookmarksResponse(), widgetsResponse()
    return { props: { initialSettings: d, ... } };
  } catch (b) {
    return a && b && a.error(b),            // <-- guarded on `a`
           { props: { initialSettings: {}, ... } };
  }
}

Two consequences worth knowing:

1. Something in that try block is throwing. The page is not "still warming up"; it has already failed and returned a degraded shape.
2. The error can be swallowed completely. a is assigned inside the same try, and the catch only logs if (a). If the logger itself is what threw, a is undefined and nothing is written anywhere — which is exactly what was observed on 2026-08-24: catch branch demonstrably taken, and not one index-tagged line in docker logs or conf/homepage/logs/homepage.log.

ONE CAUSE IS NOW KNOWN: a missing theme: key

Removing theme: from settings.yaml reproduces this deterministically. Six force-recreates over seven minutes all served initialSettings":{} with the key absent; restoring theme: dark rendered correctly on the next recreate in 12 seconds (2026-08-24). So the loader really can be thrown by config — just not by the parts you would suspect, and never with a message.

That does not explain every occurrence: the same symptom has appeared with theme: present and correct. Treat the missing key as one confirmed trigger, not the whole story.

What it is NOT — ruled out by measurement, don't re-run these

• Not a downstream data failure. /api/services, /api/bookmarks, /api/widgets and /api/hash all return 200 with fully correct content while the page serves initialSettings":{} — including the brand-new group structure, in the right order. Every input the loader awaits works when called directly.
• Not the 2026-08-24 layout rewrite. Restoring the previous, known-good settings.yaml and recreating reproduced the empty payload identically. (This matches the 2026-08-19 finding that the pre-adoption backup config reproduces it too.)
• Not /api/validate, which returns [] throughout.
• Not disk, not permissions. 206 GB free; the container runs as root and a write test into /app/config/logs succeeds.
• Probably not the log file. Rolling the 8.6 MB homepage.log aside once coincided with an immediate recovery, which looked like a lead — but the same move did nothing during the theme:-key episode. Recorded so nobody chases it twice; the coincidence was almost certainly just the intermittency.

Timing, measured rather than assumed

Wildly variable, which is the whole trap. On 2026-08-24 one recreate came up correct within 10 seconds, and three consecutive recreates ~40 minutes later were still empty after 60-120s each. The 2026-08-19 session measured a fresh container still tab-less at 4m30s twice, and healthy again after roughly an hour. docker ps reporting healthy says nothing about it — the container is serving, the page is just wrong.

Practical rule: recreate, run the one-command test, and if it is empty, go do something else and re-check. Do not start editing config — that is how an hour got spent in 2026-08-19 ruling out four causes that were never the cause (the config, the v2.0.0 release, PUID/PGID and Docker discovery, and the server side). Every one of those remains ruled out.

First thing to check, now that one cause is confirmed: diff settings.yaml against the last version that rendered. A key that Homepage's loader needs and cannot find will do this silently — theme: is the one we know about, and there may be others. git log -p -- stacks/homepage/conf/settings.yaml is faster than any amount of container archaeology.

Timing, measured rather than assumed: five minutes is NOT enough — a fresh container was still tab-less at 4m30s, twice. It was observed healthy again after a gap of roughly an hour. So the window is somewhere between those, and docker ps reporting healthy says nothing about it: the container is serving, the page is just wrong.

Practical rule: recreate, then walk away and re-check later. Do not start editing config to fix it — that is how an hour got spent ruling out four causes that were never the cause:

• Not the config. Restoring settings.yaml and services.yaml to their committed versions reproduces it, as does the pre-adoption backup config in /opt/docker-bu/conf/homepage/.
• Not the v2.0.0 release. A throwaway container on v1.13.2 (the last v1) shows identical symptoms, and the image never changed anyway — working and broken containers both report v2.0.0 / rev 17456f2.
• Not PUID/PGID, and not Docker discovery — tested with both, and with the socket unmounted entirely.
• Not a server-side failure. The server-rendered HTML still contains the tab markup, the background URL and useEqualHeights throughout, and GET /api/validate returns []. Whatever is lost, is lost client-side, and with no page error, no failed chunk and no non-200.

If someone does want to pin it: the shape (recovers on a long timescale, no client exception) points at a cache or revalidation window rather than a config fault. Start there, not in this directory.

What it is not — both obvious suspects were tested and cleared:

• Not the config changes above. Restoring settings.yaml and services.yaml to their committed pre-cleanup versions reproduces the breakage exactly. So does the pre-adoption backup config in /opt/docker-bu/conf/homepage/.
• Not the v2.0.0 release. A throwaway container on v1.13.2 (the last v1, 2026-06-09) against the same config shows identical symptoms. The image never changed anyway: the working container and the broken one both report v2.0.0 / rev 17456f2, and only one homepage image exists on the host.

What is known. The server-rendered HTML still contains the tab markup, the wallhaven background URL and useEqualHeights — so settings.yaml is being read and delivered correctly. The loss happens client-side, with no page error, no failed chunk and no non-200 beyond two unrelated Uptime Kuma widget 403s. GET /api/validate returns []. A fresh container never renders tabs here regardless of image version, config version, PUID/PGID, or whether Docker discovery is mounted at all.

Every throwaway container in that list was judged within ~30s of starting, so they were all inside the same window — and that consistency read as a reproduction when it was really the same measurement mistake five times over. This sits alongside the existing warning that docker ps health and a correct render are different questions.

Before/after evidence: ~/booth-data/homepage-cleanup/ on nh3-dev → http://10.100.10.50:8090/b/homepage-cleanup/ (24h TTL).

Theme — Australis Skyfall

The dashboard is themed with Australis Skyfall, the operator-supplied design system (handoff bundle, 2026-08-18). Skyfall is a dual-theme web system built on the OKLCH rebuild of the Australis terminal palette: one lightness law across every chromatic family, all hues cooler than neutral, a Sea neutral ramp that drifts from ice-blue to ocean green as it brightens, and a "calm depth" language of hairline + two-layer shadow on every elevated surface.

theme/ is the source; conf/custom.css is generated

theme/colors.css      vendored VERBATIM from the bundle
theme/layout.css      ditto — radii, the shadow/glow/rail system, motion
theme/typography.css  ditto — families, scale, weights, tracking
theme/fonts/          Supreme 400/500/700 woff2 (Fontshare ITF license)
theme/skyfall.css.in  the Homepage bindings — the only part that is ours
theme/build.py        inlines the fonts + tokens -> conf/custom.css

Do not hand-edit conf/custom.css. Change skyfall.css.in, run python3 stacks/homepage/theme/build.py, then deploy.

Do not hand-edit the three vendored files either. build.py records their SHA-256 and fails the build on a mismatch — a vendored file is either byte-identical to the bundle or it is a fork wearing the bundle's name. Put the override in skyfall.css.in, which is expressed entirely through Skyfall's semantic layer (--surface-*, --text-*, --border-*, --success/--danger/ --warning) and never against a raw family token or a literal colour. That is not fussiness: the theme this one replaced built a parallel palette "derived from the philosophy" and had to be torn out twice.

Light and dark (2026-08-24)

Both themes are first-class. Skyfall Day is the bundle's own light ramp — surfaces at --sea-94/96/98, text at --sea-15, and every chromatic family dropping to its -deep (L 0.48) step. Nothing about it was derived here.

Precedence, highest first:

1. an explicit choice — the toggle in the header strip, stored in localStorage under skyfall-theme;
2. the OS preference — @media (prefers-color-scheme: light), applied only while no explicit choice exists;
3. dark — Skyfall's default.

Two pieces of plumbing make that work, and both are load-bearing:

• build.py re-emits the vendored [data-theme="light"] blocks in both forms — [data-theme="light"], html.light for an explicit choice, and a copy inside the media query scoped to html:not([data-theme="dark"]):not([data-theme="light"]). That :not() pair is what lets a stored dark choice survive a light-mode OS.
• conf/custom.js renders the toggle, because Homepage will not give us its own.

⚠ Homepage's built-in theme toggle is unreachable, and reaching for it breaks the dashboard. The toggle renders only when settings.yaml leaves theme: unpinned — but with the key absent, the page's data loader throws and serves initialSettings: {} (no tab bar, no layout, no i18n). Measured 2026-08-24: six force-recreates over seven minutes all came up empty with the key removed; restoring theme: dark rendered correctly on the next recreate in 12 seconds. /api/services stays 200 and correct the whole time, which is exactly why this looks like a caching problem and is not one. Leave theme: dark pinned.

Type: one canonical face, two documented substitutions

Skyfall names Bespoke Sans (display) / Supreme (body, UI) / Victor Mono Nerd Font (data, code). Only Supreme was ever vendored into this repo, and Skyfall's own notes call Victor Mono "user-supplied", so the other two are stand-ins rather than deviations:

role	Skyfall	here
display	Bespoke Sans	Space Grotesk (variable, latin subset)
body / UI	Supreme	Supreme 400/500/700 — canonical
data / mono	Victor Mono Nerd Font	JetBrains Mono (variable)

Swapping in the real faces is a two-line change: FONTS_* in build.py and the --font-display / --font-mono overrides at the top of skyfall.css.in.

Iterating on the theme — do NOT recreate the container

custom.css is fetched per request from /api/config/custom.css, so a CSS change needs a browser reload, nothing more. It does not need a container recreate, and it therefore does not owe the multi-minute layout warm-up described below — that penalty only applies to the full tab/layout render. Conflating the two costs an operator-visible 10 minutes per attempt, which is exactly how the first version of this theme shipped overcooked (operator, 2026-08-19).

Faster still, and the right way to judge a colour change before it touches the live dashboard: inject the candidate CSS into the running page and screenshot it. No deploy, no restart, seconds per iteration.

// playwright, against the live dashboard
await p.goto('http://10.0.50.45:5100/', { waitUntil: 'networkidle' });
await p.addStyleTag({ content: candidateCss });
await p.screenshot({ path: 'preview.png', fullPage: true });

Only build + deploy once the render looks right.

The build exists for one reason: Homepage serves exactly two files out of its config directory, custom.css and custom.js. There is no static route for anything beside them, so a @font-face pointing at a vendored .woff2 would 404 — the face has to arrive inside the stylesheet as a data: URI. That is also why the background image is mounted at /app/public/images instead (see compose.yaml), which is a real static route.

Only Supreme (body/UI) is embedded. A link dashboard has no display type, so Bespoke Sans buys nothing; Victor Mono ships as 2.4 MB TTF statics per cut, which is 30x the whole stylesheet for the handful of latency figures on the cards. Both are named in the font stacks so they resolve if vendored later.

The background is generated, not stock

images/skyfall-aurora.webp came out of Arbo (irv-ml1:8201) via its t2i-ui-background workflow — "abstract full-bleed backgrounds, no subject". Job 13f0891f4e42, seed 26, flux2-klein-9b, 2048x1152, 1.6 MB PNG re-encoded to a 22 KB WebP (smooth gradients compress absurdly well).

Regenerate with a different seed, or reword, via:

curl -s -X POST http://10.100.79.3:8201/workflows/t2i-ui-background/run \
  -H 'content-type: application/json' \
  -d '{"prompt":{"kind":"raw","text":"...","negative":"..."},
       "width":2048,"height":1152,"seed":26}'
# poll GET /jobs/<id>, then GET /artifacts/<id>/0

Note the prompt field is a discriminated union, not a string — a bare string gets a 422. {"kind":"raw","text":...} is the shape.

Two deviations from the system, on the record

1. Skyfall forbids this background. Its rule is "flat semantic surfaces; no photography, no textures", with exactly one permitted decorative motif — a subtle aurora gradient, on hero and empty-state areas only, "never behind body text blocks". A dashboard is a body-text block. The background is here on the operator's explicit instruction to generate one with Arbo, and it is mitigated rather than excused: abstract with no subject, strictly cool temperature per the imagery rule, and held at opacity: 14 so card contrast never pays for it. That opacity is load-bearing, not decoration — the render is brighter than the "mostly empty negative space" brief, and turning it up is what would make the cards fight the ribbon. If it ever reads busy, the system-faithful answer is to drop the image and let --surface-sunken be the canvas.
2. Service icons stay full-colour vendor logos. Skyfall wants one cool, technical icon family. Desaturating every logo from CSS would just make them illegible, so this one is simply not fixed.

Open question

ESH-FileBot (10.0.50.70) is still described as "role TBC" — it responds to ping, but nobody has written down what it does. Worth resolving or removing.