esh-pfi-infrastructure/stacks/homepage/README.md

# homepage — the fleet dashboard

`ghcr.io/gethomepage/homepage` on **esh-docker-vm** (`10.0.50.45:5100`), behind
Traefik as `eshhome` / `eshhome.esteban.net`. Config is plain YAML — no
database, no UI-written state — which is why it belongs in this repo like any
other stack.

**Brought under version control 2026-08-17.** Before that it was edited in
place on the host, and had accumulated six hand-rolled `services.yaml.bak-*`
files as its only version history. Those were removed; git is the history now.
Edit here, then `scripts/deploy-stack.sh esh-docker-vm homepage`.

## How services get on the dashboard

Two paths, and mixing them is the classic failure:

1. **Docker label auto-discovery** — the default. A stack carries
   `homepage.group=` / `homepage.name=` / `homepage.icon=` / `homepage.description=`
   / `homepage.href=` labels and appears automatically. `conf/docker.yaml`
   wires **five** engines over plaintext `:2375`: esh-docker-vm, ana-docker,
   nh3-docker, ana-ml2, irv-ml1.
2. **Manual entries in `conf/services.yaml`** — for anything that is not a
   labelled container on one of those five hosts: hardware, BMCs, hypervisors,
   printers, and user-level systemd services (The Booth, Voice Design Studio).

⚠ **Never list a labelled container manually — it renders twice.** The
comments in `services.yaml` mark which groups are auto-populated (AI ×7,
Service Networking, Monitoring). Respect them.

⚠ **Labels only apply on container recreate.** Changing `homepage.group=` on a
compose file and running `restart` does nothing; the container must be
recreated.

## Layout

`conf/settings.yaml` owns tabs, group order, column counts and collapse state —
`services.yaml` owns *what exists*, `settings.yaml` owns *where it sits*. Four
tabs: Main, AI, Infrastructure, Toolchain. A group listed in `layout:` with no
members simply renders empty, so a group can look "dead" when its provider host
is unreachable rather than when the group is wrong.

**The organising question is "do I open this?", not "what is it?"** (operator,
2026-08-24). Every group is either **tools** — expanded, top of its tab — or
**endpoints** (an API, a broker, a background agent, an href that is a `/docs`
page or a `/ping` or nothing) — `initiallyCollapsed: true`, bottom of its tab.
A collapsed group still renders its eyebrow and rule, so the tab tells you the
thing exists without spending a row on it. `AI - Inference` holds the seats the
whole fleet runs on and it is collapsed, because you reach them through the
gateway, not by clicking them.

A second, quieter rule shapes the same block: **a group's members should all
have widgets or all not have them.** A stat strip makes a card ~50px taller, so
one widget card in a row of plain ones opens a void under the plain ones. That
is why AdGuard (3 widget cards) and Traefik (2 widget cards) are separate groups
from Dockge (5 plain cards) rather than one `Service Networking` band.

Group *membership* is not in this file — it is the `homepage.group=` label on
each container, and labels are read at container **creation**. The 2026-08-24
pass moved 28 services and the playbooks that did it are rerunnable:

```bash
scripts/elway infra-ops@10.250.50.70 --playbook playbooks/homepage-regroup-ana-docker.yaml
scripts/elway infra-ops@10.0.50.45   --playbook playbooks/homepage-regroup-esh-docker-vm.yaml
scripts/elway infra-ops@10.100.50.40 --playbook playbooks/homepage-regroup-nh3-docker.yaml
scripts/elway infra-ops@10.100.79.3  --playbook playbooks/homepage-regroup-irv-ml1.yaml
scripts/elway infra-ops@10.250.50.54 --playbook playbooks/homepage-regroup-ana-ml2.yaml
```

⚠ **The GPU-backed model seats are deliberately still named `AI - Inference`,
`AI - Eval & Retrieval`, `AI - Speech (TTS)` and `AI - Audio Tools`.** Clearer
names would cost a recreate on sixteen seats — multi-minute model reloads on
endpoints peers reach through the gateway. Order and `initiallyCollapsed` buy
the same separation for free. Don't spend that recreate on a label.

## Foot-guns found in the 2026-08-17 audit

- **`HOMEPAGE_ALLOWED_HOSTS` matches host *and port*.** The entry `10.0.50.45`
  does **not** cover `http://10.0.50.45:5100/` — that combination was being
  rejected with `Host validation failed` in the container log while the Traefik
  hostnames worked fine. **Fixed 2026-08-17**; every `host:port` the dashboard
  is reached by needs its own entry. See `.env.example`.
- **The `.env` was mode 644** with the Plex and Jellyfin API keys in it. Now
  `600`. It is root-owned, so it needs the `infra-ops` identity to edit —
  `lkraven` on this host has password-sudo only.
- **Homepage renders client-side.** Grepping the served HTML to verify a config
  change gives false readings — first a stale prerender, then an empty page.
  `GET /api/services` is the honest check; it returns the resolved group/service
  tree. Config edits also need a **container recreate**, not a restart: a
  restart keeps the writable layer and its cached render.
- **`:2375` is plaintext and unauthenticated** on all five engines. Fine on a
  trusted LAN, and unchanged by this commit, but it is real exposure: anything
  that can reach those ports has full Docker control of that host. `docker.yaml`
  carries a commented TLS example for when that stops being acceptable.
- **`ping:` cards can only be judged from esh-docker-vm.** Probing them from
  another box gives false FAILs — ICMP is filtered across some site links. All
  34 entries were verified reachable *from the dashboard host* on 2026-08-17.

## 2026-08-18 cleanup

Three fixes, all in this stack's config except where noted:

- **UltraSeedbox appeared on all four tabs.** The bookmark group had no entry
  in `settings.yaml`'s `layout:` block at all, and Homepage's documented
  behaviour is that *"if a group has no tab specified (and tabs are set on
  other groups), services and bookmarks will be shown on all tabs."* It now
  carries `tab: Main` plus `style: row` / `columns: 4`, which also turns eight
  full-width bars into a compact grid. **Any group added without a `tab:` will
  do this again** — the rule is now written at the top of the layout block.
- **Uptime Kuma rendered twice.** It was listed manually under Monitoring in
  `services.yaml` *and* labelled `homepage.group=Apps` on its container. The
  manual block is gone; the container's label now says `Monitoring` and
  carries `homepage.siteMonitor`. The container was adopted into this repo at
  `stacks/uptimekuma/` in the same commit — it had been running unmanaged.
- **Column counts were fiction.** Several groups declared more columns than
  they had members, so the last row of each was mostly dead space (Notes: 1
  card in a 4-wide row). Columns now track member counts; see the rule in
  `settings.yaml`. Check with `GET /api/services`, which prints live per-group
  counts. ⚠ **Superseded 2026-08-24** — see below.

## 2026-08-24 layout + theme pass

- **Columns are 4 everywhere now; "columns = member count" is retired.**
  `columns:` sets `lg:grid-cols-N` for one group, so tuning it per group fixes
  that group's *card width* — and the width then changed at every group
  boundary. Measured on the live board: Notes rendered a single 1464px card,
  News and Media 728px, the AI tab alternated 360 / 728 / 286 down the page.
  The old rule was avoiding dead cells in a short last row and bought a worse
  defect. A short last row is what a grid looks like; a card wider than its
  neighbours is what a mistake looks like. Rule and reasoning are at the top of
  `settings.yaml`'s `layout:` block.
- **Scriberr's `AI Systems` group was on all four tabs** — the same untabbed-
  group behaviour as UltraSeedbox in 2026-08-18, arriving this time from a
  container label rather than from this file. Relabelled to `AI - Audio Tools`
  in `stacks/scriberr/compose.yaml` (its real home, alongside Parakeet and
  Speaches) and the container recreated. The `homepage.group=AI Systems` sample
  in the repo-root `CLAUDE.md` was the source and now carries the constraint.
- **Long service names printed under their own status pill.** `.service-name`
  reserved a 78px right gutter with `padding-right` and relied on
  `overflow: hidden` to hold it — but overflow clips at the *padding* box, so
  the gutter was spill room, not a guard. Every long name on the AI tab
  collided. It holds by wrapping now; see the comment on the rule.
- **Descriptions are clamped to three lines** (floor of two, unchanged). Four-
  and five-line descriptions were dragging their whole row 30-50px taller than
  the cards beside them.
- **Icons were grey smudges.** Homepage masks every mdi glyph over
  `--color-logo-start/stop`, stock slate-400 → slate-700, and the dark stop
  sank the bottom of each glyph into the card fill. Overridden to ice → aurora
  blue. ⚠ The override must sit on `html[class]`, not `:root` — Homepage sets
  the same variables on `.theme-slate`, which is on `<html>`, and a class beats
  `:root` on the same element.
- **Bookmark groups and Jellyfin's trailing stream rows** were the last two
  un-themed components; both now use the card/eyebrow vocabulary.

Verified with Playwright against the live board — per-group card width, card
height spread, and a geometric title-vs-status collision check. Script pattern
is in the "iteration loop" note below.

## The tab bar goes missing after a recreate, then comes back on its own

**Status: intermittent, self-healing, and now HALF-DIAGNOSED. Still don't chase
it — but you can now tell in one command whether you are in it.**

After a recreate the client render can come up with no tab bar, no wallpaper
and no i18n (the search box shows the raw key `search.search`), with groups
falling back to side-by-side columns. It restores itself with no intervention.

### The one-command test

```bash
curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}'
```

`initialSettings":{"…` — healthy. `initialSettings":{}` — you are in it.

### What `initialSettings":{}` actually means (found 2026-08-24)

It is **not** a warm-up, a cache, or a partial render. It is the **catch branch**
of the page's data loader. Decompiled from `/app/.next/server/pages/index.js`:

```js
async function K() {
  let a;
  try {
    a = logger("index");                    // <-- assigned INSIDE the try
    let { providers, ...d } = getSettings();
    ... await servicesResponse(), bookmarksResponse(), widgetsResponse()
    return { props: { initialSettings: d, ... } };
  } catch (b) {
    return a && b && a.error(b),            // <-- guarded on `a`
           { props: { initialSettings: {}, ... } };
  }
}
```

Two consequences worth knowing:

1. **Something in that try block is throwing.** The page is not "still warming
   up"; it has already failed and returned a degraded shape.
2. **The error can be swallowed completely.** `a` is assigned inside the same
   try, and the catch only logs `if (a)`. If the *logger itself* is what threw,
   `a` is `undefined` and nothing is written anywhere — which is exactly what
   was observed on 2026-08-24: catch branch demonstrably taken, and not one
   `index`-tagged line in `docker logs` or `conf/homepage/logs/homepage.log`.

### ONE CAUSE IS NOW KNOWN: a missing `theme:` key

**Removing `theme:` from `settings.yaml` reproduces this deterministically.**
Six force-recreates over seven minutes all served `initialSettings":{}` with the
key absent; restoring `theme: dark` rendered correctly on the next recreate in
12 seconds (2026-08-24). So the loader really can be thrown by config — just not
by the parts you would suspect, and never with a message.

That does **not** explain every occurrence: the same symptom has appeared with
`theme:` present and correct. Treat the missing key as one confirmed trigger,
not the whole story.

### What it is NOT — ruled out by measurement, don't re-run these

- **Not a downstream data failure.** `/api/services`, `/api/bookmarks`,
  `/api/widgets` and `/api/hash` all return **200 with fully correct content**
  while the page serves `initialSettings":{}` — including the brand-new group
  structure, in the right order. Every input the loader awaits works when called
  directly.
- **Not the 2026-08-24 layout rewrite.** Restoring the *previous, known-good*
  `settings.yaml` and recreating reproduced the empty payload identically. (This
  matches the 2026-08-19 finding that the pre-adoption backup config reproduces
  it too.)
- **Not `/api/validate`,** which returns `[]` throughout.
- **Not disk, not permissions.** 206 GB free; the container runs as root and a
  write test into `/app/config/logs` succeeds.
- **Probably not the log file.** Rolling the 8.6 MB `homepage.log` aside once
  coincided with an immediate recovery, which looked like a lead — but the same
  move did nothing during the `theme:`-key episode. Recorded so nobody chases
  it twice; the coincidence was almost certainly just the intermittency.

### Timing, measured rather than assumed

Wildly variable, which is the whole trap. On 2026-08-24 one recreate came up
correct **within 10 seconds**, and three consecutive recreates ~40 minutes later
were still empty after 60-120s each. The 2026-08-19 session measured a fresh
container still tab-less at 4m30s twice, and healthy again after roughly an hour.
`docker ps` reporting `healthy` says nothing about it — the container is serving,
the page is just wrong.

**Practical rule: recreate, run the one-command test, and if it is empty, go do
something else and re-check.** Do not start editing config — that is how an hour
got spent in 2026-08-19 ruling out four causes that were never the cause (the
config, the v2.0.0 release, `PUID`/`PGID` and Docker discovery, and the server
side). Every one of those remains ruled out.

**First thing to check, now that one cause is confirmed:** diff `settings.yaml`
against the last version that rendered. A key that Homepage's loader needs and
cannot find will do this silently — `theme:` is the one we know about, and
there may be others. `git log -p -- stacks/homepage/conf/settings.yaml` is
faster than any amount of container archaeology.

**Timing, measured rather than assumed:** five minutes is NOT enough — a fresh
container was still tab-less at 4m30s, twice. It was observed healthy again
after a gap of roughly an hour. So the window is somewhere between those, and
`docker ps` reporting `healthy` says nothing about it: the container is
serving, the page is just wrong.

**Practical rule: recreate, then walk away and re-check later.** Do not start
editing config to fix it — that is how an hour got spent ruling out four
causes that were never the cause:

- **Not the config.** Restoring `settings.yaml` *and* `services.yaml` to their
  committed versions reproduces it, as does the pre-adoption backup config in
  `/opt/docker-bu/conf/homepage/`.
- **Not the v2.0.0 release.** A throwaway container on `v1.13.2` (the last v1)
  shows identical symptoms, and the image never changed anyway — working and
  broken containers both report `v2.0.0` / rev `17456f2`.
- **Not `PUID`/`PGID`**, and not Docker discovery — tested with both, and with
  the socket unmounted entirely.
- **Not a server-side failure.** The server-rendered HTML still contains the
  tab markup, the background URL and `useEqualHeights` throughout, and
  `GET /api/validate` returns `[]`. Whatever is lost, is lost client-side, and
  with no page error, no failed chunk and no non-200.

If someone does want to pin it: the shape (recovers on a long timescale, no
client exception) points at a cache or revalidation window rather than a
config fault. Start there, not in this directory.

**What it is not** — both obvious suspects were tested and cleared:

- **Not the config changes above.** Restoring `settings.yaml` *and*
  `services.yaml` to their committed pre-cleanup versions reproduces the
  breakage exactly. So does the pre-adoption backup config in
  `/opt/docker-bu/conf/homepage/`.
- **Not the v2.0.0 release.** A throwaway container on `v1.13.2` (the last
  v1, 2026-06-09) against the same config shows identical symptoms. The image
  never changed anyway: the working container and the broken one both report
  `v2.0.0` / rev `17456f2`, and only one homepage image exists on the host.

**What is known.** The *server-rendered* HTML still contains the tab markup,
the wallhaven background URL and `useEqualHeights` — so `settings.yaml` is
being read and delivered correctly. The loss happens client-side, with **no
page error, no failed chunk and no non-200** beyond two unrelated Uptime Kuma
widget 403s. `GET /api/validate` returns `[]`. A fresh container never
renders tabs here regardless of image version, config version, `PUID`/`PGID`,
or whether Docker discovery is mounted at all.

Every throwaway container in that list was judged within ~30s of starting, so
they were all inside the same window — and that consistency read as a
reproduction when it was really the same measurement mistake five times over.
This sits alongside the existing warning that `docker ps` health and a correct
render are different questions.

Before/after evidence: `~/booth-data/homepage-cleanup/` on nh3-dev →
`http://10.100.10.50:8090/b/homepage-cleanup/` (24h TTL).

## Theme — Australis Skyfall

The dashboard is themed with **Australis Skyfall**, the operator-supplied
design system (handoff bundle, 2026-08-18). Skyfall is a dual-theme web system
built on the OKLCH rebuild of the Australis terminal palette: one lightness law
across every chromatic family, all hues cooler than neutral, a Sea neutral ramp
that drifts from ice-blue to ocean green as it brightens, and a "calm depth"
language of hairline + two-layer shadow on every elevated surface.

### `theme/` is the source; `conf/custom.css` is generated

```
theme/colors.css      vendored VERBATIM from the bundle
theme/layout.css      ditto — radii, the shadow/glow/rail system, motion
theme/typography.css  ditto — families, scale, weights, tracking
theme/fonts/          Supreme 400/500/700 woff2 (Fontshare ITF license)
theme/skyfall.css.in  the Homepage bindings — the only part that is ours
theme/build.py        inlines the fonts + tokens -> conf/custom.css
```

**Do not hand-edit `conf/custom.css`.** Change `skyfall.css.in`, run
`python3 stacks/homepage/theme/build.py`, then deploy.

**Do not hand-edit the three vendored files either.** `build.py` records their
SHA-256 and **fails the build** on a mismatch — a vendored file is either
byte-identical to the bundle or it is a fork wearing the bundle's name. Put the
override in `skyfall.css.in`, which is expressed entirely through Skyfall's
semantic layer (`--surface-*`, `--text-*`, `--border-*`, `--success/--danger/
--warning`) and never against a raw family token or a literal colour. That is
not fussiness: the theme this one replaced built a parallel palette "derived
from the philosophy" and had to be torn out twice.

### Light and dark (2026-08-24)

Both themes are first-class. Skyfall Day is the bundle's own light ramp —
surfaces at `--sea-94/96/98`, text at `--sea-15`, and every chromatic family
dropping to its `-deep` (L 0.48) step. Nothing about it was derived here.

Precedence, highest first:

1. **an explicit choice** — the toggle in the header strip, stored in
   `localStorage` under `skyfall-theme`;
2. **the OS preference** — `@media (prefers-color-scheme: light)`, applied only
   while no explicit choice exists;
3. **dark** — Skyfall's default.

Two pieces of plumbing make that work, and both are load-bearing:

- **`build.py` re-emits the vendored `[data-theme="light"]` blocks** in both
  forms — `[data-theme="light"], html.light` for an explicit choice, and a
  copy inside the media query scoped to
  `html:not([data-theme="dark"]):not([data-theme="light"])`. That `:not()` pair
  is what lets a stored *dark* choice survive a light-mode OS.
- **`conf/custom.js` renders the toggle**, because Homepage will not give us
  its own.

⚠ **Homepage's built-in theme toggle is unreachable, and reaching for it breaks
the dashboard.** The toggle renders only when `settings.yaml` leaves `theme:`
unpinned — but with the key absent, the page's data loader throws and serves
`initialSettings: {}` (no tab bar, no layout, no i18n). Measured 2026-08-24:
six force-recreates over seven minutes all came up empty with the key removed;
restoring `theme: dark` rendered correctly on the next recreate in 12 seconds.
`/api/services` stays 200 and correct the whole time, which is exactly why this
looks like a caching problem and is not one. **Leave `theme: dark` pinned.**

### Type: one canonical face, two documented substitutions

Skyfall names Bespoke Sans (display) / Supreme (body, UI) / Victor Mono Nerd
Font (data, code). Only **Supreme** was ever vendored into this repo, and
Skyfall's own notes call Victor Mono "user-supplied", so the other two are
stand-ins rather than deviations:

| role | Skyfall | here |
|---|---|---|
| display | Bespoke Sans | **Space Grotesk** (variable, latin subset) |
| body / UI | Supreme | **Supreme 400/500/700** — canonical |
| data / mono | Victor Mono Nerd Font | **JetBrains Mono** (variable) |

Swapping in the real faces is a two-line change: `FONTS_*` in `build.py` and
the `--font-display` / `--font-mono` overrides at the top of `skyfall.css.in`.

### Iterating on the theme — do NOT recreate the container

`custom.css` is fetched per request from `/api/config/custom.css`, so a CSS
change needs a **browser reload**, nothing more. It does **not** need a
container recreate, and it therefore does not owe the multi-minute layout
warm-up described below — that penalty only applies to the full tab/layout
render. Conflating the two costs an operator-visible 10 minutes per attempt,
which is exactly how the first version of this theme shipped overcooked
(operator, 2026-08-19).

Faster still, and the right way to judge a colour change before it touches the
live dashboard: **inject the candidate CSS into the running page and
screenshot it.** No deploy, no restart, seconds per iteration.

```js
// playwright, against the live dashboard
await p.goto('http://10.0.50.45:5100/', { waitUntil: 'networkidle' });
await p.addStyleTag({ content: candidateCss });
await p.screenshot({ path: 'preview.png', fullPage: true });
```

Only build + deploy once the render looks right.

The build exists for one reason: Homepage serves exactly two files out of its
config directory, `custom.css` and `custom.js`. There is no static route for
anything beside them, so a `@font-face` pointing at a vendored `.woff2` would
404 — the face has to arrive inside the stylesheet as a data: URI. That is also
why the background image is mounted at `/app/public/images` instead (see
`compose.yaml`), which *is* a real static route.

Only Supreme (body/UI) is embedded. A link dashboard has no display type, so
Bespoke Sans buys nothing; Victor Mono ships as 2.4 MB TTF statics per cut,
which is 30x the whole stylesheet for the handful of latency figures on the
cards. Both are named in the font stacks so they resolve if vendored later.

### The background is generated, not stock

`images/skyfall-aurora.webp` came out of **Arbo** (irv-ml1:8201) via its
`t2i-ui-background` workflow — "abstract full-bleed backgrounds, no subject".
Job `13f0891f4e42`, seed 26, flux2-klein-9b, 2048x1152, 1.6 MB PNG re-encoded
to a 22 KB WebP (smooth gradients compress absurdly well).

Regenerate with a different seed, or reword, via:

```bash
curl -s -X POST http://10.100.79.3:8201/workflows/t2i-ui-background/run \
  -H 'content-type: application/json' \
  -d '{"prompt":{"kind":"raw","text":"...","negative":"..."},
       "width":2048,"height":1152,"seed":26}'
# poll GET /jobs/<id>, then GET /artifacts/<id>/0
```

Note the prompt field is a discriminated union, not a string — a bare string
gets a 422. `{"kind":"raw","text":...}` is the shape.

### Two deviations from the system, on the record

1. **Skyfall forbids this background.** Its rule is "flat semantic surfaces; no
   photography, no textures", with exactly one permitted decorative motif — a
   subtle aurora gradient, on hero and empty-state areas only, *"never behind
   body text blocks"*. A dashboard is a body-text block. The background is here
   on the operator's explicit instruction to generate one with Arbo, and it is
   mitigated rather than excused: abstract with no subject, strictly cool
   temperature per the imagery rule, and held at `opacity: 14` so card contrast
   never pays for it. **That opacity is load-bearing, not decoration** — the
   render is brighter than the "mostly empty negative space" brief, and turning
   it up is what would make the cards fight the ribbon. If it ever reads busy,
   the system-faithful answer is to drop the image and let `--surface-sunken`
   be the canvas.
2. **Service icons stay full-colour vendor logos.** Skyfall wants one cool,
   technical icon family. Desaturating every logo from CSS would just make them
   illegible, so this one is simply not fixed.

## Open question

`ESH-FileBot` (`10.0.50.70`) is still described as "role TBC" — it responds to
ping, but nobody has written down what it does. Worth resolving or removing.