feat(uptimekuma): rebuild on 2.5.5 as the fleet's service layer, with a client

Homepage sat dead for three days in September while every monitoring tool
reported correctly. Beszel said its host was up -- it was. Uptime Kuma was not
watching it. The dashboard fell into the seam between two working instruments.

Measured before changing anything:
  - Beszel: 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts
    table is (system, name, value, min) -- there is NO url column, so it is
    structurally incapable of "this endpoint should return 200". Not a config
    gap; the data model.
  - Uptime Kuma: 6 rows, 2 of them folders. Four real monitors, all firewalls.
  - So the two are NOT redundant. They are disjoint, and the service layer
    between them was empty.

REBUILT FROM SCRATCH, operator-authorised ("uptime-kuma was never really
used... you can even dump the existing container and config"). Nothing was
migrated, which also skipped the one-way v1->v2 database migration.

  - Pinned to 2.5.5. `:latest` is a documented trap now: upstream keeps it on
    the 1.x line, so an August 2026 pull produced an image BUILT 2024-12-20
    running 1.23.16. Verified by digest -- latest and 1 share one digest while
    2/next carry 2.5.5. Pinned exactly, not floating on 2, for the same reason.
  - Moved esh-docker-vm -> ana-docker. House placement rule puts cross-site
    services beside the Beszel and Dozzle hubs, and esh-docker-vm has wedged
    unkillably twice in four months. A monitor also cannot report the failure
    of the host it runs on, so it should not share a failure domain with the
    host layer.
  - Normalised restart: always -> unless-stopped, which the 2026-08-18 README
    flagged as worth doing on the next deliberate touch.
  - UPTIME_KUMA_DB_TYPE=sqlite in the compose skips 2.x's interactive database
    screen, so the stack comes up ready rather than parked on a form.

scripts/kuma is a first-party Socket.IO client, because both obvious paths are
wrong: there is no REST CRUD API in EITHER major version (server/routers/ holds
exactly two files, /metrics + badges + status pages), and the community wrapper
uptime-kuma-api is abandoned -- last release 2023-09-26, ceiling 1.23.1, no 2.x
support ever.

  ⚠ getMonitorList's callback returns only {ok:true}; the list arrives as a
  SEPARATE pushed monitorList event. Reading the ack yields an empty board that
  looks authoritative -- which duplicated all 13 rows on the first re-seed
  before the bug was found. The client now waits for the push, and carries a
  dedupe verb because of it.

13 monitors seeded from monitors.yaml, keyed on name so a re-run updates rather
than forking the board -- proven by re-running it (0 added, 13 updated), not
assumed. Every URL was probed before being written: all 200. A board that ships
red teaches everyone to ignore it.

Verified: 13 rows, no duplicates, all UP with "200 - OK" read from the database
WITH its WAL (a first read of kuma.db alone showed a stale 26 -- the copy
predated the deletes). Homepage renders exactly one Uptime Kuma card.

⚠ NOT YET WIRED: notification delivery. The board detects but tells nobody,
which is the same gap this work exists to close. The beszel-althing bridge
hardcodes a [Beszel] subject prefix and hub footer, so routing Kuma through it
unchanged would mislabel the alerts. Needs a decision before it is generalised.

scripts/kuma

@@ -0,0 +1,285 @@
+#!/usr/bin/env -S uv run --quiet --script
+# /// script
+# requires-python = ">=3.11"
+# dependencies = ["python-socketio[client]>=5.11", "requests>=2.31", "PyYAML>=6"]
+# ///
+"""kuma — a first-party Socket.IO client for Uptime Kuma 2.x.
+
+WHY THIS EXISTS, since the obvious answers are both wrong:
+
+  * There is NO REST CRUD API for monitors, in either major version. Checked
+    against the 2.5.5 source tree, not the docs: `server/routers/` holds exactly
+    two files -- api-router.js (Prometheus /metrics, badges, entry page) and
+    status-page-router.js. API keys unlock /metrics and badges only. Every
+    monitor operation lives on Socket.IO, the same channel the web UI uses.
+
+  * The community wrapper `uptime-kuma-api` is ABANDONED -- last release
+    2023-09-26 -- and its stated ceiling is "support for uptime kuma 1.23.0 and
+    1.23.1". It does not speak 2.x and never will. Depending on it would trade a
+    maintained server for an unmaintained client.
+
+So: a thin client of our own, against events read out of server.js at tag 2.5.5
+(`setup`, `login`, `getMonitorList`, `add`, `editMonitor`, `deleteMonitor`).
+Small on purpose -- this is four events and a socket, not a framework.
+
+USAGE
+    scripts/kuma setup                      # create the admin (first run only)
+    scripts/kuma list                       # every monitor, with its state
+    scripts/kuma seed <file.yaml>           # idempotent upsert from a file
+    scripts/kuma add <name> <url> [--type http|ping|port] [--port N]
+
+CREDENTIALS come from the vault, never from a flag:
+    secret get ana-docker/.config/uptime-kuma/admin
+Password on the command line lands in shell history and in the process table.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import subprocess
+import sys
+import time
+
+import socketio
+import yaml
+
+URL = "http://10.250.50.70:3001"
+VAULT_KEY = "ana-docker/.config/uptime-kuma/admin"
+
+# Defaults the 2.x `add` handler REQUIRES rather than defaults itself. It
+# JSON.stringify()s kafkaProducerBrokers, kafkaProducerSaslOptions, conditions
+# and rabbitmqNodes unconditionally, so they must be present even for an HTTP
+# check -- omit them and you get "undefined" written into the row.
+MONITOR_DEFAULTS = {
+    "type": "http",
+    "method": "GET",
+    "interval": 60,
+    "retryInterval": 60,
+    "resendInterval": 0,
+    "maxretries": 2,
+    "timeout": 16,
+    "maxredirects": 10,
+    "accepted_statuscodes": ["200-299"],
+    "notificationIDList": {},
+    "active": True,
+    "expiryNotification": False,
+    "ignoreTls": True,      # the fleet is full of self-signed internal certs
+    "upsideDown": False,
+    "kafkaProducerBrokers": [],
+    "kafkaProducerSaslOptions": {},
+    "conditions": [],
+    "rabbitmqNodes": [],
+}
+
+
+def vault_get(key: str) -> dict:
+    """{'username','password'} from the vault. Loud on failure -- a silent
+    empty credential would show up as a confusing auth error much later."""
+    out = subprocess.run(
+        ["secret", "get", key], capture_output=True, text=True, timeout=60
+    )
+    if out.returncode != 0:
+        sys.exit(f"vault: could not read {key}: {out.stderr.strip() or out.stdout.strip()}")
+    body = out.stdout.strip()
+    try:
+        return json.loads(body)
+    except json.JSONDecodeError:
+        # tolerate a "user:pass" single line as well
+        if ":" in body:
+            u, _, p = body.partition(":")
+            return {"username": u.strip(), "password": p.strip()}
+        sys.exit(f"vault: {key} is neither JSON nor user:pass")
+
+
+class Kuma:
+    def __init__(self, url: str = URL):
+        self.url = url
+        self.sio = socketio.Client(reconnection=False)
+        self._connected = False
+        # ⚠ THE TRAP THAT COST A DUPLICATED BOARD (2026-09-21).
+        # `getMonitorList`'s callback returns ONLY {ok: true} -- server.js:1001
+        # calls server.sendMonitorList(socket) and then acks with a bare ok. The
+        # actual list arrives as a SEPARATE pushed "monitorList" event. Reading
+        # the callback therefore yields an empty board that looks authoritative,
+        # and an idempotent seed built on it duplicates every row instead of
+        # updating it. Capture the push; never trust the ack for data.
+        self._monitor_list: dict | None = None
+
+        @self.sio.on("monitorList")
+        def _on_monitor_list(data):
+            self._monitor_list = data or {}
+
+    def __enter__(self):
+        self.sio.connect(self.url, transports=["websocket"], wait_timeout=20)
+        self._connected = True
+        # 2.x emits a burst of state on connect; give it a beat to settle so a
+        # call() issued immediately does not race the handshake.
+        time.sleep(1.0)
+        return self
+
+    def __exit__(self, *exc):
+        if self._connected:
+            try:
+                self.sio.disconnect()
+            except Exception:
+                pass
+
+    def call(self, event: str, *args, timeout: int = 30):
+        """Emit with an ack. Every Kuma event answers {ok, msg, ...}; a falsey
+        `ok` is raised rather than returned, so no caller can ignore it."""
+        res = self.sio.call(event, args if len(args) != 1 else args[0], timeout=timeout)
+        if isinstance(res, dict) and not res.get("ok", True):
+            raise RuntimeError(f"{event}: {res.get('msg', res)}")
+        return res
+
+    def need_setup(self) -> bool:
+        return bool(self.sio.call("needSetup", timeout=20))
+
+    def setup(self, username: str, password: str):
+        return self.call("setup", username, password)
+
+    def login(self, username: str, password: str):
+        return self.call("login", {"username": username, "password": password, "token": ""})
+
+    def monitors(self, timeout: int = 20) -> dict:
+        """Every monitor, keyed by id. Waits for the PUSHED `monitorList` event
+        rather than reading the ack -- see the note in __init__."""
+        self._monitor_list = None
+        self.sio.emit("getMonitorList")
+        deadline = time.time() + timeout
+        while self._monitor_list is None and time.time() < deadline:
+            self.sio.sleep(0.2)
+        if self._monitor_list is None:
+            raise RuntimeError("no monitorList event within %ss -- not logged in?" % timeout)
+        return self._monitor_list
+
+    def delete(self, monitor_id: int):
+        return self.call("deleteMonitor", monitor_id, False)
+
+    def add(self, monitor: dict):
+        return self.call("add", {**MONITOR_DEFAULTS, **monitor})
+
+    def edit(self, monitor: dict):
+        return self.call("editMonitor", monitor)
+
+
+def connected_and_logged_in() -> Kuma:
+    creds = vault_get(VAULT_KEY)
+    k = Kuma().__enter__()
+    k.login(creds["username"], creds["password"])
+    return k
+
+
+def cmd_setup(args):
+    creds = vault_get(VAULT_KEY)
+    with Kuma() as k:
+        if not k.need_setup():
+            print("already initialised — nothing to do")
+            return
+        k.setup(creds["username"], creds["password"])
+        print(f"admin '{creds['username']}' created")
+
+
+def cmd_list(args):
+    k = connected_and_logged_in()
+    try:
+        mons = k.monitors()
+        if not mons:
+            print("no monitors")
+            return
+        print(f"{'ID':>4}  {'NAME':<28} {'TYPE':<6} {'ACT':<4} TARGET")
+        for m in sorted(mons.values(), key=lambda x: (x.get("name") or "")):
+            tgt = m.get("url") or f"{m.get('hostname','')}:{m.get('port','')}"
+            print(f"{m.get('id'):>4}  {(m.get('name') or '')[:28]:<28} "
+                  f"{(m.get('type') or '')[:6]:<6} {str(m.get('active')):<4} {tgt[:52]}")
+        print(f"\n{len(mons)} monitor(s)")
+    finally:
+        k.__exit__()
+
+
+def cmd_seed(args):
+    """Idempotent: a monitor whose NAME already exists is edited, not duplicated.
+    Name is the key because a URL legitimately repeats (root and /healthz on the
+    same service), and re-running a seed must never fork the board."""
+    spec = yaml.safe_load(open(args.file))
+    wanted = spec["monitors"] if isinstance(spec, dict) else spec
+    k = connected_and_logged_in()
+    try:
+        existing = {m.get("name"): m for m in k.monitors().values()}
+        added = updated = 0
+        for m in wanted:
+            if m["name"] in existing:
+                merged = {**existing[m["name"]], **m}
+                k.edit(merged)
+                updated += 1
+                print(f"  ~ {m['name']}")
+            else:
+                k.add(m)
+                added += 1
+                print(f"  + {m['name']}")
+        print(f"\n{added} added, {updated} updated, {len(wanted)} in spec")
+    finally:
+        k.__exit__()
+
+
+def cmd_dedupe(args):
+    """Keep the LOWEST id per name, delete the rest. Needed because the first
+    version of this client read the getMonitorList ack instead of the pushed
+    event, saw an empty board, and re-added all 13 rows on a second seed."""
+    k = connected_and_logged_in()
+    try:
+        by_name: dict[str, list] = {}
+        for m in k.monitors().values():
+            by_name.setdefault(m.get("name"), []).append(m)
+        removed = 0
+        for name, group in sorted(by_name.items()):
+            if len(group) < 2:
+                continue
+            group.sort(key=lambda m: m.get("id"))
+            for dup in group[1:]:
+                k.delete(dup["id"])
+                removed += 1
+                print("  - %s (id %s)" % (name, dup["id"]))
+        print("\n%d duplicate(s) removed" % removed)
+    finally:
+        k.__exit__()
+
+
+def cmd_add(args):
+    k = connected_and_logged_in()
+    try:
+        mon = {"name": args.name, "type": args.type}
+        if args.type == "http":
+            mon["url"] = args.target
+        else:
+            mon["hostname"] = args.target
+            if args.port:
+                mon["port"] = args.port
+        k.add(mon)
+        print(f"added {args.name}")
+    finally:
+        k.__exit__()
+
+
+def main():
+    p = argparse.ArgumentParser(prog="kuma", description=__doc__.split("\n")[0])
+    sub = p.add_subparsers(dest="cmd", required=True)
+    sub.add_parser("setup").set_defaults(fn=cmd_setup)
+    sub.add_parser("list").set_defaults(fn=cmd_list)
+    s = sub.add_parser("seed"); s.add_argument("file"); s.set_defaults(fn=cmd_seed)
+    a = sub.add_parser("add")
+    a.add_argument("name"); a.add_argument("target")
+    a.add_argument("--type", default="http", choices=["http", "ping", "port"])
+    a.add_argument("--port", type=int)
+    a.set_defaults(fn=cmd_add)
+    sub.add_parser("dedupe").set_defaults(fn=cmd_dedupe)
+    args = p.parse_args()
+    try:
+        args.fn(args)
+    except Exception as exc:
+        sys.exit(f"error: {exc}")
+
+
+if __name__ == "__main__":
+    main()

stacks/uptimekuma/README.md

@@ -1,54 +1,87 @@
-# uptimekuma — service uptime monitor (esh-docker-vm)
+# uptimekuma — the fleet's SERVICE-layer monitor (ana-docker)
 
-`louislam/uptime-kuma:latest` on **esh-docker-vm** (`10.0.50.45:3001`). Widget
-slug `nethealth` feeds the Uptime Kuma card on the fleet dashboard.
+`louislam/uptime-kuma:2.5.5` on **ana-docker** (`10.250.50.70:3001`), beside the
+Beszel and Dozzle hubs.
 
-## Why it is in this repo
+## Why it exists — the lane split
 
-Adopted 2026-08-18 while cleaning up Homepage. It had run unmanaged on the
-host for ~12 months, and its `homepage.group=Apps` label put it in the wrong
-dashboard group — where it collided with a *manual* entry for the same
-service in the homepage stack's `services.yaml` under Monitoring. Result: two
-Uptime Kuma cards on the Main tab, one with the widget and one without.
+Measured 2026-09-21, after the fleet dashboard sat dead for three days while
+every monitoring tool reported correctly:
 
-That is precisely the "never list a labelled container manually" failure the
-homepage README warns about, and it survived the 2026-08-17 audit because a
-duplicate looks like two plausible cards rather than like an error. The fix
-was to keep the label as the single source of truth, move it to `Monitoring`,
-and delete the manual block.
+| tool | what it asks | scope |
+|---|---|---|
+| **Beszel** | is the box alive, and is it out of CPU / memory / disk / heat | 18 hosts × {Status, CPU, Memory, Disk, Temperature} |
+| **Uptime Kuma** (this) | **is the service actually serving** | the fleet's user-facing endpoints |
+| **Homepage** | — | **DISPLAY ONLY.** Polls 38 URLs, alerts nobody. |
 
-## Deviations from house convention (deliberate, not oversights)
+Beszel and Kuma are **not redundant** — they are disjoint, and the seam between
+them is where things die quietly. Beszel's alerts bind to a *system* with a
+threshold (`alerts` table: `system, name, value, min`); there is **no URL
+column**, so it is structurally incapable of "this endpoint should return 200".
+That is not a configuration gap, it is the data model.
 
-The compose was adopted as-run rather than normalised — this was a dashboard
-fix, not a rewrite of a service that has been stable for a year:
+On 2026-09-18 `homepage` wedged in an unkillable D-state. Beszel reported
+`esh-vm-docker: up` — correctly; the host *was* up. Kuma was not watching it.
+The dead dashboard fell between two working instruments and stayed dark for
+three days until the operator hit a 404.
 
-- `restart: always`, not `unless-stopped`. It revives on Docker daemon start
-  even if it was stopped on purpose.
-- No `healthcheck:` block. The image ships its own, which is why `docker ps`
-  reports `healthy` anyway.
+## Rebuilt from scratch, 2026-09-21
 
-Worth normalising on the next deliberate touch, not worth a restart today.
+Operator-authorised: *"uptime-kuma was never really used… you can even dump the
+existing container and config and start over from scratch."* The prior instance
+had four monitors — three firewall pings and one HTTPS check — and nothing was
+migrated. This also skipped the one-way v1→v2 database migration entirely.
 
-## The container name
+Two things changed with the rebuild:
 
-Live container was `45d2522a8cb6_uptime-kuma` — Docker's collision rename from
-some past recreate where the old container could not be removed. Recreating
-under this compose restores the plain `uptime-kuma` name. Harmless either way.
+**Pinned to `2.5.5`, and `:latest` is now a documented trap.** Upstream keeps
+`latest` on the **1.x** line: an August 2026 pull produced an image *built
+2024-12-20* running 1.23.16. Verified by digest — `latest` and `1` resolve to
+the same image, while `2`/`next` carry 2.5.5. 2.x has been stable since 2.2.0
+(2026-03-05), twelve releases, zero prereleases. Pinned exactly rather than
+floating on `2`, for the same reason `latest` burned us.
+
+**Moved esh-docker-vm → ana-docker.** House placement rule (CLAUDE.md): *cross-
+site services live on ana-docker and pull from agents on the other hosts* — a
+fleet-wide service monitor is exactly that. And `esh-docker-vm` has wedged
+unkillably twice in four months (2026-06-03, 2026-09-18), so it is the worst
+box in the fleet to host the thing that would tell us. A monitor also cannot
+report the failure of the host it runs on; Beszel covers that layer, and the
+two should not share a failure domain.
+
+Normalised at the same time: `restart: always` → `unless-stopped`, which the
+2026-08-18 README flagged as *"worth normalising on the next deliberate touch"*.
+`always` revives a container that was stopped on purpose.
+
+## Scripting it
+
+**There is no REST CRUD API — in either major version.** Checked against the
+2.5.5 source tree, not the docs: `server/routers/` contains exactly two files,
+`api-router.js` (Prometheus `/metrics`, badges, entry page) and
+`status-page-router.js`. API keys unlock `/metrics` and badges only.
+
+Automation goes over **Socket.IO**, the same channel the web UI uses;
+`server/socket-handlers/general-socket-handler.js` carries `add`,
+`editMonitor`, `deleteMonitor`, `getMonitorList`.
+
+⚠️ **Do not reach for `uptime-kuma-api`** (the Python wrapper). It is abandoned
+— last release 2023-09-26 — and its stated ceiling is *"support for uptime kuma
+1.23.0 and 1.23.1"*. It does not support 2.x and never will. Use a thin
+first-party Socket.IO client instead.
+
+## The Homepage widget is deliberately absent
+
+The compose carries **no `homepage.widget.*` labels**. The widget needs a
+published status-page slug, and a from-scratch install has none — it would poll
+a 404 forever. That matters more here than elsewhere: homepage widgets pointed
+at dead targets are the suspected mechanism behind *both* of that dashboard's
+unkillable wedges (see `incident_esh_docker_nfs_boot_race`, 2026-06-03 entry).
+
+Re-add `homepage.widget.type` / `.url` / `.slug` only once the status page
+actually exists. Label changes need `docker compose up -d`, not `restart`.
 
 ## Data
 
-All monitor definitions and history live in the named volume
-`uptimekuma_uptime-kuma`. **Recreating the container is safe. Deleting that
-volume loses every monitor and all history** — there is no config file to
-restore from, the state is entirely in the volume's SQLite DB.
-
-## Deploy
-
-```bash
-scripts/deploy-stack.sh esh-docker-vm uptimekuma --compose
-ssh lkraven@10.0.50.45 'cd /opt/docker/compose/uptimekuma && docker compose up -d'
-```
-
-Label changes need a **recreate**, not a restart — Docker only applies labels
-when the container is created. After recreating, the Homepage container also
-needs a recreate before it re-reads the discovered set.
+Named volume `uptimekuma_uptime-kuma` → `/app/data` holds every monitor
+definition and all history. Recreating the container is safe; deleting that
+volume is not.

stacks/uptimekuma/compose.yaml

@@ -1,49 +1,78 @@
-# uptime-kuma — service uptime monitor on esh-docker-vm.
+# uptime-kuma — the fleet's SERVICE-layer monitor.
 #
-# Adopted into this repo 2026-08-18. It had been running unmanaged on the
-# host since 2025 and was only noticed because its homepage label put it in
-# the wrong group: it was ALSO listed manually in the homepage stack's
-# services.yaml under Monitoring, so the dashboard rendered two Uptime Kuma
-# cards — the labelled one in Apps (with the widget) and the manual one in
-# Monitoring (without). The label is now the single source of truth and the
-# manual entry is gone.
+# REBUILT FROM SCRATCH 2026-09-21 (operator-authorised: "uptime-kuma was never
+# really used... you can even dump the existing container and config and start
+# over from scratch"). The previous instance carried four monitors, all of them
+# firewall pings, and 21-month-old code. Nothing was migrated.
 #
-# ⚠️ The compose is preserved as it ran, not normalised. Two deliberate
-#   deviations from the house conventions, left alone because this commit is
-#   a homepage fix and not a rewrite of a service that has been up for a
-#   year:
-#     - `restart: always` rather than `unless-stopped` (always revives on
-#       daemon start even after a deliberate manual stop)
-#     - no healthcheck block (the image ships its own, which is why
-#       `docker ps` reports healthy)
+# ⚠️ IMAGE TAG — DO NOT USE `:latest` HERE. Upstream keeps `latest` pointing at
+#   the 1.x line, so a pull in August 2026 handed us an image BUILT 2024-12-20
+#   (1.23.16). Measured, not assumed: `latest` and `1` resolve to the same
+#   digest, while `2`/`next` carry 2.5.5. All development is on 2.x, which has
+#   been stable since 2.2.0 (2026-03-05) across twelve releases with zero
+#   prereleases. Pinned EXACTLY rather than floating on `2` for the same reason
+#   `latest` burned us: a floating tag is a version you did not choose.
 #
-# The data volume `uptimekuma_uptime-kuma` holds every monitor definition
-# and all history. Recreating the container is safe; deleting that volume
-# is not.
+# MOVED esh-docker-vm -> ana-docker in the same rebuild, for two reasons:
+#   1. House placement rule (CLAUDE.md): "Cross-site services (e.g. Beszel hub,
+#      Dozzle hub) live on ana-docker and pull from agents on the other hosts."
+#      A fleet-wide service monitor is exactly that, and it now sits beside the
+#      Beszel hub it complements.
+#   2. esh-docker-vm has wedged unkillably TWICE in four months (2026-06-03 and
+#      2026-09-18, both homepage in D-state). The fleet's service monitor should
+#      not live on the least reliable box in the fleet — and a monitor cannot
+#      report the failure of the host it runs on. Beszel covers the host layer;
+#      this covers the service layer; they should not share a failure domain.
+#
+# THE LANE SPLIT this service exists to fill (measured 2026-09-21):
+#   Beszel  -> 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts
+#              bind to a SYSTEM with a threshold; there is no URL column, so it
+#              is structurally incapable of "this endpoint should return 200".
+#   Kuma    -> is the service actually serving. This file.
+#   Homepage-> DISPLAY ONLY. It polls 38 URLs and alerts nobody; it is a
+#              dashboard, never a monitor.
+# The gap between the first two is where homepage sat dead for three days while
+# both tools reported correctly.
 
 name: uptimekuma
 
 services:
   uptime-kuma:
-    image: louislam/uptime-kuma:latest
+    image: louislam/uptime-kuma:2.5.5
     container_name: uptime-kuma
-    restart: always
+    # Normalised from the adopted-as-run `always` — the 2026-08-18 README said
+    # this was "worth normalising on the next deliberate touch", and a rebuild
+    # from scratch is that touch. `always` revives a container that was stopped
+    # ON PURPOSE, which is the wrong behaviour for a service we may deliberately
+    # take down during maintenance.
+    restart: unless-stopped
+    environment:
+      # Skips 2.x's interactive "choose a database" first-run screen.
+      # setup-database.js: UPTIME_KUMA_DB_TYPE overrides db-config.json and
+      # writes it, so the stack comes up ready for the admin-account step
+      # instead of parking on a form. SQLite is right here -- one operator,
+      # a few dozen monitors; MariaDB is for the multi-thousand-check case.
+      UPTIME_KUMA_DB_TYPE: sqlite
     ports:
       - "3001:3001"
     volumes:
       - uptime-kuma:/app/data
     labels:
-      # Monitoring, NOT Apps — this is the group the dashboard's own layout
-      # reserves for Backrest / Beszel / Dozzle / this. See the note above.
+      # Monitoring, NOT Apps — this is the group the dashboard's layout reserves
+      # for Backrest / Beszel / Dozzle / this.
       - homepage.group=Monitoring
       - homepage.name=Uptime Kuma
       - homepage.icon=mdi-arrow-up-bold-circle
-      - homepage.description=Service Monitoring (esh)
-      - homepage.href=http://10.0.50.45:3001
-      - homepage.siteMonitor=http://10.0.50.45:3001
-      - homepage.widget.type=uptimekuma
-      - homepage.widget.url=http://10.0.50.45:3001
-      - homepage.widget.slug=nethealth
+      - homepage.description=Service monitoring (fleet)
+      - homepage.href=http://10.250.50.70:3001
+      - homepage.siteMonitor=http://10.250.50.70:3001
+      # ⚠️ NO `homepage.widget.*` LABELS YET, deliberately. The widget needs a
+      #   published status-page slug; on a from-scratch install none exists, so
+      #   the widget would poll a 404 forever. That matters more than usual
+      #   here: homepage widgets pointed at dead targets are the suspected
+      #   mechanism behind BOTH of this dashboard's unkillable D-state wedges
+      #   (see incident_esh_docker_nfs_boot_race, 2026-06-03). Re-add the widget
+      #   labels only once the status page actually exists.
     networks:
       - tnet
 

stacks/uptimekuma/monitors.yaml

@@ -0,0 +1,78 @@
+# Uptime Kuma — the fleet's service-layer monitors, as code.
+#
+#   scripts/kuma seed stacks/uptimekuma/monitors.yaml
+#
+# Idempotent: keyed on NAME, so re-running edits rather than duplicating. Name
+# and not URL, because a URL legitimately repeats (a root and a /healthz on one
+# service) and a re-run must never fork the board.
+#
+# WHAT BELONGS HERE — the lane rule, so this file does not sprawl to 115 rows:
+#   IN  — services that should ALWAYS be up, whose silent death costs us work.
+#   OUT — inference seats (they come and go BY DESIGN; a dormant seat is normal
+#         and alerting on it is pure noise), hosts and hypervisors and firewalls
+#         (Beszel's lane: 18 hosts x Status/CPU/Memory/Disk/Temp), and Uptime
+#         Kuma itself (it cannot report its own death — that is Beszel's job).
+#
+# Every URL below was probed before being written here: all returned 200 on
+# 2026-09-21. A seed that ships red on day one teaches everyone to ignore the
+# board, which is how you end up with a monitor nobody reads.
+#
+# NOT SEEDED, deliberately: "althing chamber" (10.250.50.70:7881) is on the
+# Homepage dashboard but refuses connections right now. It is either genuinely
+# down or retired; seeding it would put a red row on a brand-new board before
+# anyone has decided which. Resolve it, then add it.
+
+monitors:
+  # ---- fleet toolchain: dead = agents and the operator are blocked ----
+  - name: althing post office
+    url: http://10.100.50.40:8390/
+    description: the bus every agent session reads mail from
+
+  - name: The Booth
+    url: http://10.100.10.50:8090/healthz
+    description: operator-review surface; real healthz, not a root page
+
+  - name: task-board
+    url: http://10.250.50.70:7878
+
+  - name: vor
+    url: http://10.250.50.70:7879
+
+  # ---- the dashboard that started all this ----
+  # Dead for three days in Sept 2026 while Beszel correctly reported its host
+  # UP. This row is the entire reason the service layer exists.
+  - name: Homepage
+    url: http://10.0.50.45:5100/
+    description: fleet dashboard (esh-docker-vm) — the 2026-09-18 silent death
+
+  # ---- credentials + code: dead = nothing ships ----
+  - name: Gitea
+    url: https://gitea.phasefinal.com
+
+  - name: Vaultwarden
+    url: https://vaultwarden.phasefinal.com
+    description: the vault every agent reads credentials from
+
+  # ---- AI control plane (the gateways, NOT the seats behind them) ----
+  - name: LiteLLM Gateway
+    url: http://10.250.50.70:4000/health/liveliness
+    description: liveliness endpoint, so a model outage does not read as a gateway outage
+
+  - name: Asset Engine
+    url: http://10.250.50.70:8200/api/v1/services
+
+  - name: talk
+    url: https://talk.nh3.phasefinal.com:8092/
+    description: fleet voice bench
+
+  # ---- monitoring + backup: a blind monitor is worse than none ----
+  - name: Beszel hub
+    url: http://10.250.50.70:8090
+    description: the host layer; if this is down we are blind to 18 hosts
+
+  - name: Backrest
+    url: http://10.250.50.70:9898
+    description: restic orchestration — a silent backup failure is the expensive kind
+
+  - name: Dozzle hub
+    url: http://10.250.50.70:8088