From 3a85a6bce1108ea39973eeb903d1c973c7152296 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 21 Sep 2026 22:59:37 -0700 Subject: [PATCH] feat(uptimekuma): rebuild on 2.5.5 as the fleet's service layer, with a client MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Homepage sat dead for three days in September while every monitoring tool reported correctly. Beszel said its host was up -- it was. Uptime Kuma was not watching it. The dashboard fell into the seam between two working instruments. Measured before changing anything: - Beszel: 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts table is (system, name, value, min) -- there is NO url column, so it is structurally incapable of "this endpoint should return 200". Not a config gap; the data model. - Uptime Kuma: 6 rows, 2 of them folders. Four real monitors, all firewalls. - So the two are NOT redundant. They are disjoint, and the service layer between them was empty. REBUILT FROM SCRATCH, operator-authorised ("uptime-kuma was never really used... you can even dump the existing container and config"). Nothing was migrated, which also skipped the one-way v1->v2 database migration. - Pinned to 2.5.5. `:latest` is a documented trap now: upstream keeps it on the 1.x line, so an August 2026 pull produced an image BUILT 2024-12-20 running 1.23.16. Verified by digest -- latest and 1 share one digest while 2/next carry 2.5.5. Pinned exactly, not floating on 2, for the same reason. - Moved esh-docker-vm -> ana-docker. House placement rule puts cross-site services beside the Beszel and Dozzle hubs, and esh-docker-vm has wedged unkillably twice in four months. A monitor also cannot report the failure of the host it runs on, so it should not share a failure domain with the host layer. - Normalised restart: always -> unless-stopped, which the 2026-08-18 README flagged as worth doing on the next deliberate touch. - UPTIME_KUMA_DB_TYPE=sqlite in the compose skips 2.x's interactive database screen, so the stack comes up ready rather than parked on a form. scripts/kuma is a first-party Socket.IO client, because both obvious paths are wrong: there is no REST CRUD API in EITHER major version (server/routers/ holds exactly two files, /metrics + badges + status pages), and the community wrapper uptime-kuma-api is abandoned -- last release 2023-09-26, ceiling 1.23.1, no 2.x support ever. ⚠ getMonitorList's callback returns only {ok:true}; the list arrives as a SEPARATE pushed monitorList event. Reading the ack yields an empty board that looks authoritative -- which duplicated all 13 rows on the first re-seed before the bug was found. The client now waits for the push, and carries a dedupe verb because of it. 13 monitors seeded from monitors.yaml, keyed on name so a re-run updates rather than forking the board -- proven by re-running it (0 added, 13 updated), not assumed. Every URL was probed before being written: all 200. A board that ships red teaches everyone to ignore it. Verified: 13 rows, no duplicates, all UP with "200 - OK" read from the database WITH its WAL (a first read of kuma.db alone showed a stale 26 -- the copy predated the deletes). Homepage renders exactly one Uptime Kuma card. ⚠ NOT YET WIRED: notification delivery. The board detects but tells nobody, which is the same gap this work exists to close. The beszel-althing bridge hardcodes a [Beszel] subject prefix and hub footer, so routing Kuma through it unchanged would mislabel the alerts. Needs a decision before it is generalised. --- scripts/kuma | 285 ++++++++++++++++++++++++++++++++ stacks/uptimekuma/README.md | 115 ++++++++----- stacks/uptimekuma/compose.yaml | 87 ++++++---- stacks/uptimekuma/monitors.yaml | 78 +++++++++ 4 files changed, 495 insertions(+), 70 deletions(-) create mode 100755 scripts/kuma create mode 100644 stacks/uptimekuma/monitors.yaml diff --git a/scripts/kuma b/scripts/kuma new file mode 100755 index 0000000..d99c7a0 --- /dev/null +++ b/scripts/kuma @@ -0,0 +1,285 @@ +#!/usr/bin/env -S uv run --quiet --script +# /// script +# requires-python = ">=3.11" +# dependencies = ["python-socketio[client]>=5.11", "requests>=2.31", "PyYAML>=6"] +# /// +"""kuma — a first-party Socket.IO client for Uptime Kuma 2.x. + +WHY THIS EXISTS, since the obvious answers are both wrong: + + * There is NO REST CRUD API for monitors, in either major version. Checked + against the 2.5.5 source tree, not the docs: `server/routers/` holds exactly + two files -- api-router.js (Prometheus /metrics, badges, entry page) and + status-page-router.js. API keys unlock /metrics and badges only. Every + monitor operation lives on Socket.IO, the same channel the web UI uses. + + * The community wrapper `uptime-kuma-api` is ABANDONED -- last release + 2023-09-26 -- and its stated ceiling is "support for uptime kuma 1.23.0 and + 1.23.1". It does not speak 2.x and never will. Depending on it would trade a + maintained server for an unmaintained client. + +So: a thin client of our own, against events read out of server.js at tag 2.5.5 +(`setup`, `login`, `getMonitorList`, `add`, `editMonitor`, `deleteMonitor`). +Small on purpose -- this is four events and a socket, not a framework. + +USAGE + scripts/kuma setup # create the admin (first run only) + scripts/kuma list # every monitor, with its state + scripts/kuma seed # idempotent upsert from a file + scripts/kuma add [--type http|ping|port] [--port N] + +CREDENTIALS come from the vault, never from a flag: + secret get ana-docker/.config/uptime-kuma/admin +Password on the command line lands in shell history and in the process table. +""" + +from __future__ import annotations + +import argparse +import json +import subprocess +import sys +import time + +import socketio +import yaml + +URL = "http://10.250.50.70:3001" +VAULT_KEY = "ana-docker/.config/uptime-kuma/admin" + +# Defaults the 2.x `add` handler REQUIRES rather than defaults itself. It +# JSON.stringify()s kafkaProducerBrokers, kafkaProducerSaslOptions, conditions +# and rabbitmqNodes unconditionally, so they must be present even for an HTTP +# check -- omit them and you get "undefined" written into the row. +MONITOR_DEFAULTS = { + "type": "http", + "method": "GET", + "interval": 60, + "retryInterval": 60, + "resendInterval": 0, + "maxretries": 2, + "timeout": 16, + "maxredirects": 10, + "accepted_statuscodes": ["200-299"], + "notificationIDList": {}, + "active": True, + "expiryNotification": False, + "ignoreTls": True, # the fleet is full of self-signed internal certs + "upsideDown": False, + "kafkaProducerBrokers": [], + "kafkaProducerSaslOptions": {}, + "conditions": [], + "rabbitmqNodes": [], +} + + +def vault_get(key: str) -> dict: + """{'username','password'} from the vault. Loud on failure -- a silent + empty credential would show up as a confusing auth error much later.""" + out = subprocess.run( + ["secret", "get", key], capture_output=True, text=True, timeout=60 + ) + if out.returncode != 0: + sys.exit(f"vault: could not read {key}: {out.stderr.strip() or out.stdout.strip()}") + body = out.stdout.strip() + try: + return json.loads(body) + except json.JSONDecodeError: + # tolerate a "user:pass" single line as well + if ":" in body: + u, _, p = body.partition(":") + return {"username": u.strip(), "password": p.strip()} + sys.exit(f"vault: {key} is neither JSON nor user:pass") + + +class Kuma: + def __init__(self, url: str = URL): + self.url = url + self.sio = socketio.Client(reconnection=False) + self._connected = False + # ⚠ THE TRAP THAT COST A DUPLICATED BOARD (2026-09-21). + # `getMonitorList`'s callback returns ONLY {ok: true} -- server.js:1001 + # calls server.sendMonitorList(socket) and then acks with a bare ok. The + # actual list arrives as a SEPARATE pushed "monitorList" event. Reading + # the callback therefore yields an empty board that looks authoritative, + # and an idempotent seed built on it duplicates every row instead of + # updating it. Capture the push; never trust the ack for data. + self._monitor_list: dict | None = None + + @self.sio.on("monitorList") + def _on_monitor_list(data): + self._monitor_list = data or {} + + def __enter__(self): + self.sio.connect(self.url, transports=["websocket"], wait_timeout=20) + self._connected = True + # 2.x emits a burst of state on connect; give it a beat to settle so a + # call() issued immediately does not race the handshake. + time.sleep(1.0) + return self + + def __exit__(self, *exc): + if self._connected: + try: + self.sio.disconnect() + except Exception: + pass + + def call(self, event: str, *args, timeout: int = 30): + """Emit with an ack. Every Kuma event answers {ok, msg, ...}; a falsey + `ok` is raised rather than returned, so no caller can ignore it.""" + res = self.sio.call(event, args if len(args) != 1 else args[0], timeout=timeout) + if isinstance(res, dict) and not res.get("ok", True): + raise RuntimeError(f"{event}: {res.get('msg', res)}") + return res + + def need_setup(self) -> bool: + return bool(self.sio.call("needSetup", timeout=20)) + + def setup(self, username: str, password: str): + return self.call("setup", username, password) + + def login(self, username: str, password: str): + return self.call("login", {"username": username, "password": password, "token": ""}) + + def monitors(self, timeout: int = 20) -> dict: + """Every monitor, keyed by id. Waits for the PUSHED `monitorList` event + rather than reading the ack -- see the note in __init__.""" + self._monitor_list = None + self.sio.emit("getMonitorList") + deadline = time.time() + timeout + while self._monitor_list is None and time.time() < deadline: + self.sio.sleep(0.2) + if self._monitor_list is None: + raise RuntimeError("no monitorList event within %ss -- not logged in?" % timeout) + return self._monitor_list + + def delete(self, monitor_id: int): + return self.call("deleteMonitor", monitor_id, False) + + def add(self, monitor: dict): + return self.call("add", {**MONITOR_DEFAULTS, **monitor}) + + def edit(self, monitor: dict): + return self.call("editMonitor", monitor) + + +def connected_and_logged_in() -> Kuma: + creds = vault_get(VAULT_KEY) + k = Kuma().__enter__() + k.login(creds["username"], creds["password"]) + return k + + +def cmd_setup(args): + creds = vault_get(VAULT_KEY) + with Kuma() as k: + if not k.need_setup(): + print("already initialised — nothing to do") + return + k.setup(creds["username"], creds["password"]) + print(f"admin '{creds['username']}' created") + + +def cmd_list(args): + k = connected_and_logged_in() + try: + mons = k.monitors() + if not mons: + print("no monitors") + return + print(f"{'ID':>4} {'NAME':<28} {'TYPE':<6} {'ACT':<4} TARGET") + for m in sorted(mons.values(), key=lambda x: (x.get("name") or "")): + tgt = m.get("url") or f"{m.get('hostname','')}:{m.get('port','')}" + print(f"{m.get('id'):>4} {(m.get('name') or '')[:28]:<28} " + f"{(m.get('type') or '')[:6]:<6} {str(m.get('active')):<4} {tgt[:52]}") + print(f"\n{len(mons)} monitor(s)") + finally: + k.__exit__() + + +def cmd_seed(args): + """Idempotent: a monitor whose NAME already exists is edited, not duplicated. + Name is the key because a URL legitimately repeats (root and /healthz on the + same service), and re-running a seed must never fork the board.""" + spec = yaml.safe_load(open(args.file)) + wanted = spec["monitors"] if isinstance(spec, dict) else spec + k = connected_and_logged_in() + try: + existing = {m.get("name"): m for m in k.monitors().values()} + added = updated = 0 + for m in wanted: + if m["name"] in existing: + merged = {**existing[m["name"]], **m} + k.edit(merged) + updated += 1 + print(f" ~ {m['name']}") + else: + k.add(m) + added += 1 + print(f" + {m['name']}") + print(f"\n{added} added, {updated} updated, {len(wanted)} in spec") + finally: + k.__exit__() + + +def cmd_dedupe(args): + """Keep the LOWEST id per name, delete the rest. Needed because the first + version of this client read the getMonitorList ack instead of the pushed + event, saw an empty board, and re-added all 13 rows on a second seed.""" + k = connected_and_logged_in() + try: + by_name: dict[str, list] = {} + for m in k.monitors().values(): + by_name.setdefault(m.get("name"), []).append(m) + removed = 0 + for name, group in sorted(by_name.items()): + if len(group) < 2: + continue + group.sort(key=lambda m: m.get("id")) + for dup in group[1:]: + k.delete(dup["id"]) + removed += 1 + print(" - %s (id %s)" % (name, dup["id"])) + print("\n%d duplicate(s) removed" % removed) + finally: + k.__exit__() + + +def cmd_add(args): + k = connected_and_logged_in() + try: + mon = {"name": args.name, "type": args.type} + if args.type == "http": + mon["url"] = args.target + else: + mon["hostname"] = args.target + if args.port: + mon["port"] = args.port + k.add(mon) + print(f"added {args.name}") + finally: + k.__exit__() + + +def main(): + p = argparse.ArgumentParser(prog="kuma", description=__doc__.split("\n")[0]) + sub = p.add_subparsers(dest="cmd", required=True) + sub.add_parser("setup").set_defaults(fn=cmd_setup) + sub.add_parser("list").set_defaults(fn=cmd_list) + s = sub.add_parser("seed"); s.add_argument("file"); s.set_defaults(fn=cmd_seed) + a = sub.add_parser("add") + a.add_argument("name"); a.add_argument("target") + a.add_argument("--type", default="http", choices=["http", "ping", "port"]) + a.add_argument("--port", type=int) + a.set_defaults(fn=cmd_add) + sub.add_parser("dedupe").set_defaults(fn=cmd_dedupe) + args = p.parse_args() + try: + args.fn(args) + except Exception as exc: + sys.exit(f"error: {exc}") + + +if __name__ == "__main__": + main() diff --git a/stacks/uptimekuma/README.md b/stacks/uptimekuma/README.md index 6e000f4..d939e2e 100644 --- a/stacks/uptimekuma/README.md +++ b/stacks/uptimekuma/README.md @@ -1,54 +1,87 @@ -# uptimekuma — service uptime monitor (esh-docker-vm) +# uptimekuma — the fleet's SERVICE-layer monitor (ana-docker) -`louislam/uptime-kuma:latest` on **esh-docker-vm** (`10.0.50.45:3001`). Widget -slug `nethealth` feeds the Uptime Kuma card on the fleet dashboard. +`louislam/uptime-kuma:2.5.5` on **ana-docker** (`10.250.50.70:3001`), beside the +Beszel and Dozzle hubs. -## Why it is in this repo +## Why it exists — the lane split -Adopted 2026-08-18 while cleaning up Homepage. It had run unmanaged on the -host for ~12 months, and its `homepage.group=Apps` label put it in the wrong -dashboard group — where it collided with a *manual* entry for the same -service in the homepage stack's `services.yaml` under Monitoring. Result: two -Uptime Kuma cards on the Main tab, one with the widget and one without. +Measured 2026-09-21, after the fleet dashboard sat dead for three days while +every monitoring tool reported correctly: -That is precisely the "never list a labelled container manually" failure the -homepage README warns about, and it survived the 2026-08-17 audit because a -duplicate looks like two plausible cards rather than like an error. The fix -was to keep the label as the single source of truth, move it to `Monitoring`, -and delete the manual block. +| tool | what it asks | scope | +|---|---|---| +| **Beszel** | is the box alive, and is it out of CPU / memory / disk / heat | 18 hosts × {Status, CPU, Memory, Disk, Temperature} | +| **Uptime Kuma** (this) | **is the service actually serving** | the fleet's user-facing endpoints | +| **Homepage** | — | **DISPLAY ONLY.** Polls 38 URLs, alerts nobody. | -## Deviations from house convention (deliberate, not oversights) +Beszel and Kuma are **not redundant** — they are disjoint, and the seam between +them is where things die quietly. Beszel's alerts bind to a *system* with a +threshold (`alerts` table: `system, name, value, min`); there is **no URL +column**, so it is structurally incapable of "this endpoint should return 200". +That is not a configuration gap, it is the data model. -The compose was adopted as-run rather than normalised — this was a dashboard -fix, not a rewrite of a service that has been stable for a year: +On 2026-09-18 `homepage` wedged in an unkillable D-state. Beszel reported +`esh-vm-docker: up` — correctly; the host *was* up. Kuma was not watching it. +The dead dashboard fell between two working instruments and stayed dark for +three days until the operator hit a 404. -- `restart: always`, not `unless-stopped`. It revives on Docker daemon start - even if it was stopped on purpose. -- No `healthcheck:` block. The image ships its own, which is why `docker ps` - reports `healthy` anyway. +## Rebuilt from scratch, 2026-09-21 -Worth normalising on the next deliberate touch, not worth a restart today. +Operator-authorised: *"uptime-kuma was never really used… you can even dump the +existing container and config and start over from scratch."* The prior instance +had four monitors — three firewall pings and one HTTPS check — and nothing was +migrated. This also skipped the one-way v1→v2 database migration entirely. -## The container name +Two things changed with the rebuild: -Live container was `45d2522a8cb6_uptime-kuma` — Docker's collision rename from -some past recreate where the old container could not be removed. Recreating -under this compose restores the plain `uptime-kuma` name. Harmless either way. +**Pinned to `2.5.5`, and `:latest` is now a documented trap.** Upstream keeps +`latest` on the **1.x** line: an August 2026 pull produced an image *built +2024-12-20* running 1.23.16. Verified by digest — `latest` and `1` resolve to +the same image, while `2`/`next` carry 2.5.5. 2.x has been stable since 2.2.0 +(2026-03-05), twelve releases, zero prereleases. Pinned exactly rather than +floating on `2`, for the same reason `latest` burned us. + +**Moved esh-docker-vm → ana-docker.** House placement rule (CLAUDE.md): *cross- +site services live on ana-docker and pull from agents on the other hosts* — a +fleet-wide service monitor is exactly that. And `esh-docker-vm` has wedged +unkillably twice in four months (2026-06-03, 2026-09-18), so it is the worst +box in the fleet to host the thing that would tell us. A monitor also cannot +report the failure of the host it runs on; Beszel covers that layer, and the +two should not share a failure domain. + +Normalised at the same time: `restart: always` → `unless-stopped`, which the +2026-08-18 README flagged as *"worth normalising on the next deliberate touch"*. +`always` revives a container that was stopped on purpose. + +## Scripting it + +**There is no REST CRUD API — in either major version.** Checked against the +2.5.5 source tree, not the docs: `server/routers/` contains exactly two files, +`api-router.js` (Prometheus `/metrics`, badges, entry page) and +`status-page-router.js`. API keys unlock `/metrics` and badges only. + +Automation goes over **Socket.IO**, the same channel the web UI uses; +`server/socket-handlers/general-socket-handler.js` carries `add`, +`editMonitor`, `deleteMonitor`, `getMonitorList`. + +⚠️ **Do not reach for `uptime-kuma-api`** (the Python wrapper). It is abandoned +— last release 2023-09-26 — and its stated ceiling is *"support for uptime kuma +1.23.0 and 1.23.1"*. It does not support 2.x and never will. Use a thin +first-party Socket.IO client instead. + +## The Homepage widget is deliberately absent + +The compose carries **no `homepage.widget.*` labels**. The widget needs a +published status-page slug, and a from-scratch install has none — it would poll +a 404 forever. That matters more here than elsewhere: homepage widgets pointed +at dead targets are the suspected mechanism behind *both* of that dashboard's +unkillable wedges (see `incident_esh_docker_nfs_boot_race`, 2026-06-03 entry). + +Re-add `homepage.widget.type` / `.url` / `.slug` only once the status page +actually exists. Label changes need `docker compose up -d`, not `restart`. ## Data -All monitor definitions and history live in the named volume -`uptimekuma_uptime-kuma`. **Recreating the container is safe. Deleting that -volume loses every monitor and all history** — there is no config file to -restore from, the state is entirely in the volume's SQLite DB. - -## Deploy - -```bash -scripts/deploy-stack.sh esh-docker-vm uptimekuma --compose -ssh lkraven@10.0.50.45 'cd /opt/docker/compose/uptimekuma && docker compose up -d' -``` - -Label changes need a **recreate**, not a restart — Docker only applies labels -when the container is created. After recreating, the Homepage container also -needs a recreate before it re-reads the discovered set. +Named volume `uptimekuma_uptime-kuma` → `/app/data` holds every monitor +definition and all history. Recreating the container is safe; deleting that +volume is not. diff --git a/stacks/uptimekuma/compose.yaml b/stacks/uptimekuma/compose.yaml index 5861321..af17af8 100644 --- a/stacks/uptimekuma/compose.yaml +++ b/stacks/uptimekuma/compose.yaml @@ -1,49 +1,78 @@ -# uptime-kuma — service uptime monitor on esh-docker-vm. +# uptime-kuma — the fleet's SERVICE-layer monitor. # -# Adopted into this repo 2026-08-18. It had been running unmanaged on the -# host since 2025 and was only noticed because its homepage label put it in -# the wrong group: it was ALSO listed manually in the homepage stack's -# services.yaml under Monitoring, so the dashboard rendered two Uptime Kuma -# cards — the labelled one in Apps (with the widget) and the manual one in -# Monitoring (without). The label is now the single source of truth and the -# manual entry is gone. +# REBUILT FROM SCRATCH 2026-09-21 (operator-authorised: "uptime-kuma was never +# really used... you can even dump the existing container and config and start +# over from scratch"). The previous instance carried four monitors, all of them +# firewall pings, and 21-month-old code. Nothing was migrated. # -# ⚠️ The compose is preserved as it ran, not normalised. Two deliberate -# deviations from the house conventions, left alone because this commit is -# a homepage fix and not a rewrite of a service that has been up for a -# year: -# - `restart: always` rather than `unless-stopped` (always revives on -# daemon start even after a deliberate manual stop) -# - no healthcheck block (the image ships its own, which is why -# `docker ps` reports healthy) +# ⚠️ IMAGE TAG — DO NOT USE `:latest` HERE. Upstream keeps `latest` pointing at +# the 1.x line, so a pull in August 2026 handed us an image BUILT 2024-12-20 +# (1.23.16). Measured, not assumed: `latest` and `1` resolve to the same +# digest, while `2`/`next` carry 2.5.5. All development is on 2.x, which has +# been stable since 2.2.0 (2026-03-05) across twelve releases with zero +# prereleases. Pinned EXACTLY rather than floating on `2` for the same reason +# `latest` burned us: a floating tag is a version you did not choose. # -# The data volume `uptimekuma_uptime-kuma` holds every monitor definition -# and all history. Recreating the container is safe; deleting that volume -# is not. +# MOVED esh-docker-vm -> ana-docker in the same rebuild, for two reasons: +# 1. House placement rule (CLAUDE.md): "Cross-site services (e.g. Beszel hub, +# Dozzle hub) live on ana-docker and pull from agents on the other hosts." +# A fleet-wide service monitor is exactly that, and it now sits beside the +# Beszel hub it complements. +# 2. esh-docker-vm has wedged unkillably TWICE in four months (2026-06-03 and +# 2026-09-18, both homepage in D-state). The fleet's service monitor should +# not live on the least reliable box in the fleet — and a monitor cannot +# report the failure of the host it runs on. Beszel covers the host layer; +# this covers the service layer; they should not share a failure domain. +# +# THE LANE SPLIT this service exists to fill (measured 2026-09-21): +# Beszel -> 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts +# bind to a SYSTEM with a threshold; there is no URL column, so it +# is structurally incapable of "this endpoint should return 200". +# Kuma -> is the service actually serving. This file. +# Homepage-> DISPLAY ONLY. It polls 38 URLs and alerts nobody; it is a +# dashboard, never a monitor. +# The gap between the first two is where homepage sat dead for three days while +# both tools reported correctly. name: uptimekuma services: uptime-kuma: - image: louislam/uptime-kuma:latest + image: louislam/uptime-kuma:2.5.5 container_name: uptime-kuma - restart: always + # Normalised from the adopted-as-run `always` — the 2026-08-18 README said + # this was "worth normalising on the next deliberate touch", and a rebuild + # from scratch is that touch. `always` revives a container that was stopped + # ON PURPOSE, which is the wrong behaviour for a service we may deliberately + # take down during maintenance. + restart: unless-stopped + environment: + # Skips 2.x's interactive "choose a database" first-run screen. + # setup-database.js: UPTIME_KUMA_DB_TYPE overrides db-config.json and + # writes it, so the stack comes up ready for the admin-account step + # instead of parking on a form. SQLite is right here -- one operator, + # a few dozen monitors; MariaDB is for the multi-thousand-check case. + UPTIME_KUMA_DB_TYPE: sqlite ports: - "3001:3001" volumes: - uptime-kuma:/app/data labels: - # Monitoring, NOT Apps — this is the group the dashboard's own layout - # reserves for Backrest / Beszel / Dozzle / this. See the note above. + # Monitoring, NOT Apps — this is the group the dashboard's layout reserves + # for Backrest / Beszel / Dozzle / this. - homepage.group=Monitoring - homepage.name=Uptime Kuma - homepage.icon=mdi-arrow-up-bold-circle - - homepage.description=Service Monitoring (esh) - - homepage.href=http://10.0.50.45:3001 - - homepage.siteMonitor=http://10.0.50.45:3001 - - homepage.widget.type=uptimekuma - - homepage.widget.url=http://10.0.50.45:3001 - - homepage.widget.slug=nethealth + - homepage.description=Service monitoring (fleet) + - homepage.href=http://10.250.50.70:3001 + - homepage.siteMonitor=http://10.250.50.70:3001 + # ⚠️ NO `homepage.widget.*` LABELS YET, deliberately. The widget needs a + # published status-page slug; on a from-scratch install none exists, so + # the widget would poll a 404 forever. That matters more than usual + # here: homepage widgets pointed at dead targets are the suspected + # mechanism behind BOTH of this dashboard's unkillable D-state wedges + # (see incident_esh_docker_nfs_boot_race, 2026-06-03). Re-add the widget + # labels only once the status page actually exists. networks: - tnet diff --git a/stacks/uptimekuma/monitors.yaml b/stacks/uptimekuma/monitors.yaml new file mode 100644 index 0000000..6d55a88 --- /dev/null +++ b/stacks/uptimekuma/monitors.yaml @@ -0,0 +1,78 @@ +# Uptime Kuma — the fleet's service-layer monitors, as code. +# +# scripts/kuma seed stacks/uptimekuma/monitors.yaml +# +# Idempotent: keyed on NAME, so re-running edits rather than duplicating. Name +# and not URL, because a URL legitimately repeats (a root and a /healthz on one +# service) and a re-run must never fork the board. +# +# WHAT BELONGS HERE — the lane rule, so this file does not sprawl to 115 rows: +# IN — services that should ALWAYS be up, whose silent death costs us work. +# OUT — inference seats (they come and go BY DESIGN; a dormant seat is normal +# and alerting on it is pure noise), hosts and hypervisors and firewalls +# (Beszel's lane: 18 hosts x Status/CPU/Memory/Disk/Temp), and Uptime +# Kuma itself (it cannot report its own death — that is Beszel's job). +# +# Every URL below was probed before being written here: all returned 200 on +# 2026-09-21. A seed that ships red on day one teaches everyone to ignore the +# board, which is how you end up with a monitor nobody reads. +# +# NOT SEEDED, deliberately: "althing chamber" (10.250.50.70:7881) is on the +# Homepage dashboard but refuses connections right now. It is either genuinely +# down or retired; seeding it would put a red row on a brand-new board before +# anyone has decided which. Resolve it, then add it. + +monitors: + # ---- fleet toolchain: dead = agents and the operator are blocked ---- + - name: althing post office + url: http://10.100.50.40:8390/ + description: the bus every agent session reads mail from + + - name: The Booth + url: http://10.100.10.50:8090/healthz + description: operator-review surface; real healthz, not a root page + + - name: task-board + url: http://10.250.50.70:7878 + + - name: vor + url: http://10.250.50.70:7879 + + # ---- the dashboard that started all this ---- + # Dead for three days in Sept 2026 while Beszel correctly reported its host + # UP. This row is the entire reason the service layer exists. + - name: Homepage + url: http://10.0.50.45:5100/ + description: fleet dashboard (esh-docker-vm) — the 2026-09-18 silent death + + # ---- credentials + code: dead = nothing ships ---- + - name: Gitea + url: https://gitea.phasefinal.com + + - name: Vaultwarden + url: https://vaultwarden.phasefinal.com + description: the vault every agent reads credentials from + + # ---- AI control plane (the gateways, NOT the seats behind them) ---- + - name: LiteLLM Gateway + url: http://10.250.50.70:4000/health/liveliness + description: liveliness endpoint, so a model outage does not read as a gateway outage + + - name: Asset Engine + url: http://10.250.50.70:8200/api/v1/services + + - name: talk + url: https://talk.nh3.phasefinal.com:8092/ + description: fleet voice bench + + # ---- monitoring + backup: a blind monitor is worse than none ---- + - name: Beszel hub + url: http://10.250.50.70:8090 + description: the host layer; if this is down we are blind to 18 hosts + + - name: Backrest + url: http://10.250.50.70:9898 + description: restic orchestration — a silent backup failure is the expensive kind + + - name: Dozzle hub + url: http://10.250.50.70:8088