vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(uptimekuma): rebuild on 2.5.5 as the fleet's service layer, with a client

Browse Source 
Homepage sat dead for three days in September while every monitoring tool
reported correctly. Beszel said its host was up -- it was. Uptime Kuma was not
watching it. The dashboard fell into the seam between two working instruments.

Measured before changing anything:
  - Beszel: 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts
    table is (system, name, value, min) -- there is NO url column, so it is
    structurally incapable of "this endpoint should return 200". Not a config
    gap; the data model.
  - Uptime Kuma: 6 rows, 2 of them folders. Four real monitors, all firewalls.
  - So the two are NOT redundant. They are disjoint, and the service layer
    between them was empty.

REBUILT FROM SCRATCH, operator-authorised ("uptime-kuma was never really
used... you can even dump the existing container and config"). Nothing was
migrated, which also skipped the one-way v1->v2 database migration.

  - Pinned to 2.5.5. `:latest` is a documented trap now: upstream keeps it on
    the 1.x line, so an August 2026 pull produced an image BUILT 2024-12-20
    running 1.23.16. Verified by digest -- latest and 1 share one digest while
    2/next carry 2.5.5. Pinned exactly, not floating on 2, for the same reason.
  - Moved esh-docker-vm -> ana-docker. House placement rule puts cross-site
    services beside the Beszel and Dozzle hubs, and esh-docker-vm has wedged
    unkillably twice in four months. A monitor also cannot report the failure
    of the host it runs on, so it should not share a failure domain with the
    host layer.
  - Normalised restart: always -> unless-stopped, which the 2026-08-18 README
    flagged as worth doing on the next deliberate touch.
  - UPTIME_KUMA_DB_TYPE=sqlite in the compose skips 2.x's interactive database
    screen, so the stack comes up ready rather than parked on a form.

scripts/kuma is a first-party Socket.IO client, because both obvious paths are
wrong: there is no REST CRUD API in EITHER major version (server/routers/ holds
exactly two files, /metrics + badges + status pages), and the community wrapper
uptime-kuma-api is abandoned -- last release 2023-09-26, ceiling 1.23.1, no 2.x
support ever.

  ⚠ getMonitorList's callback returns only {ok:true}; the list arrives as a
  SEPARATE pushed monitorList event. Reading the ack yields an empty board that
  looks authoritative -- which duplicated all 13 rows on the first re-seed
  before the bug was found. The client now waits for the push, and carries a
  dedupe verb because of it.

13 monitors seeded from monitors.yaml, keyed on name so a re-run updates rather
than forking the board -- proven by re-running it (0 added, 13 updated), not
assumed. Every URL was probed before being written: all 200. A board that ships
red teaches everyone to ignore it.

Verified: 13 rows, no duplicates, all UP with "200 - OK" read from the database
WITH its WAL (a first read of kuma.db alone showed a stale 26 -- the copy
predated the deletes). Homepage renders exactly one Uptime Kuma card.

⚠ NOT YET WIRED: notification delivery. The board detects but tells nobody,
which is the same gap this work exists to close. The beszel-althing bridge
hardcodes a [Beszel] subject prefix and hub footer, so routing Kuma through it
unchanged would mislabel the alerts. Needs a decision before it is generalised.
This commit is contained in:
vh
2026-09-21 22:59:37 -07:00
parent 14bd95d76d
commit 3a85a6bce1
4 changed files with 495 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/uptimekuma/README.md
+74 -41
View File
@@ -1,54 +1,87 @@
# uptimekuma — service uptime monitor (esh-docker-vm)
# uptimekuma — the fleet's SERVICE-layer monitor (ana-docker)
`louislam/uptime-kuma:latest` on **esh-docker-vm** (`10.0.50.45:3001`). Widget
slug `nethealth` feeds the Uptime Kuma card on the fleet dashboard.
`louislam/uptime-kuma:2.5.5` on **ana-docker** (`10.250.50.70:3001`), beside the
Beszel and Dozzle hubs.
## Why it is in this repo
## Why it exists — the lane split
Adopted 2026-08-18 while cleaning up Homepage. It had run unmanaged on the
host for ~12 months, and its `homepage.group=Apps` label put it in the wrong
dashboard group — where it collided with a *manual* entry for the same
service in the homepage stack's `services.yaml` under Monitoring. Result: two
Uptime Kuma cards on the Main tab, one with the widget and one without.
Measured 2026-09-21, after the fleet dashboard sat dead for three days while
every monitoring tool reported correctly:
That is precisely the "never list a labelled container manually" failure the
homepage README warns about, and it survived the 2026-08-17 audit because a
duplicate looks like two plausible cards rather than like an error. The fix
was to keep the label as the single source of truth, move it to `Monitoring`,
and delete the manual block.
| tool | what it asks | scope |
|---|---|---|
| **Beszel** | is the box alive, and is it out of CPU / memory / disk / heat | 18 hosts × {Status, CPU, Memory, Disk, Temperature} |
| **Uptime Kuma** (this) | **is the service actually serving** | the fleet's user-facing endpoints |
| **Homepage** | — | **DISPLAY ONLY.** Polls 38 URLs, alerts nobody. |
## Deviations from house convention (deliberate, not oversights)
Beszel and Kuma are **not redundant** — they are disjoint, and the seam between
them is where things die quietly. Beszel's alerts bind to a *system* with a
threshold (`alerts` table: `system, name, value, min`); there is **no URL
column**, so it is structurally incapable of "this endpoint should return 200".
That is not a configuration gap, it is the data model.
The compose was adopted as-run rather than normalised — this was a dashboard
fix, not a rewrite of a service that has been stable for a year:
On 2026-09-18 `homepage` wedged in an unkillable D-state. Beszel reported
`esh-vm-docker: up` — correctly; the host *was* up. Kuma was not watching it.
The dead dashboard fell between two working instruments and stayed dark for
three days until the operator hit a 404.
- `restart: always`, not `unless-stopped`. It revives on Docker daemon start
even if it was stopped on purpose.
- No `healthcheck:` block. The image ships its own, which is why `docker ps`
reports `healthy` anyway.
## Rebuilt from scratch, 2026-09-21
Worth normalising on the next deliberate touch, not worth a restart today.
Operator-authorised: *"uptime-kuma was never really used… you can even dump the
existing container and config and start over from scratch."* The prior instance
had four monitors — three firewall pings and one HTTPS check — and nothing was
migrated. This also skipped the one-way v1→v2 database migration entirely.
## The container name
Two things changed with the rebuild:
Live container was `45d2522a8cb6_uptime-kuma` — Docker's collision rename from
some past recreate where the old container could not be removed. Recreating
under this compose restores the plain `uptime-kuma` name. Harmless either way.
**Pinned to `2.5.5`, and `:latest` is now a documented trap.** Upstream keeps
`latest` on the **1.x** line: an August 2026 pull produced an image *built
2024-12-20* running 1.23.16. Verified by digest — `latest` and `1` resolve to
the same image, while `2`/`next` carry 2.5.5. 2.x has been stable since 2.2.0
(2026-03-05), twelve releases, zero prereleases. Pinned exactly rather than
floating on `2`, for the same reason `latest` burned us.
**Moved esh-docker-vm → ana-docker.** House placement rule (CLAUDE.md): *cross-
site services live on ana-docker and pull from agents on the other hosts* — a
fleet-wide service monitor is exactly that. And `esh-docker-vm` has wedged
unkillably twice in four months (2026-06-03, 2026-09-18), so it is the worst
box in the fleet to host the thing that would tell us. A monitor also cannot
report the failure of the host it runs on; Beszel covers that layer, and the
two should not share a failure domain.
Normalised at the same time: `restart: always` → `unless-stopped`, which the
2026-08-18 README flagged as *"worth normalising on the next deliberate touch"*.
`always` revives a container that was stopped on purpose.
## Scripting it
**There is no REST CRUD API — in either major version.** Checked against the
2.5.5 source tree, not the docs: `server/routers/` contains exactly two files,
`api-router.js` (Prometheus `/metrics`, badges, entry page) and
`status-page-router.js`. API keys unlock `/metrics` and badges only.
Automation goes over **Socket.IO**, the same channel the web UI uses;
`server/socket-handlers/general-socket-handler.js` carries `add`,
`editMonitor`, `deleteMonitor`, `getMonitorList`.
⚠️ **Do not reach for `uptime-kuma-api`** (the Python wrapper). It is abandoned
— last release 2023-09-26 — and its stated ceiling is *"support for uptime kuma
1.23.0 and 1.23.1"*. It does not support 2.x and never will. Use a thin
first-party Socket.IO client instead.
## The Homepage widget is deliberately absent
The compose carries **no `homepage.widget.*` labels**. The widget needs a
published status-page slug, and a from-scratch install has none — it would poll
a 404 forever. That matters more here than elsewhere: homepage widgets pointed
at dead targets are the suspected mechanism behind *both* of that dashboard's
unkillable wedges (see `incident_esh_docker_nfs_boot_race`, 2026-06-03 entry).
Re-add `homepage.widget.type` / `.url` / `.slug` only once the status page
actually exists. Label changes need `docker compose up -d`, not `restart`.
## Data
All monitor definitions and history live in the named volume
`uptimekuma_uptime-kuma`. **Recreating the container is safe. Deleting that
volume loses every monitor and all history** — there is no config file to
restore from, the state is entirely in the volume's SQLite DB.
## Deploy
```bash
scripts/deploy-stack.sh esh-docker-vm uptimekuma --compose
ssh lkraven@10.0.50.45 'cd /opt/docker/compose/uptimekuma && docker compose up -d'
```
Label changes need a **recreate**, not a restart — Docker only applies labels
when the container is created. After recreating, the Homepage container also
needs a recreate before it re-reads the discovered set.
Named volume `uptimekuma_uptime-kuma` → `/app/data` holds every monitor
definition and all history. Recreating the container is safe; deleting that
volume is not.
stacks/uptimekuma/compose.yaml
+58 -29
View File
@@ -1,49 +1,78 @@
# uptime-kuma — service uptime monitor on esh-docker-vm.
# uptime-kuma — the fleet's SERVICE-layer monitor.
#
# Adopted into this repo 2026-08-18. It had been running unmanaged on the
# host since 2025 and was only noticed because its homepage label put it in
# the wrong group: it was ALSO listed manually in the homepage stack's
# services.yaml under Monitoring, so the dashboard rendered two Uptime Kuma
# cards — the labelled one in Apps (with the widget) and the manual one in
# Monitoring (without). The label is now the single source of truth and the
# manual entry is gone.
# REBUILT FROM SCRATCH 2026-09-21 (operator-authorised: "uptime-kuma was never
# really used... you can even dump the existing container and config and start
# over from scratch"). The previous instance carried four monitors, all of them
# firewall pings, and 21-month-old code. Nothing was migrated.
#
# ⚠️ The compose is preserved as it ran, not normalised. Two deliberate
# deviations from the house conventions, left alone because this commit is
# a homepage fix and not a rewrite of a service that has been up for a
# year:
# - `restart: always` rather than `unless-stopped` (always revives on
# daemon start even after a deliberate manual stop)
# - no healthcheck block (the image ships its own, which is why
# `docker ps` reports healthy)
# ⚠️ IMAGE TAG — DO NOT USE `:latest` HERE. Upstream keeps `latest` pointing at
# the 1.x line, so a pull in August 2026 handed us an image BUILT 2024-12-20
# (1.23.16). Measured, not assumed: `latest` and `1` resolve to the same
# digest, while `2`/`next` carry 2.5.5. All development is on 2.x, which has
# been stable since 2.2.0 (2026-03-05) across twelve releases with zero
# prereleases. Pinned EXACTLY rather than floating on `2` for the same reason
# `latest` burned us: a floating tag is a version you did not choose.
#
# The data volume `uptimekuma_uptime-kuma` holds every monitor definition
# and all history. Recreating the container is safe; deleting that volume
# is not.
# MOVED esh-docker-vm -> ana-docker in the same rebuild, for two reasons:
# 1. House placement rule (CLAUDE.md): "Cross-site services (e.g. Beszel hub,
# Dozzle hub) live on ana-docker and pull from agents on the other hosts."
# A fleet-wide service monitor is exactly that, and it now sits beside the
# Beszel hub it complements.
# 2. esh-docker-vm has wedged unkillably TWICE in four months (2026-06-03 and
# 2026-09-18, both homepage in D-state). The fleet's service monitor should
# not live on the least reliable box in the fleet — and a monitor cannot
# report the failure of the host it runs on. Beszel covers the host layer;
# this covers the service layer; they should not share a failure domain.
#
# THE LANE SPLIT this service exists to fill (measured 2026-09-21):
# Beszel -> 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts
# bind to a SYSTEM with a threshold; there is no URL column, so it
# is structurally incapable of "this endpoint should return 200".
# Kuma -> is the service actually serving. This file.
# Homepage-> DISPLAY ONLY. It polls 38 URLs and alerts nobody; it is a
# dashboard, never a monitor.
# The gap between the first two is where homepage sat dead for three days while
# both tools reported correctly.
name: uptimekuma
services:
uptime-kuma:
image: louislam/uptime-kuma:latest
image: louislam/uptime-kuma:2.5.5
container_name: uptime-kuma
restart: always
# Normalised from the adopted-as-run `always` — the 2026-08-18 README said
# this was "worth normalising on the next deliberate touch", and a rebuild
# from scratch is that touch. `always` revives a container that was stopped
# ON PURPOSE, which is the wrong behaviour for a service we may deliberately
# take down during maintenance.
restart: unless-stopped
environment:
# Skips 2.x's interactive "choose a database" first-run screen.
# setup-database.js: UPTIME_KUMA_DB_TYPE overrides db-config.json and
# writes it, so the stack comes up ready for the admin-account step
# instead of parking on a form. SQLite is right here -- one operator,
# a few dozen monitors; MariaDB is for the multi-thousand-check case.
UPTIME_KUMA_DB_TYPE: sqlite
ports:
- "3001:3001"
volumes:
- uptime-kuma:/app/data
labels:
# Monitoring, NOT Apps — this is the group the dashboard's own layout
# reserves for Backrest / Beszel / Dozzle / this. See the note above.
# Monitoring, NOT Apps — this is the group the dashboard's layout reserves
# for Backrest / Beszel / Dozzle / this.
- homepage.group=Monitoring
- homepage.name=Uptime Kuma
- homepage.icon=mdi-arrow-up-bold-circle
- homepage.description=Service Monitoring (esh)
- homepage.href=http://10.0.50.45:3001
- homepage.siteMonitor=http://10.0.50.45:3001
- homepage.widget.type=uptimekuma
- homepage.widget.url=http://10.0.50.45:3001
- homepage.widget.slug=nethealth
- homepage.description=Service monitoring (fleet)
- homepage.href=http://10.250.50.70:3001
- homepage.siteMonitor=http://10.250.50.70:3001
# ⚠️ NO `homepage.widget.*` LABELS YET, deliberately. The widget needs a
# published status-page slug; on a from-scratch install none exists, so
# the widget would poll a 404 forever. That matters more than usual
# here: homepage widgets pointed at dead targets are the suspected
# mechanism behind BOTH of this dashboard's unkillable D-state wedges
# (see incident_esh_docker_nfs_boot_race, 2026-06-03). Re-add the widget
# labels only once the status page actually exists.
networks:
- tnet
stacks/uptimekuma/monitors.yaml
+78
View File
@@ -0,0 +1,78 @@
# Uptime Kuma — the fleet's service-layer monitors, as code.
#
# scripts/kuma seed stacks/uptimekuma/monitors.yaml
#
# Idempotent: keyed on NAME, so re-running edits rather than duplicating. Name
# and not URL, because a URL legitimately repeats (a root and a /healthz on one
# service) and a re-run must never fork the board.
#
# WHAT BELONGS HERE — the lane rule, so this file does not sprawl to 115 rows:
# IN — services that should ALWAYS be up, whose silent death costs us work.
# OUT — inference seats (they come and go BY DESIGN; a dormant seat is normal
# and alerting on it is pure noise), hosts and hypervisors and firewalls
# (Beszel's lane: 18 hosts x Status/CPU/Memory/Disk/Temp), and Uptime
# Kuma itself (it cannot report its own death — that is Beszel's job).
#
# Every URL below was probed before being written here: all returned 200 on
# 2026-09-21. A seed that ships red on day one teaches everyone to ignore the
# board, which is how you end up with a monitor nobody reads.
#
# NOT SEEDED, deliberately: "althing chamber" (10.250.50.70:7881) is on the
# Homepage dashboard but refuses connections right now. It is either genuinely
# down or retired; seeding it would put a red row on a brand-new board before
# anyone has decided which. Resolve it, then add it.
monitors:
# ---- fleet toolchain: dead = agents and the operator are blocked ----
- name: althing post office
url: http://10.100.50.40:8390/
description: the bus every agent session reads mail from
- name: The Booth
url: http://10.100.10.50:8090/healthz
description: operator-review surface; real healthz, not a root page
- name: task-board
url: http://10.250.50.70:7878
- name: vor
url: http://10.250.50.70:7879
# ---- the dashboard that started all this ----
# Dead for three days in Sept 2026 while Beszel correctly reported its host
# UP. This row is the entire reason the service layer exists.
- name: Homepage
url: http://10.0.50.45:5100/
description: fleet dashboard (esh-docker-vm) — the 2026-09-18 silent death
# ---- credentials + code: dead = nothing ships ----
- name: Gitea
url: https://gitea.phasefinal.com
- name: Vaultwarden
url: https://vaultwarden.phasefinal.com
description: the vault every agent reads credentials from
# ---- AI control plane (the gateways, NOT the seats behind them) ----
- name: LiteLLM Gateway
url: http://10.250.50.70:4000/health/liveliness
description: liveliness endpoint, so a model outage does not read as a gateway outage
- name: Asset Engine
url: http://10.250.50.70:8200/api/v1/services
- name: talk
url: https://talk.nh3.phasefinal.com:8092/
description: fleet voice bench
# ---- monitoring + backup: a blind monitor is worse than none ----
- name: Beszel hub
url: http://10.250.50.70:8090
description: the host layer; if this is down we are blind to 18 hosts
- name: Backrest
url: http://10.250.50.70:9898
description: restic orchestration — a silent backup failure is the expensive kind
- name: Dozzle hub
url: http://10.250.50.70:8088