vh
3a85a6bce1
Homepage sat dead for three days in September while every monitoring tool
reported correctly. Beszel said its host was up -- it was. Uptime Kuma was not
watching it. The dashboard fell into the seam between two working instruments.
Measured before changing anything:
- Beszel: 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts
table is (system, name, value, min) -- there is NO url column, so it is
structurally incapable of "this endpoint should return 200". Not a config
gap; the data model.
- Uptime Kuma: 6 rows, 2 of them folders. Four real monitors, all firewalls.
- So the two are NOT redundant. They are disjoint, and the service layer
between them was empty.
REBUILT FROM SCRATCH, operator-authorised ("uptime-kuma was never really
used... you can even dump the existing container and config"). Nothing was
migrated, which also skipped the one-way v1->v2 database migration.
- Pinned to 2.5.5. `:latest` is a documented trap now: upstream keeps it on
the 1.x line, so an August 2026 pull produced an image BUILT 2024-12-20
running 1.23.16. Verified by digest -- latest and 1 share one digest while
2/next carry 2.5.5. Pinned exactly, not floating on 2, for the same reason.
- Moved esh-docker-vm -> ana-docker. House placement rule puts cross-site
services beside the Beszel and Dozzle hubs, and esh-docker-vm has wedged
unkillably twice in four months. A monitor also cannot report the failure
of the host it runs on, so it should not share a failure domain with the
host layer.
- Normalised restart: always -> unless-stopped, which the 2026-08-18 README
flagged as worth doing on the next deliberate touch.
- UPTIME_KUMA_DB_TYPE=sqlite in the compose skips 2.x's interactive database
screen, so the stack comes up ready rather than parked on a form.
scripts/kuma is a first-party Socket.IO client, because both obvious paths are
wrong: there is no REST CRUD API in EITHER major version (server/routers/ holds
exactly two files, /metrics + badges + status pages), and the community wrapper
uptime-kuma-api is abandoned -- last release 2023-09-26, ceiling 1.23.1, no 2.x
support ever.
⚠ getMonitorList's callback returns only {ok:true}; the list arrives as a
SEPARATE pushed monitorList event. Reading the ack yields an empty board that
looks authoritative -- which duplicated all 13 rows on the first re-seed
before the bug was found. The client now waits for the push, and carries a
dedupe verb because of it.
13 monitors seeded from monitors.yaml, keyed on name so a re-run updates rather
than forking the board -- proven by re-running it (0 added, 13 updated), not
assumed. Every URL was probed before being written: all 200. A board that ships
red teaches everyone to ignore it.
Verified: 13 rows, no duplicates, all UP with "200 - OK" read from the database
WITH its WAL (a first read of kuma.db alone showed a stale 26 -- the copy
predated the deletes). Homepage renders exactly one Uptime Kuma card.
⚠ NOT YET WIRED: notification delivery. The board detects but tells nobody,
which is the same gap this work exists to close. The beszel-althing bridge
hardcodes a [Beszel] subject prefix and hub footer, so routing Kuma through it
unchanged would mislabel the alerts. Needs a decision before it is generalised.
4.3 KiB
uptimekuma — the fleet's SERVICE-layer monitor (ana-docker)
louislam/uptime-kuma:2.5.5 on ana-docker (10.250.50.70:3001), beside the
Beszel and Dozzle hubs.
Why it exists — the lane split
Measured 2026-09-21, after the fleet dashboard sat dead for three days while every monitoring tool reported correctly:
| tool | what it asks | scope |
|---|---|---|
| Beszel | is the box alive, and is it out of CPU / memory / disk / heat | 18 hosts × {Status, CPU, Memory, Disk, Temperature} |
| Uptime Kuma (this) | is the service actually serving | the fleet's user-facing endpoints |
| Homepage | — | DISPLAY ONLY. Polls 38 URLs, alerts nobody. |
Beszel and Kuma are not redundant — they are disjoint, and the seam between
them is where things die quietly. Beszel's alerts bind to a system with a
threshold (alerts table: system, name, value, min); there is no URL
column, so it is structurally incapable of "this endpoint should return 200".
That is not a configuration gap, it is the data model.
On 2026-09-18 homepage wedged in an unkillable D-state. Beszel reported
esh-vm-docker: up — correctly; the host was up. Kuma was not watching it.
The dead dashboard fell between two working instruments and stayed dark for
three days until the operator hit a 404.
Rebuilt from scratch, 2026-09-21
Operator-authorised: "uptime-kuma was never really used… you can even dump the existing container and config and start over from scratch." The prior instance had four monitors — three firewall pings and one HTTPS check — and nothing was migrated. This also skipped the one-way v1→v2 database migration entirely.
Two things changed with the rebuild:
Pinned to 2.5.5, and :latest is now a documented trap. Upstream keeps
latest on the 1.x line: an August 2026 pull produced an image built
2024-12-20 running 1.23.16. Verified by digest — latest and 1 resolve to
the same image, while 2/next carry 2.5.5. 2.x has been stable since 2.2.0
(2026-03-05), twelve releases, zero prereleases. Pinned exactly rather than
floating on 2, for the same reason latest burned us.
Moved esh-docker-vm → ana-docker. House placement rule (CLAUDE.md): cross-
site services live on ana-docker and pull from agents on the other hosts — a
fleet-wide service monitor is exactly that. And esh-docker-vm has wedged
unkillably twice in four months (2026-06-03, 2026-09-18), so it is the worst
box in the fleet to host the thing that would tell us. A monitor also cannot
report the failure of the host it runs on; Beszel covers that layer, and the
two should not share a failure domain.
Normalised at the same time: restart: always → unless-stopped, which the
2026-08-18 README flagged as "worth normalising on the next deliberate touch".
always revives a container that was stopped on purpose.
Scripting it
There is no REST CRUD API — in either major version. Checked against the
2.5.5 source tree, not the docs: server/routers/ contains exactly two files,
api-router.js (Prometheus /metrics, badges, entry page) and
status-page-router.js. API keys unlock /metrics and badges only.
Automation goes over Socket.IO, the same channel the web UI uses;
server/socket-handlers/general-socket-handler.js carries add,
editMonitor, deleteMonitor, getMonitorList.
⚠️ Do not reach for uptime-kuma-api (the Python wrapper). It is abandoned
— last release 2023-09-26 — and its stated ceiling is "support for uptime kuma
1.23.0 and 1.23.1". It does not support 2.x and never will. Use a thin
first-party Socket.IO client instead.
The Homepage widget is deliberately absent
The compose carries no homepage.widget.* labels. The widget needs a
published status-page slug, and a from-scratch install has none — it would poll
a 404 forever. That matters more here than elsewhere: homepage widgets pointed
at dead targets are the suspected mechanism behind both of that dashboard's
unkillable wedges (see incident_esh_docker_nfs_boot_race, 2026-06-03 entry).
Re-add homepage.widget.type / .url / .slug only once the status page
actually exists. Label changes need docker compose up -d, not restart.
Data
Named volume uptimekuma_uptime-kuma → /app/data holds every monitor
definition and all history. Recreating the container is safe; deleting that
volume is not.