vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(uptimekuma): rebuild on 2.5.5 as the fleet's service layer, with a client

Browse Source 
Homepage sat dead for three days in September while every monitoring tool
reported correctly. Beszel said its host was up -- it was. Uptime Kuma was not
watching it. The dashboard fell into the seam between two working instruments.

Measured before changing anything:
  - Beszel: 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts
    table is (system, name, value, min) -- there is NO url column, so it is
    structurally incapable of "this endpoint should return 200". Not a config
    gap; the data model.
  - Uptime Kuma: 6 rows, 2 of them folders. Four real monitors, all firewalls.
  - So the two are NOT redundant. They are disjoint, and the service layer
    between them was empty.

REBUILT FROM SCRATCH, operator-authorised ("uptime-kuma was never really
used... you can even dump the existing container and config"). Nothing was
migrated, which also skipped the one-way v1->v2 database migration.

  - Pinned to 2.5.5. `:latest` is a documented trap now: upstream keeps it on
    the 1.x line, so an August 2026 pull produced an image BUILT 2024-12-20
    running 1.23.16. Verified by digest -- latest and 1 share one digest while
    2/next carry 2.5.5. Pinned exactly, not floating on 2, for the same reason.
  - Moved esh-docker-vm -> ana-docker. House placement rule puts cross-site
    services beside the Beszel and Dozzle hubs, and esh-docker-vm has wedged
    unkillably twice in four months. A monitor also cannot report the failure
    of the host it runs on, so it should not share a failure domain with the
    host layer.
  - Normalised restart: always -> unless-stopped, which the 2026-08-18 README
    flagged as worth doing on the next deliberate touch.
  - UPTIME_KUMA_DB_TYPE=sqlite in the compose skips 2.x's interactive database
    screen, so the stack comes up ready rather than parked on a form.

scripts/kuma is a first-party Socket.IO client, because both obvious paths are
wrong: there is no REST CRUD API in EITHER major version (server/routers/ holds
exactly two files, /metrics + badges + status pages), and the community wrapper
uptime-kuma-api is abandoned -- last release 2023-09-26, ceiling 1.23.1, no 2.x
support ever.

  ⚠ getMonitorList's callback returns only {ok:true}; the list arrives as a
  SEPARATE pushed monitorList event. Reading the ack yields an empty board that
  looks authoritative -- which duplicated all 13 rows on the first re-seed
  before the bug was found. The client now waits for the push, and carries a
  dedupe verb because of it.

13 monitors seeded from monitors.yaml, keyed on name so a re-run updates rather
than forking the board -- proven by re-running it (0 added, 13 updated), not
assumed. Every URL was probed before being written: all 200. A board that ships
red teaches everyone to ignore it.

Verified: 13 rows, no duplicates, all UP with "200 - OK" read from the database
WITH its WAL (a first read of kuma.db alone showed a stale 26 -- the copy
predated the deletes). Homepage renders exactly one Uptime Kuma card.

⚠ NOT YET WIRED: notification delivery. The board detects but tells nobody,
which is the same gap this work exists to close. The beszel-althing bridge
hardcodes a [Beszel] subject prefix and hub footer, so routing Kuma through it
unchanged would mislabel the alerts. Needs a decision before it is generalised.
This commit is contained in:
vh
2026-09-21 22:59:37 -07:00
parent 14bd95d76d
commit 3a85a6bce1
4 changed files with 495 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/uptimekuma/compose.yaml
+58 -29
View File
@@ -1,49 +1,78 @@
# uptime-kuma — service uptime monitor on esh-docker-vm.
# uptime-kuma — the fleet's SERVICE-layer monitor.
#
# Adopted into this repo 2026-08-18. It had been running unmanaged on the
# host since 2025 and was only noticed because its homepage label put it in
# the wrong group: it was ALSO listed manually in the homepage stack's
# services.yaml under Monitoring, so the dashboard rendered two Uptime Kuma
# cards — the labelled one in Apps (with the widget) and the manual one in
# Monitoring (without). The label is now the single source of truth and the
# manual entry is gone.
# REBUILT FROM SCRATCH 2026-09-21 (operator-authorised: "uptime-kuma was never
# really used... you can even dump the existing container and config and start
# over from scratch"). The previous instance carried four monitors, all of them
# firewall pings, and 21-month-old code. Nothing was migrated.
#
# ⚠️ The compose is preserved as it ran, not normalised. Two deliberate
# deviations from the house conventions, left alone because this commit is
# a homepage fix and not a rewrite of a service that has been up for a
# year:
# - `restart: always` rather than `unless-stopped` (always revives on
# daemon start even after a deliberate manual stop)
# - no healthcheck block (the image ships its own, which is why
# `docker ps` reports healthy)
# ⚠️ IMAGE TAG — DO NOT USE `:latest` HERE. Upstream keeps `latest` pointing at
# the 1.x line, so a pull in August 2026 handed us an image BUILT 2024-12-20
# (1.23.16). Measured, not assumed: `latest` and `1` resolve to the same
# digest, while `2`/`next` carry 2.5.5. All development is on 2.x, which has
# been stable since 2.2.0 (2026-03-05) across twelve releases with zero
# prereleases. Pinned EXACTLY rather than floating on `2` for the same reason
# `latest` burned us: a floating tag is a version you did not choose.
#
# The data volume `uptimekuma_uptime-kuma` holds every monitor definition
# and all history. Recreating the container is safe; deleting that volume
# is not.
# MOVED esh-docker-vm -> ana-docker in the same rebuild, for two reasons:
# 1. House placement rule (CLAUDE.md): "Cross-site services (e.g. Beszel hub,
# Dozzle hub) live on ana-docker and pull from agents on the other hosts."
# A fleet-wide service monitor is exactly that, and it now sits beside the
# Beszel hub it complements.
# 2. esh-docker-vm has wedged unkillably TWICE in four months (2026-06-03 and
# 2026-09-18, both homepage in D-state). The fleet's service monitor should
# not live on the least reliable box in the fleet — and a monitor cannot
# report the failure of the host it runs on. Beszel covers the host layer;
# this covers the service layer; they should not share a failure domain.
#
# THE LANE SPLIT this service exists to fill (measured 2026-09-21):
# Beszel -> 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts
# bind to a SYSTEM with a threshold; there is no URL column, so it
# is structurally incapable of "this endpoint should return 200".
# Kuma -> is the service actually serving. This file.
# Homepage-> DISPLAY ONLY. It polls 38 URLs and alerts nobody; it is a
# dashboard, never a monitor.
# The gap between the first two is where homepage sat dead for three days while
# both tools reported correctly.
name: uptimekuma
services:
uptime-kuma:
image: louislam/uptime-kuma:latest
image: louislam/uptime-kuma:2.5.5
container_name: uptime-kuma
restart: always
# Normalised from the adopted-as-run `always` — the 2026-08-18 README said
# this was "worth normalising on the next deliberate touch", and a rebuild
# from scratch is that touch. `always` revives a container that was stopped
# ON PURPOSE, which is the wrong behaviour for a service we may deliberately
# take down during maintenance.
restart: unless-stopped
environment:
# Skips 2.x's interactive "choose a database" first-run screen.
# setup-database.js: UPTIME_KUMA_DB_TYPE overrides db-config.json and
# writes it, so the stack comes up ready for the admin-account step
# instead of parking on a form. SQLite is right here -- one operator,
# a few dozen monitors; MariaDB is for the multi-thousand-check case.
UPTIME_KUMA_DB_TYPE: sqlite
ports:
- "3001:3001"
volumes:
- uptime-kuma:/app/data
labels:
# Monitoring, NOT Apps — this is the group the dashboard's own layout
# reserves for Backrest / Beszel / Dozzle / this. See the note above.
# Monitoring, NOT Apps — this is the group the dashboard's layout reserves
# for Backrest / Beszel / Dozzle / this.
- homepage.group=Monitoring
- homepage.name=Uptime Kuma
- homepage.icon=mdi-arrow-up-bold-circle
- homepage.description=Service Monitoring (esh)
- homepage.href=http://10.0.50.45:3001
- homepage.siteMonitor=http://10.0.50.45:3001
- homepage.widget.type=uptimekuma
- homepage.widget.url=http://10.0.50.45:3001
- homepage.widget.slug=nethealth
- homepage.description=Service monitoring (fleet)
- homepage.href=http://10.250.50.70:3001
- homepage.siteMonitor=http://10.250.50.70:3001
# ⚠️ NO `homepage.widget.*` LABELS YET, deliberately. The widget needs a
# published status-page slug; on a from-scratch install none exists, so
# the widget would poll a 404 forever. That matters more than usual
# here: homepage widgets pointed at dead targets are the suspected
# mechanism behind BOTH of this dashboard's unkillable D-state wedges
# (see incident_esh_docker_nfs_boot_race, 2026-06-03). Re-add the widget
# labels only once the status page actually exists.
networks:
- tnet