Files
esh-pfi-infrastructure/stacks
vh 3a85a6bce1 feat(uptimekuma): rebuild on 2.5.5 as the fleet's service layer, with a client
Homepage sat dead for three days in September while every monitoring tool
reported correctly. Beszel said its host was up -- it was. Uptime Kuma was not
watching it. The dashboard fell into the seam between two working instruments.

Measured before changing anything:
  - Beszel: 18 hosts x {Status, CPU, Memory, Disk, Temperature}. Its alerts
    table is (system, name, value, min) -- there is NO url column, so it is
    structurally incapable of "this endpoint should return 200". Not a config
    gap; the data model.
  - Uptime Kuma: 6 rows, 2 of them folders. Four real monitors, all firewalls.
  - So the two are NOT redundant. They are disjoint, and the service layer
    between them was empty.

REBUILT FROM SCRATCH, operator-authorised ("uptime-kuma was never really
used... you can even dump the existing container and config"). Nothing was
migrated, which also skipped the one-way v1->v2 database migration.

  - Pinned to 2.5.5. `:latest` is a documented trap now: upstream keeps it on
    the 1.x line, so an August 2026 pull produced an image BUILT 2024-12-20
    running 1.23.16. Verified by digest -- latest and 1 share one digest while
    2/next carry 2.5.5. Pinned exactly, not floating on 2, for the same reason.
  - Moved esh-docker-vm -> ana-docker. House placement rule puts cross-site
    services beside the Beszel and Dozzle hubs, and esh-docker-vm has wedged
    unkillably twice in four months. A monitor also cannot report the failure
    of the host it runs on, so it should not share a failure domain with the
    host layer.
  - Normalised restart: always -> unless-stopped, which the 2026-08-18 README
    flagged as worth doing on the next deliberate touch.
  - UPTIME_KUMA_DB_TYPE=sqlite in the compose skips 2.x's interactive database
    screen, so the stack comes up ready rather than parked on a form.

scripts/kuma is a first-party Socket.IO client, because both obvious paths are
wrong: there is no REST CRUD API in EITHER major version (server/routers/ holds
exactly two files, /metrics + badges + status pages), and the community wrapper
uptime-kuma-api is abandoned -- last release 2023-09-26, ceiling 1.23.1, no 2.x
support ever.

  ⚠ getMonitorList's callback returns only {ok:true}; the list arrives as a
  SEPARATE pushed monitorList event. Reading the ack yields an empty board that
  looks authoritative -- which duplicated all 13 rows on the first re-seed
  before the bug was found. The client now waits for the push, and carries a
  dedupe verb because of it.

13 monitors seeded from monitors.yaml, keyed on name so a re-run updates rather
than forking the board -- proven by re-running it (0 added, 13 updated), not
assumed. Every URL was probed before being written: all 200. A board that ships
red teaches everyone to ignore it.

Verified: 13 rows, no duplicates, all UP with "200 - OK" read from the database
WITH its WAL (a first read of kuma.db alone showed a stale 26 -- the copy
predated the deletes). Homepage renders exactly one Uptime Kuma card.

⚠ NOT YET WIRED: notification delivery. The board detects but tells nobody,
which is the same gap this work exists to close. The beszel-althing bridge
hardcodes a [Beszel] subject prefix and hub footer, so routing Kuma through it
unchanged would mislabel the alerts. Needs a decision before it is generalised.
2026-09-21 22:59:37 -07:00
..