feat(alerts): generalize the althing bridge, wire Kuma to it, retire chamber
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.
Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.
Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.
⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.
Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.
parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.
KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.
⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.
End-to-end, both shapes, read back from the inbox:
[Uptime Kuma] Homepage is DOWN + target + board link
[Uptime Kuma] althing (infra-ops) Testing (no fabricated subject)
[Beszel] BRIDGE RENAME CHECK + hub footer, unchanged
ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
This commit is contained in:
@@ -73,6 +73,18 @@ MONITOR_DEFAULTS = {
|
||||
}
|
||||
|
||||
|
||||
# A channel registered with isDefault + applyExisting attaches to monitors that
|
||||
# ALREADY exist, not merely to ones created afterwards -- without it, seeding
|
||||
# monitors before channels silently leaves the existing board unwired.
|
||||
NOTIFICATION_DEFAULTS = {
|
||||
"type": "webhook",
|
||||
"isDefault": True,
|
||||
"applyExisting": True,
|
||||
"webhookContentType": "json",
|
||||
"active": True,
|
||||
}
|
||||
|
||||
|
||||
def vault_get(key: str) -> dict:
|
||||
"""{'username','password'} from the vault. Loud on failure -- a silent
|
||||
empty credential would show up as a confusing auth error much later."""
|
||||
@@ -106,10 +118,16 @@ class Kuma:
|
||||
# updating it. Capture the push; never trust the ack for data.
|
||||
self._monitor_list: dict | None = None
|
||||
|
||||
self._notification_list: list | None = None
|
||||
|
||||
@self.sio.on("monitorList")
|
||||
def _on_monitor_list(data):
|
||||
self._monitor_list = data or {}
|
||||
|
||||
@self.sio.on("notificationList")
|
||||
def _on_notification_list(data):
|
||||
self._notification_list = data or []
|
||||
|
||||
def __enter__(self):
|
||||
self.sio.connect(self.url, transports=["websocket"], wait_timeout=20)
|
||||
self._connected = True
|
||||
@@ -157,6 +175,29 @@ class Kuma:
|
||||
def delete(self, monitor_id: int):
|
||||
return self.call("deleteMonitor", monitor_id, False)
|
||||
|
||||
def notifications(self, timeout: int = 10) -> list:
|
||||
"""Registered notification channels.
|
||||
|
||||
⚠ DIFFERENT PUSH RULE FROM monitorList, and the difference bit once.
|
||||
There is no `getNotificationList` event to ask with -- the server pushes
|
||||
`notificationList` ONCE, at login, and again after any change. So this
|
||||
must NOT clear the captured value before waiting: doing that discards
|
||||
the only copy it will ever be sent and then blocks for the full timeout
|
||||
before reporting an empty list, which reads exactly like "no channels
|
||||
are configured" and is a lie.
|
||||
"""
|
||||
deadline = time.time() + timeout
|
||||
while self._notification_list is None and time.time() < deadline:
|
||||
self.sio.sleep(0.2)
|
||||
return self._notification_list or []
|
||||
|
||||
def add_notification(self, notification: dict):
|
||||
"""Register (or update) a channel. notificationID None => create."""
|
||||
return self.call("addNotification", notification, None)
|
||||
|
||||
def test_notification(self, notification: dict):
|
||||
return self.call("testNotification", notification)
|
||||
|
||||
def add(self, monitor: dict):
|
||||
return self.call("add", {**MONITOR_DEFAULTS, **monitor})
|
||||
|
||||
@@ -204,8 +245,20 @@ def cmd_seed(args):
|
||||
same service), and re-running a seed must never fork the board."""
|
||||
spec = yaml.safe_load(open(args.file))
|
||||
wanted = spec["monitors"] if isinstance(spec, dict) else spec
|
||||
channels = spec.get("notifications", []) if isinstance(spec, dict) else []
|
||||
k = connected_and_logged_in()
|
||||
try:
|
||||
# Channels FIRST: a monitor created while a default channel exists
|
||||
# inherits it. Seeding monitors first and alerting second is how you get
|
||||
# a board that detects and tells nobody -- the exact gap this fleet's
|
||||
# service layer was built to close.
|
||||
have = {n.get("name") for n in k.notifications()}
|
||||
for ch in channels:
|
||||
if ch["name"] in have:
|
||||
print(f" = {ch['name']} (channel)")
|
||||
continue
|
||||
k.add_notification({**NOTIFICATION_DEFAULTS, **ch})
|
||||
print(f" + {ch['name']} (channel)")
|
||||
existing = {m.get("name"): m for m in k.monitors().values()}
|
||||
added = updated = 0
|
||||
for m in wanted:
|
||||
@@ -223,6 +276,21 @@ def cmd_seed(args):
|
||||
k.__exit__()
|
||||
|
||||
|
||||
def cmd_channels(args):
|
||||
k = connected_and_logged_in()
|
||||
try:
|
||||
ns = k.notifications()
|
||||
if not ns:
|
||||
print("no notification channels — THE BOARD ALERTS NOBODY")
|
||||
return
|
||||
for n in ns:
|
||||
cfg = json.loads(n.get("config") or "{}")
|
||||
print(" %-26s %-9s default=%-5s %s" % (
|
||||
n.get("name"), cfg.get("type"), n.get("isDefault"), cfg.get("webhookURL", "")))
|
||||
finally:
|
||||
k.__exit__()
|
||||
|
||||
|
||||
def cmd_dedupe(args):
|
||||
"""Keep the LOWEST id per name, delete the rest. Needed because the first
|
||||
version of this client read the getMonitorList ack instead of the pushed
|
||||
@@ -274,6 +342,7 @@ def main():
|
||||
a.add_argument("--port", type=int)
|
||||
a.set_defaults(fn=cmd_add)
|
||||
sub.add_parser("dedupe").set_defaults(fn=cmd_dedupe)
|
||||
sub.add_parser("channels").set_defaults(fn=cmd_channels)
|
||||
args = p.parse_args()
|
||||
try:
|
||||
args.fn(args)
|
||||
|
||||
Reference in New Issue
Block a user