Files
esh-pfi-infrastructure/services/althing-alert-bridge/bridge.py
T
vh 6f0a9b9fae feat(alerts): generalize the althing bridge, wire Kuma to it, retire chamber
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.

Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.

Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.

⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.

Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.

parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.

KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.

⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.

End-to-end, both shapes, read back from the inbox:
  [Uptime Kuma] Homepage is DOWN          + target + board link
  [Uptime Kuma] althing (infra-ops) Testing   (no fabricated subject)
  [Beszel] BRIDGE RENAME CHECK            + hub footer, unchanged

ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
2026-09-21 23:11:54 -07:00

163 lines
6.6 KiB
Python

#!/usr/bin/env python3
"""Fleet alert bridge — HTTP alerts from monitoring tools into the althing inbox.
One listener, one route per SOURCE. Each source declares its own subject prefix,
its own footer link, and how to read its payload, because the tools do not agree
on a shape and the reader needs to know which tool is talking.
POST /beszel <- Beszel, via Shoutrrr generic JSON: {title, message}
POST /kuma <- Uptime Kuma, via its webhook: {heartbeat, monitor, msg}
WHY A REGISTRY RATHER THAN A SECOND SERVICE. The prefix and footer were
hardcoded when Beszel was the only caller, so a second tool routed through it
would have arrived labelled "[Beszel]" -- an alert that lies about its own
source is worse than no alert, because it sends you to the wrong dashboard.
Generalising costs a dict; a sibling service costs a second unit, a second port
and a second thing to notice has died.
⚠ `/beszel` IS FROZEN. Its prefix, footer and default title must stay byte-
identical -- Beszel's delivery path has been verified end-to-end in production
(2026-09-10, thread 01M25Z0WFDJM92GPTJQF769HJ7) and this refactor is not
allowed to quietly change what that path emits. `deliver()` still defaults to
the Beszel route precisely so the original tests exercise it unchanged.
"""
import json
import os
import subprocess
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
def _env(*names, default=None):
"""First env var that is set. Accepts the legacy BESZEL_* names so a
half-finished deploy (new bridge, old unit) still starts instead of
crash-looping on a KeyError."""
for n in names:
if os.environ.get(n):
return os.environ[n]
if default is None:
raise KeyError(' / '.join(names))
return default
def parse_generic(payload):
"""Shoutrrr generic JSON — what Beszel sends. {title, message}."""
return payload.get('title'), payload.get('message')
def parse_kuma(payload):
"""Uptime Kuma's webhook — {heartbeat, monitor, msg}.
Kuma's own `msg` already reads like a sentence ("[Homepage] [🔴 Down]
connect ECONNREFUSED"), so it is the body. The title is built from the
monitor name and the heartbeat status so the SUBJECT alone says which
service and which direction -- that is what you see in an inbox list
without opening anything.
"""
monitor = payload.get('monitor') or {}
heartbeat = payload.get('heartbeat') or {}
body = payload.get('msg') or ''
name = monitor.get('name')
# Kuma status: 0 down, 1 up, 2 pending, 3 maintenance.
state = {0: 'DOWN', 1: 'UP', 2: 'PENDING', 3: 'MAINTENANCE'}.get(heartbeat.get('status'))
if name and state:
title = f'{name} is {state}'
else:
# NOT every Kuma notification is a monitor transition: `testNotification`
# and certificate-expiry alerts arrive with no monitor and no heartbeat.
# Fabricating "unknown monitor is ?" from those makes a real alert read
# like a bug; the message itself is the better subject.
first = body.strip().splitlines()[0] if body.strip() else ''
title = (first[:120] or 'notification')
detail = heartbeat.get('msg')
if detail and detail not in body:
body = f'{body}\n\n{detail}'.strip()
url = monitor.get('url')
if url:
body = f'{body}\n\nTarget: {url}'.strip()
return title, body
SOURCES = {
# FROZEN — see the module docstring.
'/beszel': {
'prefix': '[Beszel] ',
'default_title': 'Beszel fleet alert',
'footer': '\n\nHub: http://10.250.50.70:8090\n',
'parse': parse_generic,
},
'/kuma': {
'prefix': '[Uptime Kuma] ',
'default_title': 'Uptime Kuma alert',
'footer': '\n\nBoard: http://10.250.50.70:3001\n',
'parse': parse_kuma,
},
}
def deliver(payload, route='/beszel'):
source = SOURCES[route]
title, message = source['parse'](payload)
if title is None:
title = source['default_title']
if not isinstance(title, str) or not isinstance(message, str) or not message.strip():
raise ValueError('Expected a nonempty message and string title')
result = subprocess.run(
[_env('POSTBOX'), '--json', 'send',
'--to', _env('ALERT_RECIPIENT', 'BESZEL_ALERT_RECIPIENT'),
'--subject', source['prefix'] + title],
input=message + source['footer'],
text=True, capture_output=True, timeout=25,
)
if result.returncode:
raise RuntimeError('postbox delivery failed: ' + result.stderr.strip())
receipt = json.loads(result.stdout)
print(json.dumps({'event': 'delivered', 'route': route, 'title': title, 'receipt': receipt}), flush=True)
return receipt
class Handler(BaseHTTPRequestHandler):
def respond(self, status, body):
data = json.dumps(body).encode()
self.send_response(status)
self.send_header('Content-Type', 'application/json')
self.send_header('Content-Length', str(len(data)))
self.end_headers()
self.wfile.write(data)
def do_GET(self):
self.respond(200 if self.path == '/healthz' else 404,
{'service': 'althing-alert-bridge',
'routes': sorted(SOURCES),
'delivery': 'verified per POST'})
def do_POST(self):
if self.path not in SOURCES:
return self.respond(404, {'error': 'Unknown route', 'routes': sorted(SOURCES)})
allowed = _env('ALERT_ALLOWED_SOURCES', 'BESZEL_ALLOWED_SOURCES').split(',')
if self.client_address[0] not in allowed:
return self.respond(403, {'error': 'Source not allowed'})
try:
length = int(self.headers.get('Content-Length', '0'))
if not 0 < length <= 65536:
raise ValueError('Invalid body size')
self.connection.settimeout(10)
payload = json.loads(self.rfile.read(length))
if not isinstance(payload, dict):
raise ValueError('Expected JSON object')
receipt = deliver(payload, self.path)
except (ValueError, TypeError) as exc:
return self.respond(400, {'error': str(exc)})
except (OSError, RuntimeError, subprocess.TimeoutExpired) as exc:
print(json.dumps({'event': 'delivery_failed', 'route': self.path, 'error': str(exc)}), flush=True)
return self.respond(502, {'error': 'Althing delivery failed; inspect service journal'})
self.respond(200, {'delivered': True, 'receipt': receipt})
if __name__ == '__main__':
ThreadingHTTPServer(
(_env('ALERT_BIND_HOST', 'BESZEL_BIND_HOST'),
int(_env('ALERT_BIND_PORT', 'BESZEL_BIND_PORT', default='8096'))),
Handler,
).serve_forever()