vh
6f0a9b9fae
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.
Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.
Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.
⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.
Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.
parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.
KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.
⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.
End-to-end, both shapes, read back from the inbox:
[Uptime Kuma] Homepage is DOWN + target + board link
[Uptime Kuma] althing (infra-ops) Testing (no fabricated subject)
[Beszel] BRIDGE RENAME CHECK + hub footer, unchanged
ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
3.5 KiB
Fleet alert bridge — monitoring tools into the althing inbox
althing-alert-bridge.service runs on nh3-dev as lkraven, listening at
10.100.10.50:8096. It turns an HTTP alert into a postbox send to the
infra-ops inbox, using the established automation identity.
| route | sender | payload |
|---|---|---|
POST /beszel |
Beszel (Shoutrrr generic JSON) | {title, message} |
POST /kuma |
Uptime Kuma (webhook, JSON) | {heartbeat, monitor, msg} |
scripts/elway infra-ops@10.100.10.50 --playbook playbooks/althing-alert-bridge.yaml
systemctl status althing-alert-bridge
curl -fsS http://10.100.10.50:8096/healthz # lists the routes it serves
Why one bridge with a route registry
It was beszel-althing until 2026-09-21, with the [Beszel] subject prefix and
the Beszel hub footer hardcoded. When Uptime Kuma was rebuilt as the fleet's
service layer it needed the same path — and routing it through unchanged would
have delivered Kuma outages labelled [Beszel], pointing the reader at the
wrong dashboard. An alert that lies about its own source is worse than no
alert.
Generalising cost a dict. A sibling service would have cost a second unit, a
second port, and a second thing to notice had died. The name went with it: a
service called beszel-althing that also carries Kuma alerts is exactly the
invisible coupling that makes a future session look in the wrong place.
⚠ /beszel is frozen. Its prefix, footer and default title must stay
byte-identical — that path was verified end-to-end in production (2026-09-10,
thread 01M25Z0WFDJM92GPTJQF769HJ7) and a refactor is not allowed to quietly
change what it emits. deliver() still defaults to the Beszel route so the
original three tests exercise it unchanged, and a test asserts the Kuma footer
never leaks into a Beszel body or vice versa.
Payload shapes are handled here, not in the sending tool
Uptime Kuma can render a custom webhook body, which would have let the bridge stay dumb. It is done here instead, because Kuma's notification config lives in its own database — and that database was destroyed and rebuilt from scratch on 2026-09-21. Anything that lives only in a tool's DB is lost on the next rebuild. Format knowledge belongs in git, next to a test.
parse_kuma also handles the monitorless case: testNotification and
certificate-expiry alerts arrive with no monitor and no heartbeat, and
fabricating unknown monitor is ? from those makes a real alert read like a
bug — observed live, then fixed and pinned by a test.
Operational notes
- Source-allowlisted by IP (
ALERT_ALLOWED_SOURCES).10.250.50.70is ana-docker, which now runs both the Beszel hub and Uptime Kuma, so one entry covers both senders. - Delivery reports success only after
postboxreturns a receipt. Failures return HTTP 502 and land in the journal; there is no retry queue, so a post-office outage can lose an alert. /healthzchecks the bridge process, not the downstream inbox. End-to-end verification means sending a real POST and reading the thread back — both routes were verified that way on 2026-09-21 (postbox threaddoes not consume the inbox).- Legacy
BESZEL_*env names still resolve, so a half-finished deploy starts instead of crash-looping. New deployments useALERT_*. - To reroute, change
ALERT_RECIPIENTin the unit and redeploy. LeaveALTHING_HANDLEas infra-ops so the sender stays identifiable as automation.
python3 -m unittest discover -s services/althing-alert-bridge -p 'test_*.py' # 11 tests