vh
6f0a9b9fae
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.
Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.
Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.
⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.
Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.
parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.
KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.
⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.
End-to-end, both shapes, read back from the inbox:
[Uptime Kuma] Homepage is DOWN + target + board link
[Uptime Kuma] althing (infra-ops) Testing (no fabricated subject)
[Beszel] BRIDGE RENAME CHECK + hub footer, unchanged
ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
72 lines
3.5 KiB
Markdown
72 lines
3.5 KiB
Markdown
# Fleet alert bridge — monitoring tools into the althing inbox
|
|
|
|
`althing-alert-bridge.service` runs on nh3-dev as lkraven, listening at
|
|
`10.100.10.50:8096`. It turns an HTTP alert into a `postbox send` to the
|
|
**infra-ops** inbox, using the established automation identity.
|
|
|
|
| route | sender | payload |
|
|
|---|---|---|
|
|
| `POST /beszel` | Beszel (Shoutrrr generic JSON) | `{title, message}` |
|
|
| `POST /kuma` | Uptime Kuma (webhook, JSON) | `{heartbeat, monitor, msg}` |
|
|
|
|
```sh
|
|
scripts/elway infra-ops@10.100.10.50 --playbook playbooks/althing-alert-bridge.yaml
|
|
systemctl status althing-alert-bridge
|
|
curl -fsS http://10.100.10.50:8096/healthz # lists the routes it serves
|
|
```
|
|
|
|
## Why one bridge with a route registry
|
|
|
|
It was `beszel-althing` until 2026-09-21, with the `[Beszel]` subject prefix and
|
|
the Beszel hub footer hardcoded. When Uptime Kuma was rebuilt as the fleet's
|
|
service layer it needed the same path — and routing it through unchanged would
|
|
have delivered Kuma outages labelled `[Beszel]`, pointing the reader at the
|
|
wrong dashboard. **An alert that lies about its own source is worse than no
|
|
alert.**
|
|
|
|
Generalising cost a dict. A sibling service would have cost a second unit, a
|
|
second port, and a second thing to notice had died. The name went with it: a
|
|
service called `beszel-althing` that also carries Kuma alerts is exactly the
|
|
invisible coupling that makes a future session look in the wrong place.
|
|
|
|
⚠ **`/beszel` is frozen.** Its prefix, footer and default title must stay
|
|
byte-identical — that path was verified end-to-end in production (2026-09-10,
|
|
thread `01M25Z0WFDJM92GPTJQF769HJ7`) and a refactor is not allowed to quietly
|
|
change what it emits. `deliver()` still defaults to the Beszel route so the
|
|
original three tests exercise it unchanged, and a test asserts the Kuma footer
|
|
never leaks into a Beszel body or vice versa.
|
|
|
|
## Payload shapes are handled here, not in the sending tool
|
|
|
|
Uptime Kuma can render a custom webhook body, which would have let the bridge
|
|
stay dumb. It is done here instead, because Kuma's notification config lives in
|
|
its own database — and that database was destroyed and rebuilt from scratch on
|
|
2026-09-21. Anything that lives only in a tool's DB is lost on the next rebuild.
|
|
Format knowledge belongs in git, next to a test.
|
|
|
|
`parse_kuma` also handles the **monitorless** case: `testNotification` and
|
|
certificate-expiry alerts arrive with no monitor and no heartbeat, and
|
|
fabricating `unknown monitor is ?` from those makes a real alert read like a
|
|
bug — observed live, then fixed and pinned by a test.
|
|
|
|
## Operational notes
|
|
|
|
- Source-allowlisted by IP (`ALERT_ALLOWED_SOURCES`). `10.250.50.70` is
|
|
ana-docker, which now runs **both** the Beszel hub and Uptime Kuma, so one
|
|
entry covers both senders.
|
|
- Delivery reports success only after `postbox` returns a receipt. Failures
|
|
return HTTP 502 and land in the journal; **there is no retry queue**, so a
|
|
post-office outage can lose an alert.
|
|
- `/healthz` checks the bridge process, not the downstream inbox. End-to-end
|
|
verification means sending a real POST and reading the thread back — both
|
|
routes were verified that way on 2026-09-21 (`postbox thread` does not
|
|
consume the inbox).
|
|
- Legacy `BESZEL_*` env names still resolve, so a half-finished deploy starts
|
|
instead of crash-looping. New deployments use `ALERT_*`.
|
|
- To reroute, change `ALERT_RECIPIENT` in the unit and redeploy. Leave
|
|
`ALTHING_HANDLE` as infra-ops so the sender stays identifiable as automation.
|
|
|
|
```sh
|
|
python3 -m unittest discover -s services/althing-alert-bridge -p 'test_*.py' # 11 tests
|
|
```
|