feat(alerts): generalize the althing bridge, wire Kuma to it, retire chamber
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.
Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.
Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.
⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.
Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.
parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.
KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.
⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.
End-to-end, both shapes, read back from the inbox:
[Uptime Kuma] Homepage is DOWN + target + board link
[Uptime Kuma] althing (infra-ops) Testing (no fabricated subject)
[Beszel] BRIDGE RENAME CHECK + hub footer, unchanged
ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
This commit is contained in:
@@ -0,0 +1,71 @@
|
||||
# Fleet alert bridge — monitoring tools into the althing inbox
|
||||
|
||||
`althing-alert-bridge.service` runs on nh3-dev as lkraven, listening at
|
||||
`10.100.10.50:8096`. It turns an HTTP alert into a `postbox send` to the
|
||||
**infra-ops** inbox, using the established automation identity.
|
||||
|
||||
| route | sender | payload |
|
||||
|---|---|---|
|
||||
| `POST /beszel` | Beszel (Shoutrrr generic JSON) | `{title, message}` |
|
||||
| `POST /kuma` | Uptime Kuma (webhook, JSON) | `{heartbeat, monitor, msg}` |
|
||||
|
||||
```sh
|
||||
scripts/elway infra-ops@10.100.10.50 --playbook playbooks/althing-alert-bridge.yaml
|
||||
systemctl status althing-alert-bridge
|
||||
curl -fsS http://10.100.10.50:8096/healthz # lists the routes it serves
|
||||
```
|
||||
|
||||
## Why one bridge with a route registry
|
||||
|
||||
It was `beszel-althing` until 2026-09-21, with the `[Beszel]` subject prefix and
|
||||
the Beszel hub footer hardcoded. When Uptime Kuma was rebuilt as the fleet's
|
||||
service layer it needed the same path — and routing it through unchanged would
|
||||
have delivered Kuma outages labelled `[Beszel]`, pointing the reader at the
|
||||
wrong dashboard. **An alert that lies about its own source is worse than no
|
||||
alert.**
|
||||
|
||||
Generalising cost a dict. A sibling service would have cost a second unit, a
|
||||
second port, and a second thing to notice had died. The name went with it: a
|
||||
service called `beszel-althing` that also carries Kuma alerts is exactly the
|
||||
invisible coupling that makes a future session look in the wrong place.
|
||||
|
||||
⚠ **`/beszel` is frozen.** Its prefix, footer and default title must stay
|
||||
byte-identical — that path was verified end-to-end in production (2026-09-10,
|
||||
thread `01M25Z0WFDJM92GPTJQF769HJ7`) and a refactor is not allowed to quietly
|
||||
change what it emits. `deliver()` still defaults to the Beszel route so the
|
||||
original three tests exercise it unchanged, and a test asserts the Kuma footer
|
||||
never leaks into a Beszel body or vice versa.
|
||||
|
||||
## Payload shapes are handled here, not in the sending tool
|
||||
|
||||
Uptime Kuma can render a custom webhook body, which would have let the bridge
|
||||
stay dumb. It is done here instead, because Kuma's notification config lives in
|
||||
its own database — and that database was destroyed and rebuilt from scratch on
|
||||
2026-09-21. Anything that lives only in a tool's DB is lost on the next rebuild.
|
||||
Format knowledge belongs in git, next to a test.
|
||||
|
||||
`parse_kuma` also handles the **monitorless** case: `testNotification` and
|
||||
certificate-expiry alerts arrive with no monitor and no heartbeat, and
|
||||
fabricating `unknown monitor is ?` from those makes a real alert read like a
|
||||
bug — observed live, then fixed and pinned by a test.
|
||||
|
||||
## Operational notes
|
||||
|
||||
- Source-allowlisted by IP (`ALERT_ALLOWED_SOURCES`). `10.250.50.70` is
|
||||
ana-docker, which now runs **both** the Beszel hub and Uptime Kuma, so one
|
||||
entry covers both senders.
|
||||
- Delivery reports success only after `postbox` returns a receipt. Failures
|
||||
return HTTP 502 and land in the journal; **there is no retry queue**, so a
|
||||
post-office outage can lose an alert.
|
||||
- `/healthz` checks the bridge process, not the downstream inbox. End-to-end
|
||||
verification means sending a real POST and reading the thread back — both
|
||||
routes were verified that way on 2026-09-21 (`postbox thread` does not
|
||||
consume the inbox).
|
||||
- Legacy `BESZEL_*` env names still resolve, so a half-finished deploy starts
|
||||
instead of crash-looping. New deployments use `ALERT_*`.
|
||||
- To reroute, change `ALERT_RECIPIENT` in the unit and redeploy. Leave
|
||||
`ALTHING_HANDLE` as infra-ops so the sender stays identifiable as automation.
|
||||
|
||||
```sh
|
||||
python3 -m unittest discover -s services/althing-alert-bridge -p 'test_*.py' # 11 tests
|
||||
```
|
||||
@@ -0,0 +1,27 @@
|
||||
[Unit]
|
||||
Description=Fleet alert bridge — Beszel + Uptime Kuma alerts into the althing infra-ops inbox
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
User=lkraven
|
||||
Group=lkraven
|
||||
Environment=PYTHONUNBUFFERED=1
|
||||
Environment=ALTHING_HANDLE=infra-ops
|
||||
Environment=ALTHING_POST_OFFICE=http://10.100.50.40:8390
|
||||
Environment=POSTBOX=/home/lkraven/.local/bin/postbox
|
||||
Environment=ALERT_RECIPIENT=infra-ops
|
||||
Environment=ALERT_BIND_HOST=10.100.10.50
|
||||
# 10.250.50.70 is ana-docker, which now runs BOTH the Beszel hub and Uptime
|
||||
# Kuma, so one entry covers both senders. 10.100.10.50/127.0.0.1 are local
|
||||
# diagnostics from nh3-dev itself.
|
||||
Environment=ALERT_ALLOWED_SOURCES=10.250.50.70,10.100.10.50,127.0.0.1
|
||||
ExecStart=/usr/bin/python3 /opt/althing-alert-bridge/bridge.py
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
PrivateTmp=true
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,162 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fleet alert bridge — HTTP alerts from monitoring tools into the althing inbox.
|
||||
|
||||
One listener, one route per SOURCE. Each source declares its own subject prefix,
|
||||
its own footer link, and how to read its payload, because the tools do not agree
|
||||
on a shape and the reader needs to know which tool is talking.
|
||||
|
||||
POST /beszel <- Beszel, via Shoutrrr generic JSON: {title, message}
|
||||
POST /kuma <- Uptime Kuma, via its webhook: {heartbeat, monitor, msg}
|
||||
|
||||
WHY A REGISTRY RATHER THAN A SECOND SERVICE. The prefix and footer were
|
||||
hardcoded when Beszel was the only caller, so a second tool routed through it
|
||||
would have arrived labelled "[Beszel]" -- an alert that lies about its own
|
||||
source is worse than no alert, because it sends you to the wrong dashboard.
|
||||
Generalising costs a dict; a sibling service costs a second unit, a second port
|
||||
and a second thing to notice has died.
|
||||
|
||||
⚠ `/beszel` IS FROZEN. Its prefix, footer and default title must stay byte-
|
||||
identical -- Beszel's delivery path has been verified end-to-end in production
|
||||
(2026-09-10, thread 01M25Z0WFDJM92GPTJQF769HJ7) and this refactor is not
|
||||
allowed to quietly change what that path emits. `deliver()` still defaults to
|
||||
the Beszel route precisely so the original tests exercise it unchanged.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
|
||||
def _env(*names, default=None):
|
||||
"""First env var that is set. Accepts the legacy BESZEL_* names so a
|
||||
half-finished deploy (new bridge, old unit) still starts instead of
|
||||
crash-looping on a KeyError."""
|
||||
for n in names:
|
||||
if os.environ.get(n):
|
||||
return os.environ[n]
|
||||
if default is None:
|
||||
raise KeyError(' / '.join(names))
|
||||
return default
|
||||
|
||||
|
||||
def parse_generic(payload):
|
||||
"""Shoutrrr generic JSON — what Beszel sends. {title, message}."""
|
||||
return payload.get('title'), payload.get('message')
|
||||
|
||||
|
||||
def parse_kuma(payload):
|
||||
"""Uptime Kuma's webhook — {heartbeat, monitor, msg}.
|
||||
|
||||
Kuma's own `msg` already reads like a sentence ("[Homepage] [🔴 Down]
|
||||
connect ECONNREFUSED"), so it is the body. The title is built from the
|
||||
monitor name and the heartbeat status so the SUBJECT alone says which
|
||||
service and which direction -- that is what you see in an inbox list
|
||||
without opening anything.
|
||||
"""
|
||||
monitor = payload.get('monitor') or {}
|
||||
heartbeat = payload.get('heartbeat') or {}
|
||||
body = payload.get('msg') or ''
|
||||
name = monitor.get('name')
|
||||
# Kuma status: 0 down, 1 up, 2 pending, 3 maintenance.
|
||||
state = {0: 'DOWN', 1: 'UP', 2: 'PENDING', 3: 'MAINTENANCE'}.get(heartbeat.get('status'))
|
||||
|
||||
if name and state:
|
||||
title = f'{name} is {state}'
|
||||
else:
|
||||
# NOT every Kuma notification is a monitor transition: `testNotification`
|
||||
# and certificate-expiry alerts arrive with no monitor and no heartbeat.
|
||||
# Fabricating "unknown monitor is ?" from those makes a real alert read
|
||||
# like a bug; the message itself is the better subject.
|
||||
first = body.strip().splitlines()[0] if body.strip() else ''
|
||||
title = (first[:120] or 'notification')
|
||||
detail = heartbeat.get('msg')
|
||||
if detail and detail not in body:
|
||||
body = f'{body}\n\n{detail}'.strip()
|
||||
url = monitor.get('url')
|
||||
if url:
|
||||
body = f'{body}\n\nTarget: {url}'.strip()
|
||||
return title, body
|
||||
|
||||
|
||||
SOURCES = {
|
||||
# FROZEN — see the module docstring.
|
||||
'/beszel': {
|
||||
'prefix': '[Beszel] ',
|
||||
'default_title': 'Beszel fleet alert',
|
||||
'footer': '\n\nHub: http://10.250.50.70:8090\n',
|
||||
'parse': parse_generic,
|
||||
},
|
||||
'/kuma': {
|
||||
'prefix': '[Uptime Kuma] ',
|
||||
'default_title': 'Uptime Kuma alert',
|
||||
'footer': '\n\nBoard: http://10.250.50.70:3001\n',
|
||||
'parse': parse_kuma,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def deliver(payload, route='/beszel'):
|
||||
source = SOURCES[route]
|
||||
title, message = source['parse'](payload)
|
||||
if title is None:
|
||||
title = source['default_title']
|
||||
if not isinstance(title, str) or not isinstance(message, str) or not message.strip():
|
||||
raise ValueError('Expected a nonempty message and string title')
|
||||
result = subprocess.run(
|
||||
[_env('POSTBOX'), '--json', 'send',
|
||||
'--to', _env('ALERT_RECIPIENT', 'BESZEL_ALERT_RECIPIENT'),
|
||||
'--subject', source['prefix'] + title],
|
||||
input=message + source['footer'],
|
||||
text=True, capture_output=True, timeout=25,
|
||||
)
|
||||
if result.returncode:
|
||||
raise RuntimeError('postbox delivery failed: ' + result.stderr.strip())
|
||||
receipt = json.loads(result.stdout)
|
||||
print(json.dumps({'event': 'delivered', 'route': route, 'title': title, 'receipt': receipt}), flush=True)
|
||||
return receipt
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def respond(self, status, body):
|
||||
data = json.dumps(body).encode()
|
||||
self.send_response(status)
|
||||
self.send_header('Content-Type', 'application/json')
|
||||
self.send_header('Content-Length', str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
def do_GET(self):
|
||||
self.respond(200 if self.path == '/healthz' else 404,
|
||||
{'service': 'althing-alert-bridge',
|
||||
'routes': sorted(SOURCES),
|
||||
'delivery': 'verified per POST'})
|
||||
|
||||
def do_POST(self):
|
||||
if self.path not in SOURCES:
|
||||
return self.respond(404, {'error': 'Unknown route', 'routes': sorted(SOURCES)})
|
||||
allowed = _env('ALERT_ALLOWED_SOURCES', 'BESZEL_ALLOWED_SOURCES').split(',')
|
||||
if self.client_address[0] not in allowed:
|
||||
return self.respond(403, {'error': 'Source not allowed'})
|
||||
try:
|
||||
length = int(self.headers.get('Content-Length', '0'))
|
||||
if not 0 < length <= 65536:
|
||||
raise ValueError('Invalid body size')
|
||||
self.connection.settimeout(10)
|
||||
payload = json.loads(self.rfile.read(length))
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError('Expected JSON object')
|
||||
receipt = deliver(payload, self.path)
|
||||
except (ValueError, TypeError) as exc:
|
||||
return self.respond(400, {'error': str(exc)})
|
||||
except (OSError, RuntimeError, subprocess.TimeoutExpired) as exc:
|
||||
print(json.dumps({'event': 'delivery_failed', 'route': self.path, 'error': str(exc)}), flush=True)
|
||||
return self.respond(502, {'error': 'Althing delivery failed; inspect service journal'})
|
||||
self.respond(200, {'delivered': True, 'receipt': receipt})
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
ThreadingHTTPServer(
|
||||
(_env('ALERT_BIND_HOST', 'BESZEL_BIND_HOST'),
|
||||
int(_env('ALERT_BIND_PORT', 'BESZEL_BIND_PORT', default='8096'))),
|
||||
Handler,
|
||||
).serve_forever()
|
||||
@@ -0,0 +1,109 @@
|
||||
import importlib.util
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
spec=importlib.util.spec_from_file_location('bridge',Path(__file__).with_name('bridge.py'))
|
||||
bridge=importlib.util.module_from_spec(spec);spec.loader.exec_module(bridge)
|
||||
|
||||
class DeliveryTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.env=patch.dict(os.environ,POSTBOX='/bin/postbox',BESZEL_ALERT_RECIPIENT='infra-ops')
|
||||
self.env.start();self.addCleanup(self.env.stop)
|
||||
def test_posts_body_as_stdin_and_returns_receipt(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id": 42}','')) as run:
|
||||
self.assertEqual(bridge.deliver({'title':'Disk 85%','message':'tank is full'})['message_id'],42)
|
||||
args,kw=run.call_args
|
||||
self.assertEqual(args[0],['/bin/postbox','--json','send','--to','infra-ops','--subject','[Beszel] Disk 85%'])
|
||||
self.assertIn('tank is full',kw['input'])
|
||||
def test_delivery_failure_is_not_success(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],2,'','post office unavailable')):
|
||||
with self.assertRaises(RuntimeError):bridge.deliver({'message':'disk full'})
|
||||
def test_rejects_empty_message_without_sending(self):
|
||||
with patch.object(bridge.subprocess,'run') as run:
|
||||
with self.assertRaises(ValueError):bridge.deliver({'message':''})
|
||||
run.assert_not_called()
|
||||
|
||||
class KumaRouteTests(unittest.TestCase):
|
||||
"""The /kuma route. The point of generalising was that an alert must not
|
||||
lie about its own source -- a Kuma outage labelled [Beszel] sends you to
|
||||
the wrong dashboard, which is worse than no alert."""
|
||||
|
||||
def setUp(self):
|
||||
self.env=patch.dict(os.environ,POSTBOX='/bin/postbox',ALERT_RECIPIENT='infra-ops')
|
||||
self.env.start();self.addCleanup(self.env.stop)
|
||||
|
||||
DOWN={'monitor':{'name':'Homepage','url':'http://10.0.50.45:5100/'},
|
||||
'heartbeat':{'status':0,'msg':'connect ECONNREFUSED'},
|
||||
'msg':'[Homepage] [Down] connect ECONNREFUSED'}
|
||||
|
||||
def test_subject_names_the_service_and_the_direction(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":7}','')) as run:
|
||||
bridge.deliver(self.DOWN,'/kuma')
|
||||
argv=run.call_args[0][0]
|
||||
self.assertIn('[Uptime Kuma] Homepage is DOWN',argv)
|
||||
self.assertNotIn('[Beszel] Homepage is DOWN',argv)
|
||||
|
||||
def test_up_transition_reads_as_up(self):
|
||||
up={'monitor':{'name':'Gitea'},'heartbeat':{'status':1,'msg':'200 - OK'},'msg':'[Gitea] [Up] 200 - OK'}
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":8}','')) as run:
|
||||
bridge.deliver(up,'/kuma')
|
||||
self.assertIn('[Uptime Kuma] Gitea is UP',run.call_args[0][0])
|
||||
|
||||
def test_body_carries_the_target_and_the_kuma_board_not_the_beszel_hub(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":9}','')) as run:
|
||||
bridge.deliver(self.DOWN,'/kuma')
|
||||
body=run.call_args[1]['input']
|
||||
self.assertIn('http://10.0.50.45:5100/',body)
|
||||
self.assertIn('Board: http://10.250.50.70:3001',body)
|
||||
self.assertNotIn('Hub: http://10.250.50.70:8090',body) # the Beszel footer must not leak
|
||||
|
||||
def test_beszel_footer_is_unchanged_by_the_refactor(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":10}','')) as run:
|
||||
bridge.deliver({'title':'Disk 85%','message':'tank is full'},'/beszel')
|
||||
body=run.call_args[1]['input']
|
||||
self.assertIn('Hub: http://10.250.50.70:8090',body)
|
||||
self.assertNotIn('Board:',body)
|
||||
|
||||
def test_a_nameless_alert_prefers_its_message_over_a_fabricated_subject(self):
|
||||
"""Superseded an earlier assertion that this should read "unknown
|
||||
monitor is DOWN". It should not: a real monitor always carries a name,
|
||||
so a nameless payload is degenerate, and "something broke" tells the
|
||||
reader more than a placeholder that looks like a bug."""
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":11}','')) as run:
|
||||
argv=None
|
||||
bridge.deliver({'heartbeat':{'status':0},'msg':'something broke'},'/kuma')
|
||||
argv=run.call_args[0][0]
|
||||
self.assertIn('[Uptime Kuma] something broke',argv)
|
||||
self.assertNotIn('unknown monitor',' '.join(argv))
|
||||
|
||||
def test_empty_kuma_message_is_refused_without_sending(self):
|
||||
with patch.object(bridge.subprocess,'run') as run:
|
||||
with self.assertRaises(ValueError):
|
||||
bridge.deliver({'monitor':{'name':'X'},'heartbeat':{'status':0},'msg':''},'/kuma')
|
||||
run.assert_not_called()
|
||||
|
||||
def test_a_monitorless_notification_uses_its_own_message_as_the_subject(self):
|
||||
"""testNotification and cert-expiry arrive with no monitor and no
|
||||
heartbeat. Fabricating "unknown monitor is ?" from those makes a real
|
||||
alert read like a bug -- observed live on 2026-09-21."""
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":13}','')) as run:
|
||||
bridge.deliver({'msg':'althing (infra-ops) Testing'},'/kuma')
|
||||
argv=run.call_args[0][0]
|
||||
self.assertIn('[Uptime Kuma] althing (infra-ops) Testing',argv)
|
||||
self.assertNotIn('unknown monitor',' '.join(argv))
|
||||
self.assertNotIn('is ?',' '.join(argv))
|
||||
|
||||
def test_legacy_beszel_env_names_still_resolve(self):
|
||||
"""A half-finished deploy (new bridge, old unit) must start, not crash."""
|
||||
with patch.dict(os.environ,{'ALERT_RECIPIENT':'','BESZEL_ALERT_RECIPIENT':'infra-ops'}):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":12}','')) as run:
|
||||
bridge.deliver({'title':'t','message':'m'})
|
||||
self.assertIn('infra-ops',run.call_args[0][0])
|
||||
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -1,36 +0,0 @@
|
||||
# Beszel to althing
|
||||
|
||||
`beszel-althing.service` runs on nh3-dev as lkraven, listening at
|
||||
`10.100.10.50:8096`. Beszel sends Shoutrrr generic JSON to `/beszel`; the
|
||||
bridge invokes the supported `postbox send` command with the body on stdin.
|
||||
It uses the established infra-ops automation identity and sends to infra-ops.
|
||||
|
||||
Deploy from the repository root:
|
||||
|
||||
```sh
|
||||
scripts/elway infra-ops@10.100.10.50 --playbook playbooks/beszel-althing.yaml
|
||||
```
|
||||
|
||||
The service accepts requests only from ana-docker and local diagnostic
|
||||
addresses. There is no public listener or new althing handle. It reports
|
||||
success only after postbox returns a delivery receipt. Failures return HTTP
|
||||
502 and appear in the system journal; there is no hidden retry queue. A
|
||||
post-office outage can therefore lose an alert, and needs independent health
|
||||
monitoring if guaranteed delivery during such outages becomes a requirement.
|
||||
|
||||
```sh
|
||||
systemctl status beszel-althing
|
||||
sudo -n journalctl -u beszel-althing --since '1 hour ago'
|
||||
curl -fsS http://10.100.10.50:8096/healthz
|
||||
```
|
||||
|
||||
`/healthz` checks the bridge process, not the downstream inbox. End-to-end
|
||||
verification requires a real Beszel threshold transition plus its althing
|
||||
receipt. The first verified alert is recorded in `stacks/beszel/README.md`.
|
||||
|
||||
To reroute later, change `BESZEL_ALERT_RECIPIENT` in the canonical unit to
|
||||
`miranda`, deploy, and trigger another end-to-end test. Leave `ALTHING_HANDLE`
|
||||
as infra-ops so the sender remains identifiable as infrastructure automation.
|
||||
The operator explicitly chose infra-ops for now.
|
||||
|
||||
Run `python3 -m unittest discover -s services/beszel-althing -p 'test_*.py'`.
|
||||
@@ -1,24 +0,0 @@
|
||||
[Unit]
|
||||
Description=Beszel alerts to the althing infra-ops inbox
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
User=lkraven
|
||||
Group=lkraven
|
||||
Environment=PYTHONUNBUFFERED=1
|
||||
Environment=ALTHING_HANDLE=infra-ops
|
||||
Environment=ALTHING_POST_OFFICE=http://10.100.50.40:8390
|
||||
Environment=POSTBOX=/home/lkraven/.local/bin/postbox
|
||||
Environment=BESZEL_ALERT_RECIPIENT=infra-ops
|
||||
Environment=BESZEL_BIND_HOST=10.100.10.50
|
||||
Environment=BESZEL_ALLOWED_SOURCES=10.250.50.70,10.100.10.50,127.0.0.1
|
||||
ExecStart=/usr/bin/python3 /opt/beszel-althing/bridge.py
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
PrivateTmp=true
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -1,63 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Receive Beszel's Shoutrrr JSON and deliver through the supported postbox CLI."""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
|
||||
def deliver(payload):
|
||||
title = payload.get('title', 'Beszel fleet alert')
|
||||
message = payload.get('message')
|
||||
if not isinstance(title, str) or not isinstance(message, str) or not message.strip():
|
||||
raise ValueError('Expected a nonempty message and string title')
|
||||
result = subprocess.run(
|
||||
[os.environ['POSTBOX'], '--json', 'send', '--to', os.environ['BESZEL_ALERT_RECIPIENT'],
|
||||
'--subject', '[Beszel] ' + title],
|
||||
input=message + '\n\nHub: http://10.250.50.70:8090\n',
|
||||
text=True, capture_output=True, timeout=25,
|
||||
)
|
||||
if result.returncode:
|
||||
raise RuntimeError('postbox delivery failed: ' + result.stderr.strip())
|
||||
receipt = json.loads(result.stdout)
|
||||
print(json.dumps({'event': 'delivered', 'title': title, 'receipt': receipt}), flush=True)
|
||||
return receipt
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def respond(self, status, body):
|
||||
data = json.dumps(body).encode()
|
||||
self.send_response(status)
|
||||
self.send_header('Content-Type', 'application/json')
|
||||
self.send_header('Content-Length', str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
def do_GET(self):
|
||||
self.respond(200 if self.path == '/healthz' else 404,
|
||||
{'service': 'beszel-althing', 'delivery': 'verified per POST'})
|
||||
|
||||
def do_POST(self):
|
||||
if self.path != '/beszel':
|
||||
return self.respond(404, {'error': 'Unknown route'})
|
||||
if self.client_address[0] not in os.environ['BESZEL_ALLOWED_SOURCES'].split(','):
|
||||
return self.respond(403, {'error': 'Source not allowed'})
|
||||
try:
|
||||
length = int(self.headers.get('Content-Length', '0'))
|
||||
if not 0 < length <= 65536:
|
||||
raise ValueError('Invalid body size')
|
||||
self.connection.settimeout(10)
|
||||
payload = json.loads(self.rfile.read(length))
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError('Expected JSON object')
|
||||
receipt = deliver(payload)
|
||||
except (ValueError, TypeError) as exc:
|
||||
return self.respond(400, {'error': str(exc)})
|
||||
except (OSError, RuntimeError, subprocess.TimeoutExpired) as exc:
|
||||
print(json.dumps({'event': 'delivery_failed', 'error': str(exc)}), flush=True)
|
||||
return self.respond(502, {'error': 'Althing delivery failed; inspect service journal'})
|
||||
self.respond(200, {'delivered': True, 'receipt': receipt})
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
ThreadingHTTPServer((os.environ['BESZEL_BIND_HOST'], int(os.environ.get('BESZEL_BIND_PORT', '8096'))), Handler).serve_forever()
|
||||
@@ -1,30 +0,0 @@
|
||||
import importlib.util
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
spec=importlib.util.spec_from_file_location('bridge',Path(__file__).with_name('bridge.py'))
|
||||
bridge=importlib.util.module_from_spec(spec);spec.loader.exec_module(bridge)
|
||||
|
||||
class DeliveryTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.env=patch.dict(os.environ,POSTBOX='/bin/postbox',BESZEL_ALERT_RECIPIENT='infra-ops')
|
||||
self.env.start();self.addCleanup(self.env.stop)
|
||||
def test_posts_body_as_stdin_and_returns_receipt(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id": 42}','')) as run:
|
||||
self.assertEqual(bridge.deliver({'title':'Disk 85%','message':'tank is full'})['message_id'],42)
|
||||
args,kw=run.call_args
|
||||
self.assertEqual(args[0],['/bin/postbox','--json','send','--to','infra-ops','--subject','[Beszel] Disk 85%'])
|
||||
self.assertIn('tank is full',kw['input'])
|
||||
def test_delivery_failure_is_not_success(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],2,'','post office unavailable')):
|
||||
with self.assertRaises(RuntimeError):bridge.deliver({'message':'disk full'})
|
||||
def test_rejects_empty_message_without_sending(self):
|
||||
with patch.object(bridge.subprocess,'run') as run:
|
||||
with self.assertRaises(ValueError):bridge.deliver({'message':''})
|
||||
run.assert_not_called()
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user