Wire Beszel fleet filesystems, GPU telemetry, dashboard and alerts

2026-09-10 08:35:28 -07:00
parent 20bbb95113
commit eb75713c1b
18 changed files with 438 additions and 84 deletions
@@ -0,0 +1,36 @@
+# Beszel to althing
+
+`beszel-althing.service` runs on nh3-dev as lkraven, listening at
+`10.100.10.50:8096`. Beszel sends Shoutrrr generic JSON to `/beszel`; the
+bridge invokes the supported `postbox send` command with the body on stdin.
+It uses the established infra-ops automation identity and sends to infra-ops.
+
+Deploy from the repository root:
+
+```sh
+scripts/elway infra-ops@10.100.10.50 --playbook playbooks/beszel-althing.yaml
+```
+
+The service accepts requests only from ana-docker and local diagnostic
+addresses. There is no public listener or new althing handle. It reports
+success only after postbox returns a delivery receipt. Failures return HTTP
+502 and appear in the system journal; there is no hidden retry queue. A
+post-office outage can therefore lose an alert, and needs independent health
+monitoring if guaranteed delivery during such outages becomes a requirement.
+
+```sh
+systemctl status beszel-althing
+sudo -n journalctl -u beszel-althing --since '1 hour ago'
+curl -fsS http://10.100.10.50:8096/healthz
+```
+
+`/healthz` checks the bridge process, not the downstream inbox. End-to-end
+verification requires a real Beszel threshold transition plus its althing
+receipt. The first verified alert is recorded in `stacks/beszel/README.md`.
+
+To reroute later, change `BESZEL_ALERT_RECIPIENT` in the canonical unit to
+`miranda`, deploy, and trigger another end-to-end test. Leave `ALTHING_HANDLE`
+as infra-ops so the sender remains identifiable as infrastructure automation.
+The operator explicitly chose infra-ops for now.
+
+Run `python3 -m unittest discover -s services/beszel-althing -p 'test_*.py'`.