feat(alerts): generalize the althing bridge, wire Kuma to it, retire chamber
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.
Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.
Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.
⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.
Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.
parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.
KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.
⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.
End-to-end, both shapes, read back from the inbox:
[Uptime Kuma] Homepage is DOWN + target + board link
[Uptime Kuma] althing (infra-ops) Testing (no fabricated subject)
[Beszel] BRIDGE RENAME CHECK + hub footer, unchanged
ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
This commit is contained in:
+330
-250
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,33 @@
|
||||
steps:
|
||||
# The service was `beszel-althing` until 2026-09-21, when it gained a /kuma
|
||||
# route and the Beszel-specific name became a lie. Stop and remove the old
|
||||
# unit FIRST: both bind 10.100.10.50:8096, so they cannot run side by side.
|
||||
- name: Stop and remove the superseded beszel-althing unit
|
||||
sudo: true
|
||||
shell: >-
|
||||
systemctl disable --now beszel-althing.service 2>/dev/null || true;
|
||||
rm -f /etc/systemd/system/beszel-althing.service;
|
||||
rm -rf /opt/beszel-althing;
|
||||
systemctl daemon-reload
|
||||
- name: Install fleet alert bridge
|
||||
sudo: true
|
||||
upload:
|
||||
src: services/althing-alert-bridge/bridge.py
|
||||
dest: /opt/althing-alert-bridge/bridge.py
|
||||
mode: '0644'
|
||||
- name: Install fleet alert bridge unit
|
||||
sudo: true
|
||||
upload:
|
||||
src: services/althing-alert-bridge/althing-alert-bridge.service
|
||||
dest: /etc/systemd/system/althing-alert-bridge.service
|
||||
mode: '0644'
|
||||
- name: Start fleet alert bridge
|
||||
sudo: true
|
||||
shell: systemctl daemon-reload && systemctl enable althing-alert-bridge.service && systemctl restart althing-alert-bridge.service
|
||||
verify:
|
||||
- name: Verify bridge process and BOTH routes are advertised
|
||||
shell: >-
|
||||
systemctl is-active althing-alert-bridge.service &&
|
||||
curl --retry 5 --retry-connrefused --retry-delay 1 -fsS http://10.100.10.50:8096/healthz | grep -q '"/kuma"' &&
|
||||
curl -fsS http://10.100.10.50:8096/healthz | grep -q '"/beszel"'
|
||||
changed_when: 'false'
|
||||
@@ -1,20 +0,0 @@
|
||||
steps:
|
||||
- name: Install Beszel alert bridge
|
||||
sudo: true
|
||||
upload:
|
||||
src: services/beszel-althing/bridge.py
|
||||
dest: /opt/beszel-althing/bridge.py
|
||||
mode: '0644'
|
||||
- name: Install Beszel alert bridge unit
|
||||
sudo: true
|
||||
upload:
|
||||
src: services/beszel-althing/beszel-althing.service
|
||||
dest: /etc/systemd/system/beszel-althing.service
|
||||
mode: '0644'
|
||||
- name: Start Beszel alert bridge
|
||||
sudo: true
|
||||
shell: systemctl daemon-reload && systemctl enable beszel-althing.service && systemctl restart beszel-althing.service
|
||||
verify:
|
||||
- name: Verify bridge process
|
||||
shell: systemctl is-active beszel-althing.service && curl --retry 5 --retry-connrefused --retry-delay 1 -fsS http://10.100.10.50:8096/healthz
|
||||
changed_when: 'false'
|
||||
@@ -1,147 +0,0 @@
|
||||
# Deploy althing-chamber (https://gitea.phasefinal.com/vh/althing) to a
|
||||
# Docker host following the PFI /opt/docker/ convention (ana-docker by
|
||||
# default, but the playbook works against any host with Docker in place).
|
||||
#
|
||||
# Brings up FOUR compose services:
|
||||
# althing-chamber — FastAPI/HTMX web UI, port 7881 host → 8000 container
|
||||
# althing-forseti — moderator daemon, no port
|
||||
# althing-agent-runner — Phase 2 worldtree-driver dispatcher, no port
|
||||
# althing-valkey — Phase 3.1 valkey 8 alpine, pub/sub bridge (internal only)
|
||||
#
|
||||
# The three althing services use the same image (built from vh/althing);
|
||||
# valkey is a stock upstream image. SQLite-backed state shares between the
|
||||
# three althing services via the bind-mount under /opt/docker/conf/althing-chamber/data;
|
||||
# streaming events ride pub/sub on the docker default network via valkey.
|
||||
#
|
||||
# Idempotent: rerunning is safe. Creates-gates skip work that's already
|
||||
# done; `docker compose up -d` is itself idempotent (no restart unless
|
||||
# compose content or env changed).
|
||||
#
|
||||
# Usage:
|
||||
# scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml
|
||||
# scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml --var ref=v0.1.0
|
||||
#
|
||||
# Prereqs on the target host:
|
||||
# - Docker + docker compose plugin
|
||||
# - Target user (lkraven) has git SSH access to gitea.phasefinal.com
|
||||
# — either SSH key authorized in gitea, or the repo is HTTPS-reachable
|
||||
# if you swap `repo_url` below.
|
||||
# - Target user is in the `docker` group.
|
||||
|
||||
vars:
|
||||
repo_url: git@gitea.phasefinal.com:vh/althing.git
|
||||
ref: main
|
||||
build_dir: /opt/docker/build/althing-chamber
|
||||
image_tag: althing-chamber:local
|
||||
compose_dir: /opt/docker/compose/althing-chamber
|
||||
data_dir: /opt/docker/conf/althing-chamber/data
|
||||
host_port: "7881"
|
||||
|
||||
steps:
|
||||
# ── host-side directory prep ─────────────────────────────────────────
|
||||
- name: Ensure /opt/docker/build parent exists
|
||||
shell: mkdir -p /opt/docker/build
|
||||
sudo: true
|
||||
creates: /opt/docker/build
|
||||
|
||||
- name: Chown /opt/docker/build to lkraven (only if mkdir'd by root above)
|
||||
shell: chown lkraven:lkraven /opt/docker/build
|
||||
sudo: true
|
||||
when: '[ "$(stat -c %U /opt/docker/build)" != lkraven ]'
|
||||
|
||||
# ── fetch / sync source ─────────────────────────────────────────────
|
||||
- name: Clone althing repo if absent
|
||||
# Auto-accept the first-run host key so the playbook doesn't hang
|
||||
# prompting for yes/no.
|
||||
shell: GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git clone {{ repo_url }} {{ build_dir }}
|
||||
creates: "{{ build_dir }}/.git"
|
||||
|
||||
- name: Fetch from origin
|
||||
shell: cd {{ build_dir }} && git fetch --quiet origin
|
||||
|
||||
- name: Reset working tree to {{ ref }}
|
||||
# Accept either a branch name (resolves via origin/<ref>) or a
|
||||
# full/short SHA (resolves directly). CI passes the triggering
|
||||
# commit SHA via --var ref=${{ github.sha }}; manual runs pass
|
||||
# branch names like main / v0.1.0.
|
||||
shell: |
|
||||
cd {{ build_dir }}
|
||||
if sha=$(git rev-parse --verify --quiet "origin/{{ ref }}^{commit}"); then :;
|
||||
elif sha=$(git rev-parse --verify --quiet "{{ ref }}^{commit}"); then :;
|
||||
else echo "elway: ref not found: {{ ref }}" >&2; exit 1; fi
|
||||
git reset --hard "$sha"
|
||||
# Report ok (no-change) when the tree was already at the requested
|
||||
# ref — saves a noisy CHANGED status line on no-op reruns.
|
||||
changed_when: '[ "$(cd {{ build_dir }} && git rev-parse HEAD)" != "$(cd {{ build_dir }} && (git rev-parse --verify --quiet "origin/{{ ref }}^{commit}" || git rev-parse --verify --quiet "{{ ref }}^{commit}"))" ]'
|
||||
|
||||
# ── image build ─────────────────────────────────────────────────────
|
||||
- name: Build image {{ image_tag }}
|
||||
shell: cd {{ build_dir }} && docker build -t {{ image_tag }} .
|
||||
# Docker build reuses layer cache and is fast on reruns, but it
|
||||
# always runs — we can't cheaply know up-front whether anything
|
||||
# downstream has changed. Leave it in the always-run lane; Docker
|
||||
# itself handles the no-op efficiently.
|
||||
|
||||
# ── compose + data dirs ─────────────────────────────────────────────
|
||||
- name: Ensure compose dir exists
|
||||
shell: mkdir -p {{ compose_dir }}
|
||||
creates: "{{ compose_dir }}"
|
||||
|
||||
- name: Ensure data dir exists
|
||||
# Single bind-mount shared between chamber + forseti. Created as
|
||||
# lkraven (uid 1000 on these hosts), matching the container's `app`
|
||||
# user — no chown dance needed.
|
||||
shell: mkdir -p {{ data_dir }}
|
||||
creates: "{{ data_dir }}"
|
||||
|
||||
# ── deploy compose files ────────────────────────────────────────────
|
||||
- name: Upload compose.yaml
|
||||
upload:
|
||||
src: stacks/althing-chamber/compose.yaml
|
||||
dest: "{{ compose_dir }}/compose.yaml"
|
||||
mode: "0644"
|
||||
|
||||
- name: Seed .env from template (only if absent)
|
||||
upload:
|
||||
src: stacks/althing-chamber/.env.example
|
||||
dest: "{{ compose_dir }}/.env"
|
||||
mode: "0644"
|
||||
when: "[ ! -f {{ compose_dir }}/.env ]"
|
||||
|
||||
# ── bring up + wait for ready ───────────────────────────────────────
|
||||
- name: docker compose up -d
|
||||
shell: cd {{ compose_dir }} && docker compose up -d
|
||||
|
||||
- name: Wait for chamber /health to respond
|
||||
# Chamber's healthcheck is internal (inside the container's network);
|
||||
# this host-side poll confirms the published port is reachable too.
|
||||
# Short retry loop — docker compose up returns before the FastAPI
|
||||
# app finishes booting.
|
||||
shell: |
|
||||
for i in $(seq 1 30); do
|
||||
curl -sf -o /dev/null http://localhost:{{ host_port }}/health && exit 0
|
||||
sleep 2
|
||||
done
|
||||
exit 1
|
||||
changed_when: "false"
|
||||
|
||||
verify:
|
||||
- name: chamber /health returns 200
|
||||
shell: curl -sf -o /dev/null http://localhost:{{ host_port }}/health
|
||||
changed_when: "false"
|
||||
|
||||
- name: chamber container running
|
||||
shell: docker ps --filter name=^/althing-chamber$ --format '{{.Status}}' | grep -q '^Up'
|
||||
changed_when: "false"
|
||||
|
||||
- name: forseti container running
|
||||
shell: docker ps --filter name=^/althing-forseti$ --format '{{.Status}}' | grep -q '^Up'
|
||||
changed_when: "false"
|
||||
|
||||
- name: agent-runner container running
|
||||
shell: docker ps --filter name=^/althing-agent-runner$ --format '{{.Status}}' | grep -q '^Up'
|
||||
changed_when: "false"
|
||||
|
||||
- name: valkey container running + healthy
|
||||
shell: docker ps --filter name=^/althing-valkey$ --format '{{.Status}}' | grep -q 'healthy'
|
||||
changed_when: "false"
|
||||
@@ -73,6 +73,18 @@ MONITOR_DEFAULTS = {
|
||||
}
|
||||
|
||||
|
||||
# A channel registered with isDefault + applyExisting attaches to monitors that
|
||||
# ALREADY exist, not merely to ones created afterwards -- without it, seeding
|
||||
# monitors before channels silently leaves the existing board unwired.
|
||||
NOTIFICATION_DEFAULTS = {
|
||||
"type": "webhook",
|
||||
"isDefault": True,
|
||||
"applyExisting": True,
|
||||
"webhookContentType": "json",
|
||||
"active": True,
|
||||
}
|
||||
|
||||
|
||||
def vault_get(key: str) -> dict:
|
||||
"""{'username','password'} from the vault. Loud on failure -- a silent
|
||||
empty credential would show up as a confusing auth error much later."""
|
||||
@@ -106,10 +118,16 @@ class Kuma:
|
||||
# updating it. Capture the push; never trust the ack for data.
|
||||
self._monitor_list: dict | None = None
|
||||
|
||||
self._notification_list: list | None = None
|
||||
|
||||
@self.sio.on("monitorList")
|
||||
def _on_monitor_list(data):
|
||||
self._monitor_list = data or {}
|
||||
|
||||
@self.sio.on("notificationList")
|
||||
def _on_notification_list(data):
|
||||
self._notification_list = data or []
|
||||
|
||||
def __enter__(self):
|
||||
self.sio.connect(self.url, transports=["websocket"], wait_timeout=20)
|
||||
self._connected = True
|
||||
@@ -157,6 +175,29 @@ class Kuma:
|
||||
def delete(self, monitor_id: int):
|
||||
return self.call("deleteMonitor", monitor_id, False)
|
||||
|
||||
def notifications(self, timeout: int = 10) -> list:
|
||||
"""Registered notification channels.
|
||||
|
||||
⚠ DIFFERENT PUSH RULE FROM monitorList, and the difference bit once.
|
||||
There is no `getNotificationList` event to ask with -- the server pushes
|
||||
`notificationList` ONCE, at login, and again after any change. So this
|
||||
must NOT clear the captured value before waiting: doing that discards
|
||||
the only copy it will ever be sent and then blocks for the full timeout
|
||||
before reporting an empty list, which reads exactly like "no channels
|
||||
are configured" and is a lie.
|
||||
"""
|
||||
deadline = time.time() + timeout
|
||||
while self._notification_list is None and time.time() < deadline:
|
||||
self.sio.sleep(0.2)
|
||||
return self._notification_list or []
|
||||
|
||||
def add_notification(self, notification: dict):
|
||||
"""Register (or update) a channel. notificationID None => create."""
|
||||
return self.call("addNotification", notification, None)
|
||||
|
||||
def test_notification(self, notification: dict):
|
||||
return self.call("testNotification", notification)
|
||||
|
||||
def add(self, monitor: dict):
|
||||
return self.call("add", {**MONITOR_DEFAULTS, **monitor})
|
||||
|
||||
@@ -204,8 +245,20 @@ def cmd_seed(args):
|
||||
same service), and re-running a seed must never fork the board."""
|
||||
spec = yaml.safe_load(open(args.file))
|
||||
wanted = spec["monitors"] if isinstance(spec, dict) else spec
|
||||
channels = spec.get("notifications", []) if isinstance(spec, dict) else []
|
||||
k = connected_and_logged_in()
|
||||
try:
|
||||
# Channels FIRST: a monitor created while a default channel exists
|
||||
# inherits it. Seeding monitors first and alerting second is how you get
|
||||
# a board that detects and tells nobody -- the exact gap this fleet's
|
||||
# service layer was built to close.
|
||||
have = {n.get("name") for n in k.notifications()}
|
||||
for ch in channels:
|
||||
if ch["name"] in have:
|
||||
print(f" = {ch['name']} (channel)")
|
||||
continue
|
||||
k.add_notification({**NOTIFICATION_DEFAULTS, **ch})
|
||||
print(f" + {ch['name']} (channel)")
|
||||
existing = {m.get("name"): m for m in k.monitors().values()}
|
||||
added = updated = 0
|
||||
for m in wanted:
|
||||
@@ -223,6 +276,21 @@ def cmd_seed(args):
|
||||
k.__exit__()
|
||||
|
||||
|
||||
def cmd_channels(args):
|
||||
k = connected_and_logged_in()
|
||||
try:
|
||||
ns = k.notifications()
|
||||
if not ns:
|
||||
print("no notification channels — THE BOARD ALERTS NOBODY")
|
||||
return
|
||||
for n in ns:
|
||||
cfg = json.loads(n.get("config") or "{}")
|
||||
print(" %-26s %-9s default=%-5s %s" % (
|
||||
n.get("name"), cfg.get("type"), n.get("isDefault"), cfg.get("webhookURL", "")))
|
||||
finally:
|
||||
k.__exit__()
|
||||
|
||||
|
||||
def cmd_dedupe(args):
|
||||
"""Keep the LOWEST id per name, delete the rest. Needed because the first
|
||||
version of this client read the getMonitorList ack instead of the pushed
|
||||
@@ -274,6 +342,7 @@ def main():
|
||||
a.add_argument("--port", type=int)
|
||||
a.set_defaults(fn=cmd_add)
|
||||
sub.add_parser("dedupe").set_defaults(fn=cmd_dedupe)
|
||||
sub.add_parser("channels").set_defaults(fn=cmd_channels)
|
||||
args = p.parse_args()
|
||||
try:
|
||||
args.fn(args)
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
# Fleet alert bridge — monitoring tools into the althing inbox
|
||||
|
||||
`althing-alert-bridge.service` runs on nh3-dev as lkraven, listening at
|
||||
`10.100.10.50:8096`. It turns an HTTP alert into a `postbox send` to the
|
||||
**infra-ops** inbox, using the established automation identity.
|
||||
|
||||
| route | sender | payload |
|
||||
|---|---|---|
|
||||
| `POST /beszel` | Beszel (Shoutrrr generic JSON) | `{title, message}` |
|
||||
| `POST /kuma` | Uptime Kuma (webhook, JSON) | `{heartbeat, monitor, msg}` |
|
||||
|
||||
```sh
|
||||
scripts/elway infra-ops@10.100.10.50 --playbook playbooks/althing-alert-bridge.yaml
|
||||
systemctl status althing-alert-bridge
|
||||
curl -fsS http://10.100.10.50:8096/healthz # lists the routes it serves
|
||||
```
|
||||
|
||||
## Why one bridge with a route registry
|
||||
|
||||
It was `beszel-althing` until 2026-09-21, with the `[Beszel]` subject prefix and
|
||||
the Beszel hub footer hardcoded. When Uptime Kuma was rebuilt as the fleet's
|
||||
service layer it needed the same path — and routing it through unchanged would
|
||||
have delivered Kuma outages labelled `[Beszel]`, pointing the reader at the
|
||||
wrong dashboard. **An alert that lies about its own source is worse than no
|
||||
alert.**
|
||||
|
||||
Generalising cost a dict. A sibling service would have cost a second unit, a
|
||||
second port, and a second thing to notice had died. The name went with it: a
|
||||
service called `beszel-althing` that also carries Kuma alerts is exactly the
|
||||
invisible coupling that makes a future session look in the wrong place.
|
||||
|
||||
⚠ **`/beszel` is frozen.** Its prefix, footer and default title must stay
|
||||
byte-identical — that path was verified end-to-end in production (2026-09-10,
|
||||
thread `01M25Z0WFDJM92GPTJQF769HJ7`) and a refactor is not allowed to quietly
|
||||
change what it emits. `deliver()` still defaults to the Beszel route so the
|
||||
original three tests exercise it unchanged, and a test asserts the Kuma footer
|
||||
never leaks into a Beszel body or vice versa.
|
||||
|
||||
## Payload shapes are handled here, not in the sending tool
|
||||
|
||||
Uptime Kuma can render a custom webhook body, which would have let the bridge
|
||||
stay dumb. It is done here instead, because Kuma's notification config lives in
|
||||
its own database — and that database was destroyed and rebuilt from scratch on
|
||||
2026-09-21. Anything that lives only in a tool's DB is lost on the next rebuild.
|
||||
Format knowledge belongs in git, next to a test.
|
||||
|
||||
`parse_kuma` also handles the **monitorless** case: `testNotification` and
|
||||
certificate-expiry alerts arrive with no monitor and no heartbeat, and
|
||||
fabricating `unknown monitor is ?` from those makes a real alert read like a
|
||||
bug — observed live, then fixed and pinned by a test.
|
||||
|
||||
## Operational notes
|
||||
|
||||
- Source-allowlisted by IP (`ALERT_ALLOWED_SOURCES`). `10.250.50.70` is
|
||||
ana-docker, which now runs **both** the Beszel hub and Uptime Kuma, so one
|
||||
entry covers both senders.
|
||||
- Delivery reports success only after `postbox` returns a receipt. Failures
|
||||
return HTTP 502 and land in the journal; **there is no retry queue**, so a
|
||||
post-office outage can lose an alert.
|
||||
- `/healthz` checks the bridge process, not the downstream inbox. End-to-end
|
||||
verification means sending a real POST and reading the thread back — both
|
||||
routes were verified that way on 2026-09-21 (`postbox thread` does not
|
||||
consume the inbox).
|
||||
- Legacy `BESZEL_*` env names still resolve, so a half-finished deploy starts
|
||||
instead of crash-looping. New deployments use `ALERT_*`.
|
||||
- To reroute, change `ALERT_RECIPIENT` in the unit and redeploy. Leave
|
||||
`ALTHING_HANDLE` as infra-ops so the sender stays identifiable as automation.
|
||||
|
||||
```sh
|
||||
python3 -m unittest discover -s services/althing-alert-bridge -p 'test_*.py' # 11 tests
|
||||
```
|
||||
@@ -0,0 +1,27 @@
|
||||
[Unit]
|
||||
Description=Fleet alert bridge — Beszel + Uptime Kuma alerts into the althing infra-ops inbox
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
User=lkraven
|
||||
Group=lkraven
|
||||
Environment=PYTHONUNBUFFERED=1
|
||||
Environment=ALTHING_HANDLE=infra-ops
|
||||
Environment=ALTHING_POST_OFFICE=http://10.100.50.40:8390
|
||||
Environment=POSTBOX=/home/lkraven/.local/bin/postbox
|
||||
Environment=ALERT_RECIPIENT=infra-ops
|
||||
Environment=ALERT_BIND_HOST=10.100.10.50
|
||||
# 10.250.50.70 is ana-docker, which now runs BOTH the Beszel hub and Uptime
|
||||
# Kuma, so one entry covers both senders. 10.100.10.50/127.0.0.1 are local
|
||||
# diagnostics from nh3-dev itself.
|
||||
Environment=ALERT_ALLOWED_SOURCES=10.250.50.70,10.100.10.50,127.0.0.1
|
||||
ExecStart=/usr/bin/python3 /opt/althing-alert-bridge/bridge.py
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
PrivateTmp=true
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,162 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fleet alert bridge — HTTP alerts from monitoring tools into the althing inbox.
|
||||
|
||||
One listener, one route per SOURCE. Each source declares its own subject prefix,
|
||||
its own footer link, and how to read its payload, because the tools do not agree
|
||||
on a shape and the reader needs to know which tool is talking.
|
||||
|
||||
POST /beszel <- Beszel, via Shoutrrr generic JSON: {title, message}
|
||||
POST /kuma <- Uptime Kuma, via its webhook: {heartbeat, monitor, msg}
|
||||
|
||||
WHY A REGISTRY RATHER THAN A SECOND SERVICE. The prefix and footer were
|
||||
hardcoded when Beszel was the only caller, so a second tool routed through it
|
||||
would have arrived labelled "[Beszel]" -- an alert that lies about its own
|
||||
source is worse than no alert, because it sends you to the wrong dashboard.
|
||||
Generalising costs a dict; a sibling service costs a second unit, a second port
|
||||
and a second thing to notice has died.
|
||||
|
||||
⚠ `/beszel` IS FROZEN. Its prefix, footer and default title must stay byte-
|
||||
identical -- Beszel's delivery path has been verified end-to-end in production
|
||||
(2026-09-10, thread 01M25Z0WFDJM92GPTJQF769HJ7) and this refactor is not
|
||||
allowed to quietly change what that path emits. `deliver()` still defaults to
|
||||
the Beszel route precisely so the original tests exercise it unchanged.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
|
||||
def _env(*names, default=None):
|
||||
"""First env var that is set. Accepts the legacy BESZEL_* names so a
|
||||
half-finished deploy (new bridge, old unit) still starts instead of
|
||||
crash-looping on a KeyError."""
|
||||
for n in names:
|
||||
if os.environ.get(n):
|
||||
return os.environ[n]
|
||||
if default is None:
|
||||
raise KeyError(' / '.join(names))
|
||||
return default
|
||||
|
||||
|
||||
def parse_generic(payload):
|
||||
"""Shoutrrr generic JSON — what Beszel sends. {title, message}."""
|
||||
return payload.get('title'), payload.get('message')
|
||||
|
||||
|
||||
def parse_kuma(payload):
|
||||
"""Uptime Kuma's webhook — {heartbeat, monitor, msg}.
|
||||
|
||||
Kuma's own `msg` already reads like a sentence ("[Homepage] [🔴 Down]
|
||||
connect ECONNREFUSED"), so it is the body. The title is built from the
|
||||
monitor name and the heartbeat status so the SUBJECT alone says which
|
||||
service and which direction -- that is what you see in an inbox list
|
||||
without opening anything.
|
||||
"""
|
||||
monitor = payload.get('monitor') or {}
|
||||
heartbeat = payload.get('heartbeat') or {}
|
||||
body = payload.get('msg') or ''
|
||||
name = monitor.get('name')
|
||||
# Kuma status: 0 down, 1 up, 2 pending, 3 maintenance.
|
||||
state = {0: 'DOWN', 1: 'UP', 2: 'PENDING', 3: 'MAINTENANCE'}.get(heartbeat.get('status'))
|
||||
|
||||
if name and state:
|
||||
title = f'{name} is {state}'
|
||||
else:
|
||||
# NOT every Kuma notification is a monitor transition: `testNotification`
|
||||
# and certificate-expiry alerts arrive with no monitor and no heartbeat.
|
||||
# Fabricating "unknown monitor is ?" from those makes a real alert read
|
||||
# like a bug; the message itself is the better subject.
|
||||
first = body.strip().splitlines()[0] if body.strip() else ''
|
||||
title = (first[:120] or 'notification')
|
||||
detail = heartbeat.get('msg')
|
||||
if detail and detail not in body:
|
||||
body = f'{body}\n\n{detail}'.strip()
|
||||
url = monitor.get('url')
|
||||
if url:
|
||||
body = f'{body}\n\nTarget: {url}'.strip()
|
||||
return title, body
|
||||
|
||||
|
||||
SOURCES = {
|
||||
# FROZEN — see the module docstring.
|
||||
'/beszel': {
|
||||
'prefix': '[Beszel] ',
|
||||
'default_title': 'Beszel fleet alert',
|
||||
'footer': '\n\nHub: http://10.250.50.70:8090\n',
|
||||
'parse': parse_generic,
|
||||
},
|
||||
'/kuma': {
|
||||
'prefix': '[Uptime Kuma] ',
|
||||
'default_title': 'Uptime Kuma alert',
|
||||
'footer': '\n\nBoard: http://10.250.50.70:3001\n',
|
||||
'parse': parse_kuma,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def deliver(payload, route='/beszel'):
|
||||
source = SOURCES[route]
|
||||
title, message = source['parse'](payload)
|
||||
if title is None:
|
||||
title = source['default_title']
|
||||
if not isinstance(title, str) or not isinstance(message, str) or not message.strip():
|
||||
raise ValueError('Expected a nonempty message and string title')
|
||||
result = subprocess.run(
|
||||
[_env('POSTBOX'), '--json', 'send',
|
||||
'--to', _env('ALERT_RECIPIENT', 'BESZEL_ALERT_RECIPIENT'),
|
||||
'--subject', source['prefix'] + title],
|
||||
input=message + source['footer'],
|
||||
text=True, capture_output=True, timeout=25,
|
||||
)
|
||||
if result.returncode:
|
||||
raise RuntimeError('postbox delivery failed: ' + result.stderr.strip())
|
||||
receipt = json.loads(result.stdout)
|
||||
print(json.dumps({'event': 'delivered', 'route': route, 'title': title, 'receipt': receipt}), flush=True)
|
||||
return receipt
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def respond(self, status, body):
|
||||
data = json.dumps(body).encode()
|
||||
self.send_response(status)
|
||||
self.send_header('Content-Type', 'application/json')
|
||||
self.send_header('Content-Length', str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
def do_GET(self):
|
||||
self.respond(200 if self.path == '/healthz' else 404,
|
||||
{'service': 'althing-alert-bridge',
|
||||
'routes': sorted(SOURCES),
|
||||
'delivery': 'verified per POST'})
|
||||
|
||||
def do_POST(self):
|
||||
if self.path not in SOURCES:
|
||||
return self.respond(404, {'error': 'Unknown route', 'routes': sorted(SOURCES)})
|
||||
allowed = _env('ALERT_ALLOWED_SOURCES', 'BESZEL_ALLOWED_SOURCES').split(',')
|
||||
if self.client_address[0] not in allowed:
|
||||
return self.respond(403, {'error': 'Source not allowed'})
|
||||
try:
|
||||
length = int(self.headers.get('Content-Length', '0'))
|
||||
if not 0 < length <= 65536:
|
||||
raise ValueError('Invalid body size')
|
||||
self.connection.settimeout(10)
|
||||
payload = json.loads(self.rfile.read(length))
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError('Expected JSON object')
|
||||
receipt = deliver(payload, self.path)
|
||||
except (ValueError, TypeError) as exc:
|
||||
return self.respond(400, {'error': str(exc)})
|
||||
except (OSError, RuntimeError, subprocess.TimeoutExpired) as exc:
|
||||
print(json.dumps({'event': 'delivery_failed', 'route': self.path, 'error': str(exc)}), flush=True)
|
||||
return self.respond(502, {'error': 'Althing delivery failed; inspect service journal'})
|
||||
self.respond(200, {'delivered': True, 'receipt': receipt})
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
ThreadingHTTPServer(
|
||||
(_env('ALERT_BIND_HOST', 'BESZEL_BIND_HOST'),
|
||||
int(_env('ALERT_BIND_PORT', 'BESZEL_BIND_PORT', default='8096'))),
|
||||
Handler,
|
||||
).serve_forever()
|
||||
@@ -0,0 +1,109 @@
|
||||
import importlib.util
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
spec=importlib.util.spec_from_file_location('bridge',Path(__file__).with_name('bridge.py'))
|
||||
bridge=importlib.util.module_from_spec(spec);spec.loader.exec_module(bridge)
|
||||
|
||||
class DeliveryTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.env=patch.dict(os.environ,POSTBOX='/bin/postbox',BESZEL_ALERT_RECIPIENT='infra-ops')
|
||||
self.env.start();self.addCleanup(self.env.stop)
|
||||
def test_posts_body_as_stdin_and_returns_receipt(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id": 42}','')) as run:
|
||||
self.assertEqual(bridge.deliver({'title':'Disk 85%','message':'tank is full'})['message_id'],42)
|
||||
args,kw=run.call_args
|
||||
self.assertEqual(args[0],['/bin/postbox','--json','send','--to','infra-ops','--subject','[Beszel] Disk 85%'])
|
||||
self.assertIn('tank is full',kw['input'])
|
||||
def test_delivery_failure_is_not_success(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],2,'','post office unavailable')):
|
||||
with self.assertRaises(RuntimeError):bridge.deliver({'message':'disk full'})
|
||||
def test_rejects_empty_message_without_sending(self):
|
||||
with patch.object(bridge.subprocess,'run') as run:
|
||||
with self.assertRaises(ValueError):bridge.deliver({'message':''})
|
||||
run.assert_not_called()
|
||||
|
||||
class KumaRouteTests(unittest.TestCase):
|
||||
"""The /kuma route. The point of generalising was that an alert must not
|
||||
lie about its own source -- a Kuma outage labelled [Beszel] sends you to
|
||||
the wrong dashboard, which is worse than no alert."""
|
||||
|
||||
def setUp(self):
|
||||
self.env=patch.dict(os.environ,POSTBOX='/bin/postbox',ALERT_RECIPIENT='infra-ops')
|
||||
self.env.start();self.addCleanup(self.env.stop)
|
||||
|
||||
DOWN={'monitor':{'name':'Homepage','url':'http://10.0.50.45:5100/'},
|
||||
'heartbeat':{'status':0,'msg':'connect ECONNREFUSED'},
|
||||
'msg':'[Homepage] [Down] connect ECONNREFUSED'}
|
||||
|
||||
def test_subject_names_the_service_and_the_direction(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":7}','')) as run:
|
||||
bridge.deliver(self.DOWN,'/kuma')
|
||||
argv=run.call_args[0][0]
|
||||
self.assertIn('[Uptime Kuma] Homepage is DOWN',argv)
|
||||
self.assertNotIn('[Beszel] Homepage is DOWN',argv)
|
||||
|
||||
def test_up_transition_reads_as_up(self):
|
||||
up={'monitor':{'name':'Gitea'},'heartbeat':{'status':1,'msg':'200 - OK'},'msg':'[Gitea] [Up] 200 - OK'}
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":8}','')) as run:
|
||||
bridge.deliver(up,'/kuma')
|
||||
self.assertIn('[Uptime Kuma] Gitea is UP',run.call_args[0][0])
|
||||
|
||||
def test_body_carries_the_target_and_the_kuma_board_not_the_beszel_hub(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":9}','')) as run:
|
||||
bridge.deliver(self.DOWN,'/kuma')
|
||||
body=run.call_args[1]['input']
|
||||
self.assertIn('http://10.0.50.45:5100/',body)
|
||||
self.assertIn('Board: http://10.250.50.70:3001',body)
|
||||
self.assertNotIn('Hub: http://10.250.50.70:8090',body) # the Beszel footer must not leak
|
||||
|
||||
def test_beszel_footer_is_unchanged_by_the_refactor(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":10}','')) as run:
|
||||
bridge.deliver({'title':'Disk 85%','message':'tank is full'},'/beszel')
|
||||
body=run.call_args[1]['input']
|
||||
self.assertIn('Hub: http://10.250.50.70:8090',body)
|
||||
self.assertNotIn('Board:',body)
|
||||
|
||||
def test_a_nameless_alert_prefers_its_message_over_a_fabricated_subject(self):
|
||||
"""Superseded an earlier assertion that this should read "unknown
|
||||
monitor is DOWN". It should not: a real monitor always carries a name,
|
||||
so a nameless payload is degenerate, and "something broke" tells the
|
||||
reader more than a placeholder that looks like a bug."""
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":11}','')) as run:
|
||||
argv=None
|
||||
bridge.deliver({'heartbeat':{'status':0},'msg':'something broke'},'/kuma')
|
||||
argv=run.call_args[0][0]
|
||||
self.assertIn('[Uptime Kuma] something broke',argv)
|
||||
self.assertNotIn('unknown monitor',' '.join(argv))
|
||||
|
||||
def test_empty_kuma_message_is_refused_without_sending(self):
|
||||
with patch.object(bridge.subprocess,'run') as run:
|
||||
with self.assertRaises(ValueError):
|
||||
bridge.deliver({'monitor':{'name':'X'},'heartbeat':{'status':0},'msg':''},'/kuma')
|
||||
run.assert_not_called()
|
||||
|
||||
def test_a_monitorless_notification_uses_its_own_message_as_the_subject(self):
|
||||
"""testNotification and cert-expiry arrive with no monitor and no
|
||||
heartbeat. Fabricating "unknown monitor is ?" from those makes a real
|
||||
alert read like a bug -- observed live on 2026-09-21."""
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":13}','')) as run:
|
||||
bridge.deliver({'msg':'althing (infra-ops) Testing'},'/kuma')
|
||||
argv=run.call_args[0][0]
|
||||
self.assertIn('[Uptime Kuma] althing (infra-ops) Testing',argv)
|
||||
self.assertNotIn('unknown monitor',' '.join(argv))
|
||||
self.assertNotIn('is ?',' '.join(argv))
|
||||
|
||||
def test_legacy_beszel_env_names_still_resolve(self):
|
||||
"""A half-finished deploy (new bridge, old unit) must start, not crash."""
|
||||
with patch.dict(os.environ,{'ALERT_RECIPIENT':'','BESZEL_ALERT_RECIPIENT':'infra-ops'}):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id":12}','')) as run:
|
||||
bridge.deliver({'title':'t','message':'m'})
|
||||
self.assertIn('infra-ops',run.call_args[0][0])
|
||||
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -1,36 +0,0 @@
|
||||
# Beszel to althing
|
||||
|
||||
`beszel-althing.service` runs on nh3-dev as lkraven, listening at
|
||||
`10.100.10.50:8096`. Beszel sends Shoutrrr generic JSON to `/beszel`; the
|
||||
bridge invokes the supported `postbox send` command with the body on stdin.
|
||||
It uses the established infra-ops automation identity and sends to infra-ops.
|
||||
|
||||
Deploy from the repository root:
|
||||
|
||||
```sh
|
||||
scripts/elway infra-ops@10.100.10.50 --playbook playbooks/beszel-althing.yaml
|
||||
```
|
||||
|
||||
The service accepts requests only from ana-docker and local diagnostic
|
||||
addresses. There is no public listener or new althing handle. It reports
|
||||
success only after postbox returns a delivery receipt. Failures return HTTP
|
||||
502 and appear in the system journal; there is no hidden retry queue. A
|
||||
post-office outage can therefore lose an alert, and needs independent health
|
||||
monitoring if guaranteed delivery during such outages becomes a requirement.
|
||||
|
||||
```sh
|
||||
systemctl status beszel-althing
|
||||
sudo -n journalctl -u beszel-althing --since '1 hour ago'
|
||||
curl -fsS http://10.100.10.50:8096/healthz
|
||||
```
|
||||
|
||||
`/healthz` checks the bridge process, not the downstream inbox. End-to-end
|
||||
verification requires a real Beszel threshold transition plus its althing
|
||||
receipt. The first verified alert is recorded in `stacks/beszel/README.md`.
|
||||
|
||||
To reroute later, change `BESZEL_ALERT_RECIPIENT` in the canonical unit to
|
||||
`miranda`, deploy, and trigger another end-to-end test. Leave `ALTHING_HANDLE`
|
||||
as infra-ops so the sender remains identifiable as infrastructure automation.
|
||||
The operator explicitly chose infra-ops for now.
|
||||
|
||||
Run `python3 -m unittest discover -s services/beszel-althing -p 'test_*.py'`.
|
||||
@@ -1,24 +0,0 @@
|
||||
[Unit]
|
||||
Description=Beszel alerts to the althing infra-ops inbox
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
User=lkraven
|
||||
Group=lkraven
|
||||
Environment=PYTHONUNBUFFERED=1
|
||||
Environment=ALTHING_HANDLE=infra-ops
|
||||
Environment=ALTHING_POST_OFFICE=http://10.100.50.40:8390
|
||||
Environment=POSTBOX=/home/lkraven/.local/bin/postbox
|
||||
Environment=BESZEL_ALERT_RECIPIENT=infra-ops
|
||||
Environment=BESZEL_BIND_HOST=10.100.10.50
|
||||
Environment=BESZEL_ALLOWED_SOURCES=10.250.50.70,10.100.10.50,127.0.0.1
|
||||
ExecStart=/usr/bin/python3 /opt/beszel-althing/bridge.py
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
PrivateTmp=true
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -1,63 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Receive Beszel's Shoutrrr JSON and deliver through the supported postbox CLI."""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
|
||||
def deliver(payload):
|
||||
title = payload.get('title', 'Beszel fleet alert')
|
||||
message = payload.get('message')
|
||||
if not isinstance(title, str) or not isinstance(message, str) or not message.strip():
|
||||
raise ValueError('Expected a nonempty message and string title')
|
||||
result = subprocess.run(
|
||||
[os.environ['POSTBOX'], '--json', 'send', '--to', os.environ['BESZEL_ALERT_RECIPIENT'],
|
||||
'--subject', '[Beszel] ' + title],
|
||||
input=message + '\n\nHub: http://10.250.50.70:8090\n',
|
||||
text=True, capture_output=True, timeout=25,
|
||||
)
|
||||
if result.returncode:
|
||||
raise RuntimeError('postbox delivery failed: ' + result.stderr.strip())
|
||||
receipt = json.loads(result.stdout)
|
||||
print(json.dumps({'event': 'delivered', 'title': title, 'receipt': receipt}), flush=True)
|
||||
return receipt
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def respond(self, status, body):
|
||||
data = json.dumps(body).encode()
|
||||
self.send_response(status)
|
||||
self.send_header('Content-Type', 'application/json')
|
||||
self.send_header('Content-Length', str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
def do_GET(self):
|
||||
self.respond(200 if self.path == '/healthz' else 404,
|
||||
{'service': 'beszel-althing', 'delivery': 'verified per POST'})
|
||||
|
||||
def do_POST(self):
|
||||
if self.path != '/beszel':
|
||||
return self.respond(404, {'error': 'Unknown route'})
|
||||
if self.client_address[0] not in os.environ['BESZEL_ALLOWED_SOURCES'].split(','):
|
||||
return self.respond(403, {'error': 'Source not allowed'})
|
||||
try:
|
||||
length = int(self.headers.get('Content-Length', '0'))
|
||||
if not 0 < length <= 65536:
|
||||
raise ValueError('Invalid body size')
|
||||
self.connection.settimeout(10)
|
||||
payload = json.loads(self.rfile.read(length))
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError('Expected JSON object')
|
||||
receipt = deliver(payload)
|
||||
except (ValueError, TypeError) as exc:
|
||||
return self.respond(400, {'error': str(exc)})
|
||||
except (OSError, RuntimeError, subprocess.TimeoutExpired) as exc:
|
||||
print(json.dumps({'event': 'delivery_failed', 'error': str(exc)}), flush=True)
|
||||
return self.respond(502, {'error': 'Althing delivery failed; inspect service journal'})
|
||||
self.respond(200, {'delivered': True, 'receipt': receipt})
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
ThreadingHTTPServer((os.environ['BESZEL_BIND_HOST'], int(os.environ.get('BESZEL_BIND_PORT', '8096'))), Handler).serve_forever()
|
||||
@@ -1,30 +0,0 @@
|
||||
import importlib.util
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
spec=importlib.util.spec_from_file_location('bridge',Path(__file__).with_name('bridge.py'))
|
||||
bridge=importlib.util.module_from_spec(spec);spec.loader.exec_module(bridge)
|
||||
|
||||
class DeliveryTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.env=patch.dict(os.environ,POSTBOX='/bin/postbox',BESZEL_ALERT_RECIPIENT='infra-ops')
|
||||
self.env.start();self.addCleanup(self.env.stop)
|
||||
def test_posts_body_as_stdin_and_returns_receipt(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id": 42}','')) as run:
|
||||
self.assertEqual(bridge.deliver({'title':'Disk 85%','message':'tank is full'})['message_id'],42)
|
||||
args,kw=run.call_args
|
||||
self.assertEqual(args[0],['/bin/postbox','--json','send','--to','infra-ops','--subject','[Beszel] Disk 85%'])
|
||||
self.assertIn('tank is full',kw['input'])
|
||||
def test_delivery_failure_is_not_success(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],2,'','post office unavailable')):
|
||||
with self.assertRaises(RuntimeError):bridge.deliver({'message':'disk full'})
|
||||
def test_rejects_empty_message_without_sending(self):
|
||||
with patch.object(bridge.subprocess,'run') as run:
|
||||
with self.assertRaises(ValueError):bridge.deliver({'message':''})
|
||||
run.assert_not_called()
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -1,27 +0,0 @@
|
||||
# althing-chamber stack tunables. Copy to `.env` on ana-docker before deploying.
|
||||
#
|
||||
# The deploy playbook seeds `.env` from this template on first run only —
|
||||
# it won't clobber an existing `.env`.
|
||||
|
||||
# Image tag. Built locally from the vh/althing git repo by the playbook.
|
||||
ALTHING_IMAGE=althing-chamber:local
|
||||
|
||||
# Host port exposing the chamber UI (container always listens on 8000
|
||||
# internally). Internal-only — no Traefik.
|
||||
# Chamber's compiled-in default port is 7878 but that collides with
|
||||
# task-board's 7878 on the same host. 7881 is the canonical fleet slot,
|
||||
# adjacent to task-board:7878 and vor:7879.
|
||||
ALTHING_PORT=7881
|
||||
|
||||
# Bind address for the host port. 0.0.0.0 = LAN-reachable (default for an
|
||||
# internal-only tool).
|
||||
ALTHING_BIND=0.0.0.0
|
||||
|
||||
# Host path for SQLite + state. Container runs as uid 1000 (matches
|
||||
# lkraven on these hosts) so the playbook's mkdir without sudo produces
|
||||
# a writable dir.
|
||||
#
|
||||
# This single dir is the only persistent state — althing's SQLite DB,
|
||||
# read_cursors, notification_state, hand_queue, floor_grants all live
|
||||
# here. Restic backs it up via the standard /opt/docker tree.
|
||||
ALTHING_DATA_DIR=/opt/docker/conf/althing-chamber/data
|
||||
@@ -1,110 +0,0 @@
|
||||
# althing-chamber
|
||||
|
||||
Web UI + moderator daemon + agent-runner + Valkey IPC bridge for the
|
||||
althing inter-agent message bus. Four services on the compose default
|
||||
network. The chamber serves FastAPI/HTMX, the forseti daemon runs the
|
||||
moderation + curation loops, the agent-runner daemon claims floor
|
||||
grants for worldtree-driver agents and dispatches them to Worldtree's
|
||||
conversation API, and Valkey carries the Phase 3.1 cross-process
|
||||
streaming events (msg_start/delta/thinking/complete/curated) that are
|
||||
too high-volume to flow through SQLite. SQLite-backed state still
|
||||
shares between chamber + forseti + agent-runner via the bind-mount;
|
||||
streaming events ride pub/sub on the docker default network.
|
||||
|
||||
**Server:** ana-docker
|
||||
**URL:** `http://10.250.50.70:7881` (configurable via `.env`)
|
||||
**Upstream repo:** [vh/althing](https://gitea.phasefinal.com/vh/althing)
|
||||
**Image:** `althing-chamber:local` — built on the host from the git repo
|
||||
by the deploy playbook. Not pulled from a registry.
|
||||
|
||||
## Services in this stack
|
||||
|
||||
| Container | Role | Port | Healthcheck |
|
||||
|---|---|---|---|
|
||||
| `althing-chamber` | FastAPI/HTMX web UI; SSE subscribers; `/health` endpoint | host 7881 → container 8000 | `python urllib /health` |
|
||||
| `althing-forseti` | Moderator + curator daemon; writes events that chamber's bridge picks up | — (no HTTP) | none (process-up signal only) |
|
||||
| `althing-agent-runner` | Phase 2: claims worldtree-driver floor grants and dispatches to Worldtree's conversation API | — (no HTTP) | none (process-up signal only) |
|
||||
| `althing-valkey` | Phase 3.1: Valkey 8 redis-protocol pub/sub bridge — carries streaming events from agent-runner → chamber | — (internal-only, `valkey:6379` on default network) | `valkey-cli ping` |
|
||||
|
||||
The three althing services (chamber/forseti/agent-runner) use the same
|
||||
`${ALTHING_IMAGE}`; the `command:` line in compose picks which entrypoint
|
||||
runs in each container. Valkey is a stock upstream image (no custom build).
|
||||
chamber + agent-runner `depends_on: valkey: service_healthy` so the pub/sub
|
||||
bridge is up before either side starts publishing or subscribing.
|
||||
|
||||
The agent-runner is safe to enable preemptively — when no
|
||||
`driver=worldtree` handles are declared in config, it polls
|
||||
`floor_grants` and sleeps when the query returns empty. Multi-instance
|
||||
safe via the atomic `UPDATE … WHERE consumed_at IS NULL` claim primitive.
|
||||
|
||||
## Deploy
|
||||
|
||||
Two paths — automated (preferred) and manual (escape hatch / first-time).
|
||||
|
||||
### Automated (Gitea Actions, push-to-main)
|
||||
|
||||
The vh/althing repo ships `.gitea/workflows/deploy.yaml`. Every push to
|
||||
main + manual `workflow_dispatch` triggers the elway playbook below
|
||||
pinned to the triggering commit SHA. A reference copy of the workflow
|
||||
lives next to this README at
|
||||
[`gitea-workflow-deploy.yaml.example`](gitea-workflow-deploy.yaml.example);
|
||||
the canonical source is in the vh/althing repo. The example header
|
||||
lists the two repo secrets required (`DEPLOY_SSH_KEY`,
|
||||
`MGMT_REPO_TOKEN`).
|
||||
|
||||
### Manual (elway from a workstation)
|
||||
|
||||
```bash
|
||||
# First deploy (or update to latest main)
|
||||
scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml
|
||||
|
||||
# Pin to a specific ref (tag, branch, or commit SHA)
|
||||
scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml --var ref=v0.1.0
|
||||
```
|
||||
|
||||
## Path layout (on ana-docker)
|
||||
|
||||
| Host path | Container path | Purpose | Restic? |
|
||||
|---|---|---|---|
|
||||
| `/opt/docker/build/althing-chamber/` | — | git checkout used as docker build context | excluded |
|
||||
| `/opt/docker/compose/althing-chamber/` | — | compose.yaml + .env | included (via `/opt/docker`) |
|
||||
| `/opt/docker/conf/althing-chamber/data/` | `/app/data` | SQLite + state for BOTH services | **included** |
|
||||
|
||||
## Network model
|
||||
|
||||
Internal tooling, LAN-only. Chamber's container port 8000 is published
|
||||
on the host at `0.0.0.0:7881` (configurable via `ALTHING_BIND` /
|
||||
`ALTHING_PORT`); access is direct via `http://10.250.50.70:7881`. No
|
||||
Traefik, no TLS terminator, no public hostname.
|
||||
|
||||
Forseti has no port — it's a daemon. The two services communicate only
|
||||
via the shared SQLite file under the `${ALTHING_DATA_DIR}` bind-mount;
|
||||
no docker network coupling beyond compose's default bridge.
|
||||
|
||||
## Env-var contract
|
||||
|
||||
| Var | In env? | In `~/.althing/config.yaml`? | Notes |
|
||||
|---|---|---|---|
|
||||
| `ALTHING_ROOT` | ✓ (set in compose) | n/a | Overrides the default `~/.althing/` dir. Container sets `/app/data`. |
|
||||
| `ALTHING_DB` | ✓ (set in compose) | n/a | Explicit SQLite path. Defaults to `${ALTHING_ROOT}/althing.db`. |
|
||||
| `ALTHING_BIND` | ✓ (deploy-time) | also accepted | Bind address for the chamber HTTP server. Container always sets 0.0.0.0 internally. |
|
||||
| `ALTHING_PORT` | ✓ (deploy-time) | also accepted | Chamber listens here internally (always 8000 inside the container). |
|
||||
|
||||
Env-var support for `ALTHING_BIND` / `ALTHING_PORT` was added on the
|
||||
galdrabok side as part of the container-deploy cycle (env > config.yaml
|
||||
> defaults precedence).
|
||||
|
||||
## First-deploy sequence (when galdrabok's Dockerfile lands)
|
||||
|
||||
1. Generate a deploy keypair on ana-docker (private stays on host;
|
||||
pubkey lands in `~lkraven/.ssh/authorized_keys`).
|
||||
2. Wire two secrets in `vh/althing` Actions settings:
|
||||
- `DEPLOY_SSH_KEY` — the private key from step 1.
|
||||
- `MGMT_REPO_TOKEN` — Gitea PAT with `read:repository` on this repo,
|
||||
used by the workflow to clone the management repo for the playbook.
|
||||
3. Push to vh/althing's main branch (or trigger `workflow_dispatch`);
|
||||
the Actions runner clones both repos, configures SSH, runs
|
||||
`scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml --var ref=$SHA`.
|
||||
4. Playbook: clones into `/opt/docker/build/althing-chamber`, builds
|
||||
the image, uploads compose + .env (one-time seed), brings both
|
||||
services up, polls `/health` until 200.
|
||||
@@ -1,124 +0,0 @@
|
||||
# althing-chamber stack — chamber + forseti + agent-runner + valkey.
|
||||
#
|
||||
# Four services on the compose default network:
|
||||
# althing-chamber — FastAPI/HTMX app (port 7881 host → 8000 container)
|
||||
# althing-forseti — moderator daemon (no port; cross-process glue via the DB)
|
||||
# althing-agent-runner — Phase 2 daemon: claims worldtree-driver grants and
|
||||
# dispatches to Worldtree's conversation API (no port)
|
||||
# valkey — Phase 3.1 sibling service: Valkey 8 alpine, redis-protocol
|
||||
# pub/sub bridge for cross-process streaming events from
|
||||
# agent-runner → chamber SSE subscribers. Reached via
|
||||
# docker DNS at `valkey:6379` on the default network.
|
||||
# No exposed port; internal-only.
|
||||
#
|
||||
# Three storage / IPC channels:
|
||||
# - SQLite bind-mount at /app/data backs the althing.db state shared by
|
||||
# chamber + forseti + agent-runner (eventbus.bridge_from_db drains DB
|
||||
# commits into chamber's SSE).
|
||||
# - Valkey pub/sub on docker-default network carries the Phase 3 streaming
|
||||
# events (msg_start/thinking/delta/complete/curated) that don't go
|
||||
# through SQLite — too high-volume + ephemeral for the DB.
|
||||
# - chamber's healthcheck-blocked startup gate on valkey ensures the
|
||||
# subscriber side is up before chamber begins handling SSE traffic.
|
||||
#
|
||||
# Image is built on the host from the vh/althing git repo by the deploy
|
||||
# playbook (`playbooks/deploy-althing-chamber.yaml`), which clones into
|
||||
# /opt/docker/build/althing-chamber and runs `docker build -t
|
||||
# althing-chamber:local .` before installing this compose and bringing
|
||||
# both services up. No registry.
|
||||
#
|
||||
# Internal tooling — accessed directly at http://10.250.50.70:7881 over
|
||||
# the LAN; does NOT traverse Traefik. State persists under
|
||||
# /opt/docker/conf/althing-chamber/data on the host.
|
||||
#
|
||||
# All tunables live in .env — edit that, not this file.
|
||||
|
||||
services:
|
||||
valkey:
|
||||
image: valkey/valkey:8-alpine
|
||||
container_name: althing-valkey
|
||||
restart: unless-stopped
|
||||
# No exposed port — chamber + agent-runner reach via docker DNS
|
||||
# at `valkey:6379` on the compose default network. No volume —
|
||||
# the pub/sub channel doesn't persist anything between restarts.
|
||||
healthcheck:
|
||||
test: ["CMD", "valkey-cli", "ping"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
althing-chamber:
|
||||
image: ${ALTHING_IMAGE}
|
||||
container_name: althing-chamber
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
valkey:
|
||||
condition: service_healthy
|
||||
ports:
|
||||
- "${ALTHING_BIND:-0.0.0.0}:${ALTHING_PORT}:8000"
|
||||
environment:
|
||||
# ALTHING_ROOT moves the entire ~/.althing dir; ALTHING_DB additionally
|
||||
# pins the SQLite path explicitly. Both point inside the bind-mount.
|
||||
- ALTHING_ROOT=/app/data
|
||||
- ALTHING_DB=/app/data/althing.db
|
||||
# ALTHING_BIND / ALTHING_PORT — galdrabok-side env-var precedence
|
||||
# (env > config.yaml > defaults) is being added in the same cycle
|
||||
# as this scaffold lands. Container always listens on 8000 internally;
|
||||
# the host port mapping above is the only externally-visible knob.
|
||||
- ALTHING_BIND=0.0.0.0
|
||||
- ALTHING_PORT=8000
|
||||
volumes:
|
||||
- ${ALTHING_DATA_DIR}:/app/data
|
||||
command: ["althing-chamber"]
|
||||
healthcheck:
|
||||
# Liveness probe — chamber's /health endpoint returns 200 with no DB
|
||||
# read (true liveness, not readiness). galdrabok adds this endpoint
|
||||
# in the same cycle as this scaffold; container will crashloop on
|
||||
# healthcheck until that lands.
|
||||
test: ["CMD-SHELL", "python -c 'import urllib.request,sys; r=urllib.request.urlopen(\"http://127.0.0.1:8000/health\",timeout=3); sys.exit(0 if r.status==200 else 1)' || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
labels:
|
||||
- homepage.group=Toolchain
|
||||
- homepage.name=althing chamber
|
||||
- homepage.icon=mdi-bullhorn
|
||||
- homepage.description=Web UI for the althing inter-agent message bus
|
||||
- homepage.href=http://10.250.50.70:${ALTHING_PORT}
|
||||
|
||||
althing-forseti:
|
||||
image: ${ALTHING_IMAGE}
|
||||
container_name: althing-forseti
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- ALTHING_ROOT=/app/data
|
||||
- ALTHING_DB=/app/data/althing.db
|
||||
volumes:
|
||||
- ${ALTHING_DATA_DIR}:/app/data
|
||||
command: ["althing-forseti"]
|
||||
# No healthcheck — the forseti CLI doesn't expose one. Liveness signal
|
||||
# for ops is "container hasn't exited" + chamber-side observation that
|
||||
# bridge_from_db events are flowing.
|
||||
|
||||
althing-agent-runner:
|
||||
image: ${ALTHING_IMAGE}
|
||||
container_name: althing-agent-runner
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
valkey:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
- ALTHING_ROOT=/app/data
|
||||
- ALTHING_DB=/app/data/althing.db
|
||||
volumes:
|
||||
- ${ALTHING_DATA_DIR}:/app/data
|
||||
command: ["althing-agent-runner"]
|
||||
# Phase 2 daemon (added 2026-05-16). Polls floor_grants WHERE
|
||||
# consumed_at IS NULL AND agents.driver='worldtree', claims via
|
||||
# atomic UPDATE, calls Worldtree's conversation API, posts the
|
||||
# agent's response back through the bus as a broadcast. Harmless
|
||||
# when no driver=worldtree handles are declared — runner sleeps
|
||||
# at poll_interval_seconds. Multi-instance safe via claim_grant's
|
||||
# atomic UPDATE; no flock required. Same lack-of-healthcheck story
|
||||
# as forseti (CLI doesn't expose one).
|
||||
@@ -1,91 +0,0 @@
|
||||
# Gitea Actions workflow for althing-chamber.
|
||||
#
|
||||
# THIS FILE LIVES IN THE VH/ALTHING REPO, NOT HERE.
|
||||
# Copy to vh/althing:.gitea/workflows/deploy.yaml and commit.
|
||||
# (The canonical copy lives in the althing repo; this file is a
|
||||
# reference for what shape the workflow takes.)
|
||||
#
|
||||
# What it does on every push to main (and on manual workflow_dispatch):
|
||||
# 1. Checks out althing itself (the triggering repo).
|
||||
# 2. Checks out vh/esh-pfi-infrastructure to pick up the elway
|
||||
# playbook and helper scripts.
|
||||
# 3. Configures SSH so elway can reach ana-docker.
|
||||
# 4. Runs `scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml`
|
||||
# pinning to the commit SHA that triggered the workflow.
|
||||
#
|
||||
# Required Actions secrets (configure under
|
||||
# https://gitea.phasefinal.com/vh/althing/settings/actions/secrets,
|
||||
# or org-level for reuse across repos):
|
||||
#
|
||||
# DEPLOY_SSH_KEY Private SSH key whose pubkey is in
|
||||
# ~lkraven/.ssh/authorized_keys on ana-docker.
|
||||
# Used by the runner to invoke the elway playbook.
|
||||
# Generate fresh; don't reuse a personal key.
|
||||
#
|
||||
# MGMT_REPO_TOKEN Gitea PAT (read:repository scope) on
|
||||
# vh/esh-pfi-infrastructure, used to clone the
|
||||
# management repo. Generate at
|
||||
# https://gitea.phasefinal.com/-/user/settings/applications.
|
||||
|
||||
name: Deploy althing-chamber
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
# `pfi-fleet` matches the central runner on ana-docker. Pin to
|
||||
# `ana-docker` instead if you want to refuse running on a future
|
||||
# site-local runner. The runner's label embeds a default image
|
||||
# (node:20-bookworm-slim) — has node + git out of the box, so
|
||||
# actions/checkout@v4 (a JS action) works without a custom
|
||||
# container. We just apt-install python3 + pyyaml for elway.
|
||||
runs-on: pfi-fleet
|
||||
|
||||
steps:
|
||||
- name: Install playbook prerequisites
|
||||
run: |
|
||||
apt-get update -qq
|
||||
apt-get install -y --no-install-recommends \
|
||||
python3 python3-yaml openssh-client
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
- name: Checkout althing (triggering repo)
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Checkout management repo (eshpfi-management)
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
repository: vh/esh-pfi-infrastructure
|
||||
token: ${{ secrets.MGMT_REPO_TOKEN }}
|
||||
path: _mgmt
|
||||
|
||||
- name: Configure SSH to ana-docker
|
||||
run: |
|
||||
mkdir -p ~/.ssh
|
||||
# The DEPLOY_SSH_KEY secret is the full private key contents,
|
||||
# newline-terminated. ssh refuses keys that aren't 0600.
|
||||
printf '%s\n' "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/id_ed25519
|
||||
chmod 600 ~/.ssh/id_ed25519
|
||||
|
||||
# ssh_config alias so elway resolves "ana-docker" the same
|
||||
# way it would on a workstation. accept-new is fine for a
|
||||
# fresh job container — host key gets cached for the lifetime
|
||||
# of this job only.
|
||||
cat > ~/.ssh/config <<'EOF'
|
||||
Host ana-docker
|
||||
HostName 10.250.50.70
|
||||
User lkraven
|
||||
IdentityFile ~/.ssh/id_ed25519
|
||||
StrictHostKeyChecking accept-new
|
||||
EOF
|
||||
chmod 600 ~/.ssh/config
|
||||
|
||||
- name: Deploy althing-chamber (elway playbook, pinned to this commit)
|
||||
working-directory: _mgmt
|
||||
run: |
|
||||
scripts/elway ana-docker \
|
||||
--playbook playbooks/deploy-althing-chamber.yaml \
|
||||
--var ref=${{ github.sha }}
|
||||
@@ -120,7 +120,7 @@ corviduo-dev remains monitored but its alert policy was not changed.
|
||||
Notification URL:
|
||||
`generic://10.100.10.50:8096/beszel?disabletls=yes&template=json`
|
||||
|
||||
The bridge at `services/beszel-althing/` forwards through `postbox` to the
|
||||
The bridge at `services/althing-alert-bridge/` forwards through `postbox` to the
|
||||
**infra-ops inbox**, as the operator requested. Existing unused email delivery
|
||||
was replaced with this verified route. Miranda is a later cutover, not enabled.
|
||||
See that service's README for operation and recipient changes.
|
||||
|
||||
@@ -17,10 +17,22 @@
|
||||
# 2026-09-21. A seed that ships red on day one teaches everyone to ignore the
|
||||
# board, which is how you end up with a monitor nobody reads.
|
||||
#
|
||||
# NOT SEEDED, deliberately: "althing chamber" (10.250.50.70:7881) is on the
|
||||
# Homepage dashboard but refuses connections right now. It is either genuinely
|
||||
# down or retired; seeding it would put a red row on a brand-new board before
|
||||
# anyone has decided which. Resolve it, then add it.
|
||||
# "althing chamber" was a candidate and was RETIRED instead (operator,
|
||||
# 2026-09-21): three of its four containers had never started since being
|
||||
# created on 2026-09-19, so :7881 refused. Stack, host dirs and image removed.
|
||||
|
||||
# ---- where alerts GO ---------------------------------------------------------
|
||||
# Seeded BEFORE the monitors, and `applyExisting` attaches the channel to rows
|
||||
# that already exist. A board that detects and notifies nobody is precisely the
|
||||
# failure this service layer was built to close -- Homepage's own healthcheck
|
||||
# caught its 2026-09-18 death correctly and nothing was subscribed.
|
||||
#
|
||||
# Route: Kuma -> althing-alert-bridge (/kuma) -> postbox -> infra-ops inbox.
|
||||
# Same path Beszel uses, different route, so the subject says which tool spoke:
|
||||
# "[Uptime Kuma] Homepage is DOWN" rather than a Beszel-labelled lie.
|
||||
notifications:
|
||||
- name: althing (infra-ops)
|
||||
webhookURL: http://10.100.10.50:8096/kuma
|
||||
|
||||
monitors:
|
||||
# ---- fleet toolchain: dead = agents and the operator are blocked ----
|
||||
|
||||
Reference in New Issue
Block a user