feat(alerts): generalize the althing bridge, wire Kuma to it, retire chamber

THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.

Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.

Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.

⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.

Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.

parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.

KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.

⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.

End-to-end, both shapes, read back from the inbox:
  [Uptime Kuma] Homepage is DOWN          + target + board link
  [Uptime Kuma] althing (infra-ops) Testing   (no fabricated subject)
  [Beszel] BRIDGE RENAME CHECK            + hub footer, unchanged

ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
This commit is contained in:
vh
2026-09-21 23:11:54 -07:00
parent 3a85a6bce1
commit 6f0a9b9fae
19 changed files with 818 additions and 927 deletions
-27
View File
@@ -1,27 +0,0 @@
# althing-chamber stack tunables. Copy to `.env` on ana-docker before deploying.
#
# The deploy playbook seeds `.env` from this template on first run only —
# it won't clobber an existing `.env`.
# Image tag. Built locally from the vh/althing git repo by the playbook.
ALTHING_IMAGE=althing-chamber:local
# Host port exposing the chamber UI (container always listens on 8000
# internally). Internal-only — no Traefik.
# Chamber's compiled-in default port is 7878 but that collides with
# task-board's 7878 on the same host. 7881 is the canonical fleet slot,
# adjacent to task-board:7878 and vor:7879.
ALTHING_PORT=7881
# Bind address for the host port. 0.0.0.0 = LAN-reachable (default for an
# internal-only tool).
ALTHING_BIND=0.0.0.0
# Host path for SQLite + state. Container runs as uid 1000 (matches
# lkraven on these hosts) so the playbook's mkdir without sudo produces
# a writable dir.
#
# This single dir is the only persistent state — althing's SQLite DB,
# read_cursors, notification_state, hand_queue, floor_grants all live
# here. Restic backs it up via the standard /opt/docker tree.
ALTHING_DATA_DIR=/opt/docker/conf/althing-chamber/data
-110
View File
@@ -1,110 +0,0 @@
# althing-chamber
Web UI + moderator daemon + agent-runner + Valkey IPC bridge for the
althing inter-agent message bus. Four services on the compose default
network. The chamber serves FastAPI/HTMX, the forseti daemon runs the
moderation + curation loops, the agent-runner daemon claims floor
grants for worldtree-driver agents and dispatches them to Worldtree's
conversation API, and Valkey carries the Phase 3.1 cross-process
streaming events (msg_start/delta/thinking/complete/curated) that are
too high-volume to flow through SQLite. SQLite-backed state still
shares between chamber + forseti + agent-runner via the bind-mount;
streaming events ride pub/sub on the docker default network.
**Server:** ana-docker
**URL:** `http://10.250.50.70:7881` (configurable via `.env`)
**Upstream repo:** [vh/althing](https://gitea.phasefinal.com/vh/althing)
**Image:** `althing-chamber:local` — built on the host from the git repo
by the deploy playbook. Not pulled from a registry.
## Services in this stack
| Container | Role | Port | Healthcheck |
|---|---|---|---|
| `althing-chamber` | FastAPI/HTMX web UI; SSE subscribers; `/health` endpoint | host 7881 → container 8000 | `python urllib /health` |
| `althing-forseti` | Moderator + curator daemon; writes events that chamber's bridge picks up | — (no HTTP) | none (process-up signal only) |
| `althing-agent-runner` | Phase 2: claims worldtree-driver floor grants and dispatches to Worldtree's conversation API | — (no HTTP) | none (process-up signal only) |
| `althing-valkey` | Phase 3.1: Valkey 8 redis-protocol pub/sub bridge — carries streaming events from agent-runner → chamber | — (internal-only, `valkey:6379` on default network) | `valkey-cli ping` |
The three althing services (chamber/forseti/agent-runner) use the same
`${ALTHING_IMAGE}`; the `command:` line in compose picks which entrypoint
runs in each container. Valkey is a stock upstream image (no custom build).
chamber + agent-runner `depends_on: valkey: service_healthy` so the pub/sub
bridge is up before either side starts publishing or subscribing.
The agent-runner is safe to enable preemptively — when no
`driver=worldtree` handles are declared in config, it polls
`floor_grants` and sleeps when the query returns empty. Multi-instance
safe via the atomic `UPDATE … WHERE consumed_at IS NULL` claim primitive.
## Deploy
Two paths — automated (preferred) and manual (escape hatch / first-time).
### Automated (Gitea Actions, push-to-main)
The vh/althing repo ships `.gitea/workflows/deploy.yaml`. Every push to
main + manual `workflow_dispatch` triggers the elway playbook below
pinned to the triggering commit SHA. A reference copy of the workflow
lives next to this README at
[`gitea-workflow-deploy.yaml.example`](gitea-workflow-deploy.yaml.example);
the canonical source is in the vh/althing repo. The example header
lists the two repo secrets required (`DEPLOY_SSH_KEY`,
`MGMT_REPO_TOKEN`).
### Manual (elway from a workstation)
```bash
# First deploy (or update to latest main)
scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml
# Pin to a specific ref (tag, branch, or commit SHA)
scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml --var ref=v0.1.0
```
## Path layout (on ana-docker)
| Host path | Container path | Purpose | Restic? |
|---|---|---|---|
| `/opt/docker/build/althing-chamber/` | — | git checkout used as docker build context | excluded |
| `/opt/docker/compose/althing-chamber/` | — | compose.yaml + .env | included (via `/opt/docker`) |
| `/opt/docker/conf/althing-chamber/data/` | `/app/data` | SQLite + state for BOTH services | **included** |
## Network model
Internal tooling, LAN-only. Chamber's container port 8000 is published
on the host at `0.0.0.0:7881` (configurable via `ALTHING_BIND` /
`ALTHING_PORT`); access is direct via `http://10.250.50.70:7881`. No
Traefik, no TLS terminator, no public hostname.
Forseti has no port — it's a daemon. The two services communicate only
via the shared SQLite file under the `${ALTHING_DATA_DIR}` bind-mount;
no docker network coupling beyond compose's default bridge.
## Env-var contract
| Var | In env? | In `~/.althing/config.yaml`? | Notes |
|---|---|---|---|
| `ALTHING_ROOT` | ✓ (set in compose) | n/a | Overrides the default `~/.althing/` dir. Container sets `/app/data`. |
| `ALTHING_DB` | ✓ (set in compose) | n/a | Explicit SQLite path. Defaults to `${ALTHING_ROOT}/althing.db`. |
| `ALTHING_BIND` | ✓ (deploy-time) | also accepted | Bind address for the chamber HTTP server. Container always sets 0.0.0.0 internally. |
| `ALTHING_PORT` | ✓ (deploy-time) | also accepted | Chamber listens here internally (always 8000 inside the container). |
Env-var support for `ALTHING_BIND` / `ALTHING_PORT` was added on the
galdrabok side as part of the container-deploy cycle (env > config.yaml
> defaults precedence).
## First-deploy sequence (when galdrabok's Dockerfile lands)
1. Generate a deploy keypair on ana-docker (private stays on host;
pubkey lands in `~lkraven/.ssh/authorized_keys`).
2. Wire two secrets in `vh/althing` Actions settings:
- `DEPLOY_SSH_KEY` — the private key from step 1.
- `MGMT_REPO_TOKEN` — Gitea PAT with `read:repository` on this repo,
used by the workflow to clone the management repo for the playbook.
3. Push to vh/althing's main branch (or trigger `workflow_dispatch`);
the Actions runner clones both repos, configures SSH, runs
`scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml --var ref=$SHA`.
4. Playbook: clones into `/opt/docker/build/althing-chamber`, builds
the image, uploads compose + .env (one-time seed), brings both
services up, polls `/health` until 200.
-124
View File
@@ -1,124 +0,0 @@
# althing-chamber stack — chamber + forseti + agent-runner + valkey.
#
# Four services on the compose default network:
# althing-chamber — FastAPI/HTMX app (port 7881 host → 8000 container)
# althing-forseti — moderator daemon (no port; cross-process glue via the DB)
# althing-agent-runner — Phase 2 daemon: claims worldtree-driver grants and
# dispatches to Worldtree's conversation API (no port)
# valkey — Phase 3.1 sibling service: Valkey 8 alpine, redis-protocol
# pub/sub bridge for cross-process streaming events from
# agent-runner → chamber SSE subscribers. Reached via
# docker DNS at `valkey:6379` on the default network.
# No exposed port; internal-only.
#
# Three storage / IPC channels:
# - SQLite bind-mount at /app/data backs the althing.db state shared by
# chamber + forseti + agent-runner (eventbus.bridge_from_db drains DB
# commits into chamber's SSE).
# - Valkey pub/sub on docker-default network carries the Phase 3 streaming
# events (msg_start/thinking/delta/complete/curated) that don't go
# through SQLite — too high-volume + ephemeral for the DB.
# - chamber's healthcheck-blocked startup gate on valkey ensures the
# subscriber side is up before chamber begins handling SSE traffic.
#
# Image is built on the host from the vh/althing git repo by the deploy
# playbook (`playbooks/deploy-althing-chamber.yaml`), which clones into
# /opt/docker/build/althing-chamber and runs `docker build -t
# althing-chamber:local .` before installing this compose and bringing
# both services up. No registry.
#
# Internal tooling — accessed directly at http://10.250.50.70:7881 over
# the LAN; does NOT traverse Traefik. State persists under
# /opt/docker/conf/althing-chamber/data on the host.
#
# All tunables live in .env — edit that, not this file.
services:
valkey:
image: valkey/valkey:8-alpine
container_name: althing-valkey
restart: unless-stopped
# No exposed port — chamber + agent-runner reach via docker DNS
# at `valkey:6379` on the compose default network. No volume —
# the pub/sub channel doesn't persist anything between restarts.
healthcheck:
test: ["CMD", "valkey-cli", "ping"]
interval: 5s
timeout: 3s
retries: 5
althing-chamber:
image: ${ALTHING_IMAGE}
container_name: althing-chamber
restart: unless-stopped
depends_on:
valkey:
condition: service_healthy
ports:
- "${ALTHING_BIND:-0.0.0.0}:${ALTHING_PORT}:8000"
environment:
# ALTHING_ROOT moves the entire ~/.althing dir; ALTHING_DB additionally
# pins the SQLite path explicitly. Both point inside the bind-mount.
- ALTHING_ROOT=/app/data
- ALTHING_DB=/app/data/althing.db
# ALTHING_BIND / ALTHING_PORT — galdrabok-side env-var precedence
# (env > config.yaml > defaults) is being added in the same cycle
# as this scaffold lands. Container always listens on 8000 internally;
# the host port mapping above is the only externally-visible knob.
- ALTHING_BIND=0.0.0.0
- ALTHING_PORT=8000
volumes:
- ${ALTHING_DATA_DIR}:/app/data
command: ["althing-chamber"]
healthcheck:
# Liveness probe — chamber's /health endpoint returns 200 with no DB
# read (true liveness, not readiness). galdrabok adds this endpoint
# in the same cycle as this scaffold; container will crashloop on
# healthcheck until that lands.
test: ["CMD-SHELL", "python -c 'import urllib.request,sys; r=urllib.request.urlopen(\"http://127.0.0.1:8000/health\",timeout=3); sys.exit(0 if r.status==200 else 1)' || exit 1"]
interval: 30s
timeout: 5s
retries: 3
start_period: 30s
labels:
- homepage.group=Toolchain
- homepage.name=althing chamber
- homepage.icon=mdi-bullhorn
- homepage.description=Web UI for the althing inter-agent message bus
- homepage.href=http://10.250.50.70:${ALTHING_PORT}
althing-forseti:
image: ${ALTHING_IMAGE}
container_name: althing-forseti
restart: unless-stopped
environment:
- ALTHING_ROOT=/app/data
- ALTHING_DB=/app/data/althing.db
volumes:
- ${ALTHING_DATA_DIR}:/app/data
command: ["althing-forseti"]
# No healthcheck — the forseti CLI doesn't expose one. Liveness signal
# for ops is "container hasn't exited" + chamber-side observation that
# bridge_from_db events are flowing.
althing-agent-runner:
image: ${ALTHING_IMAGE}
container_name: althing-agent-runner
restart: unless-stopped
depends_on:
valkey:
condition: service_healthy
environment:
- ALTHING_ROOT=/app/data
- ALTHING_DB=/app/data/althing.db
volumes:
- ${ALTHING_DATA_DIR}:/app/data
command: ["althing-agent-runner"]
# Phase 2 daemon (added 2026-05-16). Polls floor_grants WHERE
# consumed_at IS NULL AND agents.driver='worldtree', claims via
# atomic UPDATE, calls Worldtree's conversation API, posts the
# agent's response back through the bus as a broadcast. Harmless
# when no driver=worldtree handles are declared — runner sleeps
# at poll_interval_seconds. Multi-instance safe via claim_grant's
# atomic UPDATE; no flock required. Same lack-of-healthcheck story
# as forseti (CLI doesn't expose one).
@@ -1,91 +0,0 @@
# Gitea Actions workflow for althing-chamber.
#
# THIS FILE LIVES IN THE VH/ALTHING REPO, NOT HERE.
# Copy to vh/althing:.gitea/workflows/deploy.yaml and commit.
# (The canonical copy lives in the althing repo; this file is a
# reference for what shape the workflow takes.)
#
# What it does on every push to main (and on manual workflow_dispatch):
# 1. Checks out althing itself (the triggering repo).
# 2. Checks out vh/esh-pfi-infrastructure to pick up the elway
# playbook and helper scripts.
# 3. Configures SSH so elway can reach ana-docker.
# 4. Runs `scripts/elway ana-docker --playbook playbooks/deploy-althing-chamber.yaml`
# pinning to the commit SHA that triggered the workflow.
#
# Required Actions secrets (configure under
# https://gitea.phasefinal.com/vh/althing/settings/actions/secrets,
# or org-level for reuse across repos):
#
# DEPLOY_SSH_KEY Private SSH key whose pubkey is in
# ~lkraven/.ssh/authorized_keys on ana-docker.
# Used by the runner to invoke the elway playbook.
# Generate fresh; don't reuse a personal key.
#
# MGMT_REPO_TOKEN Gitea PAT (read:repository scope) on
# vh/esh-pfi-infrastructure, used to clone the
# management repo. Generate at
# https://gitea.phasefinal.com/-/user/settings/applications.
name: Deploy althing-chamber
on:
push:
branches: [main]
workflow_dispatch:
jobs:
deploy:
# `pfi-fleet` matches the central runner on ana-docker. Pin to
# `ana-docker` instead if you want to refuse running on a future
# site-local runner. The runner's label embeds a default image
# (node:20-bookworm-slim) — has node + git out of the box, so
# actions/checkout@v4 (a JS action) works without a custom
# container. We just apt-install python3 + pyyaml for elway.
runs-on: pfi-fleet
steps:
- name: Install playbook prerequisites
run: |
apt-get update -qq
apt-get install -y --no-install-recommends \
python3 python3-yaml openssh-client
rm -rf /var/lib/apt/lists/*
- name: Checkout althing (triggering repo)
uses: actions/checkout@v4
- name: Checkout management repo (eshpfi-management)
uses: actions/checkout@v4
with:
repository: vh/esh-pfi-infrastructure
token: ${{ secrets.MGMT_REPO_TOKEN }}
path: _mgmt
- name: Configure SSH to ana-docker
run: |
mkdir -p ~/.ssh
# The DEPLOY_SSH_KEY secret is the full private key contents,
# newline-terminated. ssh refuses keys that aren't 0600.
printf '%s\n' "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
# ssh_config alias so elway resolves "ana-docker" the same
# way it would on a workstation. accept-new is fine for a
# fresh job container — host key gets cached for the lifetime
# of this job only.
cat > ~/.ssh/config <<'EOF'
Host ana-docker
HostName 10.250.50.70
User lkraven
IdentityFile ~/.ssh/id_ed25519
StrictHostKeyChecking accept-new
EOF
chmod 600 ~/.ssh/config
- name: Deploy althing-chamber (elway playbook, pinned to this commit)
working-directory: _mgmt
run: |
scripts/elway ana-docker \
--playbook playbooks/deploy-althing-chamber.yaml \
--var ref=${{ github.sha }}
+1 -1
View File
@@ -120,7 +120,7 @@ corviduo-dev remains monitored but its alert policy was not changed.
Notification URL:
`generic://10.100.10.50:8096/beszel?disabletls=yes&template=json`
The bridge at `services/beszel-althing/` forwards through `postbox` to the
The bridge at `services/althing-alert-bridge/` forwards through `postbox` to the
**infra-ops inbox**, as the operator requested. Existing unused email delivery
was replaced with this verified route. Miranda is a later cutover, not enabled.
See that service's README for operation and recipient changes.
+16 -4
View File
@@ -17,10 +17,22 @@
# 2026-09-21. A seed that ships red on day one teaches everyone to ignore the
# board, which is how you end up with a monitor nobody reads.
#
# NOT SEEDED, deliberately: "althing chamber" (10.250.50.70:7881) is on the
# Homepage dashboard but refuses connections right now. It is either genuinely
# down or retired; seeding it would put a red row on a brand-new board before
# anyone has decided which. Resolve it, then add it.
# "althing chamber" was a candidate and was RETIRED instead (operator,
# 2026-09-21): three of its four containers had never started since being
# created on 2026-09-19, so :7881 refused. Stack, host dirs and image removed.
# ---- where alerts GO ---------------------------------------------------------
# Seeded BEFORE the monitors, and `applyExisting` attaches the channel to rows
# that already exist. A board that detects and notifies nobody is precisely the
# failure this service layer was built to close -- Homepage's own healthcheck
# caught its 2026-09-18 death correctly and nothing was subscribed.
#
# Route: Kuma -> althing-alert-bridge (/kuma) -> postbox -> infra-ops inbox.
# Same path Beszel uses, different route, so the subject says which tool spoke:
# "[Uptime Kuma] Homepage is DOWN" rather than a Beszel-labelled lie.
notifications:
- name: althing (infra-ops)
webhookURL: http://10.100.10.50:8096/kuma
monitors:
# ---- fleet toolchain: dead = agents and the operator are blocked ----