vh
6f0a9b9fae
THE BRIDGE. `beszel-althing` hardcoded a "[Beszel] " subject prefix and a
Beszel hub footer from when Beszel was its only caller. Routing Uptime Kuma
through it unchanged would have delivered Kuma outages labelled [Beszel],
pointing the reader at the wrong dashboard -- an alert that lies about its own
source is worse than no alert.
Now a route registry: /beszel and /kuma, each with its own prefix, footer and
payload parser, because the tools do not agree on a shape (Beszel sends
{title, message}; Kuma sends {heartbeat, monitor, msg}). Generalising cost a
dict; a sibling service would have cost a second unit, a second port and a
second thing to notice had died.
Renamed beszel-althing -> althing-alert-bridge with it. A service named after
one consumer that carries two is the invisible coupling that sends a future
session looking in the wrong place.
⚠ /beszel IS FROZEN and this refactor proves it rather than claiming it. The
three original tests were kept BYTE-UNCHANGED -- including the one asserting
the exact postbox argv -- and deliver() still defaults to the Beszel route so
they exercise it. A new test asserts the Kuma footer never leaks into a Beszel
body or vice versa. Verified live after the rename: a real POST to /beszel
landed as "[Beszel] BRIDGE RENAME CHECK" with the correct hub footer, read back
from the thread rather than trusted from the receipt.
Payload shapes are parsed HERE, not via Kuma's custom-webhook-body feature,
because Kuma's notification config lives in its own database -- and that
database was destroyed and rebuilt from scratch hours ago. Anything living only
in a tool's DB is lost on the next rebuild; format knowledge belongs in git,
next to a test.
parse_kuma also handles the monitorless case. testNotification and cert-expiry
alerts carry no monitor and no heartbeat, and the first cut fabricated "unknown
monitor is ?" from them -- caught by sending a real one and reading the subject,
not by the suite. Fixed, pinned, and the earlier test asserting the bad
behaviour was corrected rather than worked around.
KUMA IS NOW WIRED. scripts/kuma gained notification support and the channel is
in monitors.yaml, seeded BEFORE the monitors and with applyExisting, so a
rebuild restores alerting and not just detection. Ground truth from the DB:
13 of 13 monitors carry the channel.
⚠ A THIRD instance of the same class of bug, worth naming: notifications() is
pushed as `notificationList` at LOGIN ONLY -- there is no event to ask with. The
first cut cleared the captured value before waiting, discarding the only copy it
would ever be sent, then blocked for the full timeout and reported an empty
list. That reads exactly like "no channels configured" and is a lie. Same family
as the getMonitorList ack-vs-push trap, different shape.
End-to-end, both shapes, read back from the inbox:
[Uptime Kuma] Homepage is DOWN + target + board link
[Uptime Kuma] althing (infra-ops) Testing (no fabricated subject)
[Beszel] BRIDGE RENAME CHECK + hub footer, unchanged
ALTHING CHAMBER RETIRED (operator). Three of its four containers had never
started -- created 2026-09-19, StartedAt epoch-zero, 0 restarts -- so :7881
refused, and nothing was watching it. Only its valkey was running, on the
project's own network with no external consumer. Stack, compose/build/conf dirs
and the local image removed; the Homepage card went with the label.
6.5 KiB
Beszel fleet monitoring
Priority 2 completed 2026-09-11: pfi-postgres, esh-vm-db, pbs-ana and pbs-nh3
added with 16 alerts. Live samples verified, including both PBS datastores.
Fleet 17/18 up (known fv-ml1 outage). See configs/beszel-agent/PRIORITY2.md.
Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the 2026-09-10 wiring. The hub also retains corviduo-dev's existing registration.
2026-09-11: five native agents added (ana-nas plus pfi-pve, nh3-pve,
esh-pve and esh-pve-nas); 13 systems total and 20 additional alert rules.
See configs/beszel-agent/README.md. Subsequently nh3-nas was installed via
DSM Docker and registered with four alerts (14 registrations total). Recovery
verification at 2026-09-12 01:51Z: all six priority-1 hosts up, Synology live
filesystem samples verified; fleet 13/14 up with known fv-ml1 outage.
| Host | Compose directory under /opt/docker/compose/ |
Additional filesystems |
|---|---|---|
| ana-docker | beszel | /mnt/backup |
| fv-ml1 | beszel-agent-ana | /tank, /home |
| nh3-docker | beszel-agent-nh3 | none |
| esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books |
| irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy |
| vm-esh-nas | beszel-agent-esh-nas | /mnt/books, /mnt/share, /mnt/music, /mnt/media |
| nh3-dev | beszel | /mnt/backup, /mnt/smithy |
Use infra-ops@<ip> with passwordless sudo, except vm-esh-nas:
lkraven@10.0.50.154 has Docker access. Irvine's hub address is
100.64.0.6; its retired 10.100.79.3 address caused silent loss of monitoring.
Filesystems and deployment
The canonical source is stacks/beszel/. Keep existing Compose project
names/directories and named volumes to preserve agent identity and history.
The deployment helper supports DEPLOY_DEST_STACK for legacy stack names
and DEPLOY_SUDO=1 for root-owned directories. Example:
DEPLOY_SUDO=1 DEPLOY_DEST_STACK=beszel-agent-ana \
scripts/deploy-stack.sh infra-ops@10.251.50.54 beszel --compose
Agents use host-specific overrides selected by live .env:
COMPOSE_PROFILES=agent
COMPOSE_FILE=compose.yaml:hosts/fv-ml1.yaml
BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home
Docker agents need actual read-only bind mounts under /extra-filesystems.
EXTRA_FILESYSTEMS=/tank alone does not expose the host filesystem. The
overrides provide those mounts; the environment refers to the container paths.
Mounts are observed before deployment. Network filesystems provide usage, not
block-device I/O counters. Shared ZFS datasets expose their available quota,
which differs from raw pool allocation and snapshot-inclusive usage.
After deployment, validate with docker compose config --quiet, then
docker compose up -d beszel-agent; restart alone does not apply env or mounts.
The reusable playbook is playbooks/beszel-filesystems.yaml with stack_dir,
host_name, and extra_fs variables. Environment backups are kept in
.env.before-fleet-wiring-20260910 on each host.
nh3-dev has the older docker-compose command; use that spelling. It also
requires the external traefik-net network to exist even for the agent profile.
Only the Beszel agent is started there. On other hosts, use docker compose.
Auth supports a hub SSH public key (BESZEL_HUB_KEY) or outbound token mode
(BESZEL_TOKEN and HUB_URL). Existing auth was preserved; nh3-dev uses the
hub public key. Never copy live tokens into version control.
GPU telemetry
fv-ml1 and irv-ml1 use henrygd/beszel-agent-nvidia:0.18.7 with NVIDIA
utility access to all GPUs. Both hosts already have NVIDIA Container Toolkit.
This collects per-card utilization, VRAM, temperature, and power draw without
changing the serving containers or GPU power limits. Verified hub samples
include both RTX PRO 6000 Blackwell cards, the RTX 3090, and the RTX A6000.
Power charts are actual GPU watts, not total wall power or a PSU/circuit sizing recommendation. Other system components and workload peaks still matter.
Homepage
The existing Docker-discovered Monitoring card carries the native Beszel
widget, version 2. No manual Beszel entry is added to services.yaml.
Leaving systemId unset gives the fleet overview (systems/up).
The dedicated PocketBase superuser is beszel-monitoring@phasefinal.com.
Its credential is stored in Vaultwarden as ana-docker/beszel-monitoring and
in Homepage's live .env as HOMEPAGE_VAR_BESZEL_USERNAME and
HOMEPAGE_VAR_BESZEL_PASSWORD. Labels contain only Homepage placeholders.
The operator's existing account was not reset.
Verify one card and its real widget response:
curl -fsS http://10.0.50.45:5100/api/services |
jq '[.[] | .services[]? | select(.name=="Beszel")] | length'
curl -fsS 'http://10.0.50.45:5100/api/services/proxy?group=Monitoring&service=Beszel&endpoint=systems&index=0' |
jq '{totalItems, systems: [.items[] | {name,status}]}'
Alerts
Thirty rules cover the seven hosts above under the existing operator user:
| Condition | Threshold | Duration |
|---|---|---|
| Disk (root or any extra filesystem) | >85% | 5 minutes |
| CPU | >95% | 15 minutes |
| Memory | >90% | 10 minutes |
| Offline | down | 2 minutes |
| Temperature (fv-ml1 and irv-ml1) | >85 C | 5 minutes |
CPU thresholds are sustained-load warnings; expected long-running compute may need tuning. GPU utilization alone is not an alarm because busy GPUs are normal. corviduo-dev remains monitored but its alert policy was not changed.
Notification URL:
generic://10.100.10.50:8096/beszel?disabletls=yes&template=json
The bridge at services/althing-alert-bridge/ forwards through postbox to the
infra-ops inbox, as the operator requested. Existing unused email delivery
was replaced with this verified route. Miranda is a later cutover, not enabled.
See that service's README for operation and recipient changes.
Acceptance on 2026-09-10: fv-ml1 Disk was temporarily lowered to 1%/1 minute;
the real alert reached althing at 15:29:45Z, thread
01M25Z0WFDJM92GPTJQF769HJ7. The threshold was then restored to 85%/5 minutes.
Verification uses postbox thread, which does not consume the inbox.
This installed configuration monitors filesystem capacity. It does not yet wire ZFS pool degradation, SMART, scrubs, or an independent hub-down watchdog. Those require separate follow-up; a green usage chart does not attest to pool health.
References: additional disks, GPU telemetry, Homepage widget.