Wire Beszel fleet filesystems, GPU telemetry, dashboard and alerts
This commit is contained in:
@@ -0,0 +1,46 @@
|
||||
# Beszel fleet wiring — 2026-09-10
|
||||
|
||||
Operator requested `/tmp/beszel.md` handoff execution, selected **infra-ops inbox**
|
||||
for alerts (Miranda later), approved creation of a dedicated monitoring superuser,
|
||||
and asked for GPU usage/power telemetry and the card's health detail.
|
||||
|
||||
Completed: all seven requested hosts up, alongside previously registered
|
||||
corviduo-dev (8/8). nh3-docker revived; nh3-dev added. vm-esh-nas was already up,
|
||||
contrary to the handoff; access is lkraven, not infra-ops. Irvine's agent was
|
||||
healthy but its hub record still pointed at retired 10.100.79.3; fixed to
|
||||
100.64.0.6. althing-post-office container remained up throughout.
|
||||
|
||||
Docker agents need bind mounts, not merely EXTRA_FILESYSTEMS=/tank. Host
|
||||
overrides under stacks/beszel/hosts provide read-only mounts. Existing project
|
||||
directories/volumes preserved using new deploy-stack options DEPLOY_DEST_STACK
|
||||
and DEPLOY_SUDO=1. nh3-dev uses legacy docker-compose and needed the external
|
||||
traefik-net network even with agent-only profile. No host Docker upgrade.
|
||||
|
||||
ana-ml2 tank: 4548.68 / 8791.46 GiB (~51.7%). ana-docker root: ~83.1%, close
|
||||
to 85% disk warning. irv-ml1 storetank ~77.4%. NVIDIA agent 0.18.7 on both
|
||||
GPU hosts reports all four cards' utilization, VRAM and watts. No GPU power
|
||||
limits or serving workloads changed. GPU watts do not size a whole-host PSU.
|
||||
|
||||
Homepage uses existing discovery labels and version-2 widget; verified one
|
||||
card and live authenticated data. This overview shows systems/up only;
|
||||
reachability is not a degraded-health score. Per-system widget can expose
|
||||
CPU/memory/root disk/network; hub charts contain the additional disks and GPUs.
|
||||
|
||||
Approved dedicated PocketBase superuser beszel-monitoring@phasefinal.com,
|
||||
Vaultwarden ana-docker/beszel-monitoring; Homepage live .env contains its
|
||||
credential, labels only placeholders. Existing operator login unchanged.
|
||||
|
||||
Thirty rules: disk >85% for 5m, CPU >95% for 15m, memory >90% for 10m,
|
||||
offline 2m on all seven, temperature >85C for 5m on GPU hosts. Existing unused
|
||||
email route replaced with verified webhook. nh3-dev system service
|
||||
beszel-althing forwards JSON via supported postbox CLI, sender/recipient
|
||||
infra-ops; configurable recipient for later Miranda move. See service README.
|
||||
|
||||
Real alert test: ana-ml2 Disk 1%/1m fired at 15:29:45Z into althing thread
|
||||
01M25Z0WFDJM92GPTJQF769HJ7, receipt confirmed infra-ops reachable. Restored
|
||||
85%/5m afterward. Fixed hub appURL from localhost to 10.250.50.70:8090 for
|
||||
clickable alert deep links. Inbox verification did not mark mail read.
|
||||
|
||||
Still separate: ZFS degradation/SMART/scrubs and independent hub/bridge/post
|
||||
office outage detection. Bridge deliberately has no hidden delivery queue;
|
||||
downstream failure is logged and HTTP 502, not a claimed delivery.
|
||||
@@ -9,6 +9,8 @@ _Last updated: 2026-09-10 07:05 PT (**Pfish-6** = run-6 NVFP4 is the standing se
|
||||
|
||||
## Repo purpose
|
||||
|
||||
- **2026-09-10 Beszel fleet wiring:** all seven requested hosts plus existing corviduo-dev report up. `/tank` and other data filesystems now have real usage metrics; NVIDIA telemetry covers ana-ml2 and irv-ml1. Thirty alerts deliver to **infra-ops**, explicitly chosen by operator; Miranda routing is deferred. A real low-threshold disk alert reached althing, then the threshold was restored to 85%/5 min. Homepage has one native overview widget (reachability counts, not degraded health). Dedicated superuser approved and stored in Vaultwarden. See `persistent-memory.d/2026-09-10-beszel-fleet-wiring.md` and `stacks/beszel/README.md`.
|
||||
|
||||
Reference workspace for PFI infrastructure: server inventory, canonical
|
||||
Docker Compose stacks, ops playbooks, and conventions. Authoritative
|
||||
copies of compose files live on the servers under
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
steps:
|
||||
- name: Install Beszel alert bridge
|
||||
sudo: true
|
||||
upload:
|
||||
src: services/beszel-althing/bridge.py
|
||||
dest: /opt/beszel-althing/bridge.py
|
||||
mode: '0644'
|
||||
- name: Install Beszel alert bridge unit
|
||||
sudo: true
|
||||
upload:
|
||||
src: services/beszel-althing/beszel-althing.service
|
||||
dest: /etc/systemd/system/beszel-althing.service
|
||||
mode: '0644'
|
||||
- name: Start Beszel alert bridge
|
||||
sudo: true
|
||||
shell: systemctl daemon-reload && systemctl enable beszel-althing.service && systemctl restart beszel-althing.service
|
||||
verify:
|
||||
- name: Verify bridge process
|
||||
shell: systemctl is-active beszel-althing.service && curl --retry 5 --retry-connrefused --retry-delay 1 -fsS http://10.100.10.50:8096/healthz
|
||||
changed_when: 'false'
|
||||
@@ -0,0 +1,22 @@
|
||||
steps:
|
||||
- name: Back up agent environment before filesystem wiring
|
||||
sudo: true
|
||||
shell: cp -p {{ stack_dir }}/.env {{ stack_dir }}/.env.before-fleet-wiring-20260910
|
||||
creates: '{{ stack_dir }}/.env.before-fleet-wiring-20260910'
|
||||
- name: Select canonical host filesystem override
|
||||
sudo: true
|
||||
shell: |
|
||||
set -euo pipefail
|
||||
cd {{ stack_dir }}
|
||||
sed -i '/^COMPOSE_FILE=/d; /^BESZEL_EXTRA_FS=/d' .env
|
||||
printf '\nCOMPOSE_FILE=compose.yaml:hosts/{{ host_name }}.yaml\nBESZEL_EXTRA_FS={{ extra_fs }}\n' >> .env
|
||||
docker compose config --quiet
|
||||
docker compose up -d --pull never beszel-agent
|
||||
verify:
|
||||
- name: Verify running mounts and process health
|
||||
sudo: true
|
||||
shell: |
|
||||
docker inspect beszel-agent --format '{{json .Mounts}}'
|
||||
docker inspect beszel-agent | python3 -c 'import json,sys; print([e for e in json.load(sys.stdin)[0]["Config"]["Env"] if e.startswith("EXTRA_FILESYSTEMS=")])'
|
||||
docker exec beszel-agent /agent health
|
||||
changed_when: 'false'
|
||||
+24
-10
@@ -25,6 +25,9 @@
|
||||
# scripts/deploy-stack.sh <host> <stack> --yes # skip prompt (use sparingly)
|
||||
# scripts/deploy-stack.sh <host> <stack> --compose # push only compose side
|
||||
# scripts/deploy-stack.sh <host> <stack> --conf # push only conf side
|
||||
# Optional environment:
|
||||
# DEPLOY_DEST_STACK=<name> retain a legacy remote stack directory/project
|
||||
# DEPLOY_SUDO=1 use passwordless sudo for remote files and rsync
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
@@ -67,6 +70,7 @@ STACK=
|
||||
ASSUME_YES=0
|
||||
DO_COMPOSE=1
|
||||
DO_CONF=1
|
||||
DEST_STACK=${DEPLOY_DEST_STACK:-}
|
||||
for a in "$@"; do
|
||||
case "$a" in
|
||||
--yes|-y) ASSUME_YES=1 ;;
|
||||
@@ -85,6 +89,8 @@ done
|
||||
|
||||
[ -n "$HOST" ] || { echo "usage: $(basename "$0") <host> <stack>" >&2; exit 2; }
|
||||
[ -n "$STACK" ] || { echo "usage: $(basename "$0") <host> <stack>" >&2; exit 2; }
|
||||
DEST_STACK=${DEST_STACK:-$STACK}
|
||||
[[ "$DEST_STACK" =~ ^[a-zA-Z0-9][a-zA-Z0-9_-]*$ ]] || { echo "invalid DEPLOY_DEST_STACK" >&2; exit 2; }
|
||||
|
||||
resolve_target() {
|
||||
# ssh-target file wins when present (may carry user@ or non-default port);
|
||||
@@ -105,6 +111,12 @@ resolve_target() {
|
||||
}
|
||||
|
||||
TARGET=$(resolve_target "$HOST")
|
||||
REMOTE_PREFIX=
|
||||
RSYNC_REMOTE=()
|
||||
if [ "${DEPLOY_SUDO:-0}" = 1 ]; then
|
||||
REMOTE_PREFIX='sudo -n '
|
||||
RSYNC_REMOTE=(--rsync-path='sudo -n rsync')
|
||||
fi
|
||||
STACK_DIR="$STACKS_DIR/$STACK"
|
||||
|
||||
[ -d "$STACK_DIR" ] || { echo "error: $STACK_DIR not found — author the canonical stack first (see stacks/<other>/ for examples)" >&2; exit 2; }
|
||||
@@ -112,10 +124,10 @@ STACK_DIR="$STACKS_DIR/$STACK"
|
||||
# Collect the two src/dest pairs we need to consider.
|
||||
PAIRS=() # each entry: "<kind>|<src>|<dest>"
|
||||
if [ "$DO_COMPOSE" -eq 1 ]; then
|
||||
PAIRS+=("compose|$STACK_DIR/|$TARGET:/opt/docker/compose/$STACK/")
|
||||
PAIRS+=("compose|$STACK_DIR/|$TARGET:/opt/docker/compose/$DEST_STACK/")
|
||||
fi
|
||||
if [ "$DO_CONF" -eq 1 ] && [ -d "$STACK_DIR/conf" ]; then
|
||||
PAIRS+=("conf|$STACK_DIR/conf/|$TARGET:/opt/docker/conf/$STACK/")
|
||||
PAIRS+=("conf|$STACK_DIR/conf/|$TARGET:/opt/docker/conf/$DEST_STACK/")
|
||||
fi
|
||||
|
||||
[ "${#PAIRS[@]}" -gt 0 ] || { echo "nothing to deploy"; exit 0; }
|
||||
@@ -133,9 +145,9 @@ for entry in "${PAIRS[@]}"; do
|
||||
|
||||
# Pre-create the remote dir. Without this, rsync against a nonexistent
|
||||
# destination can fail in ways the dry-run doesn't surface cleanly.
|
||||
remote_path="/opt/docker/$kind/$STACK/"
|
||||
remote_path="/opt/docker/$kind/$DEST_STACK/"
|
||||
if ! ssh -n -o BatchMode=yes -o ConnectTimeout=10 "$TARGET" \
|
||||
"mkdir -p '$remote_path'" 2>/dev/null; then
|
||||
"${REMOTE_PREFIX}mkdir -p '$remote_path'" 2>/dev/null; then
|
||||
echo "error: could not create $remote_path on $TARGET (check perms / ssh)" >&2
|
||||
exit 2
|
||||
fi
|
||||
@@ -143,6 +155,7 @@ for entry in "${PAIRS[@]}"; do
|
||||
tmp_out=$(mktemp) tmp_err=$(mktemp)
|
||||
rc=0
|
||||
rsync -az --delete --dry-run \
|
||||
"${RSYNC_REMOTE[@]}" \
|
||||
--out-format='%i %n' \
|
||||
"${EXCLUDES[@]}" "${extra[@]}" \
|
||||
"$src" "$dest" >"$tmp_out" 2>"$tmp_err" || rc=$?
|
||||
@@ -190,9 +203,9 @@ if [ "$any_change" -eq 0 ]; then
|
||||
for entry in "${PAIRS[@]}"; do
|
||||
IFS='|' read -r kind _ _ <<<"$entry"
|
||||
raw=${RAW_RSYNC_OUT_BY_KIND[$kind]:-}
|
||||
remote_path="/opt/docker/$kind/$STACK/"
|
||||
remote_path="/opt/docker/$kind/$DEST_STACK/"
|
||||
remote_count=$(ssh -n -o BatchMode=yes "$TARGET" \
|
||||
"find '$remote_path' -mindepth 1 -maxdepth 1 2>/dev/null | wc -l" \
|
||||
"${REMOTE_PREFIX}find '$remote_path' -mindepth 1 -maxdepth 1 2>/dev/null | wc -l" \
|
||||
2>/dev/null || echo "?")
|
||||
printf ' %s: remote has %s entries, rsync itemize output:\n' "$kind" "$remote_count"
|
||||
if [ -z "$raw" ]; then
|
||||
@@ -209,7 +222,7 @@ divider() { printf '\n%s\n' "---------------------------------------------------
|
||||
|
||||
for entry in "${PAIRS[@]}"; do
|
||||
IFS='|' read -r kind src dest <<<"$entry"
|
||||
remote_base="/opt/docker/$kind/$STACK"
|
||||
remote_base="/opt/docker/$kind/$DEST_STACK"
|
||||
changed=${CHANGED_FILES_BY_KIND[$kind]:-}
|
||||
deleted=${DELETED_FILES_BY_KIND[$kind]:-}
|
||||
[ -z "$changed$deleted" ] && continue
|
||||
@@ -224,10 +237,10 @@ for entry in "${PAIRS[@]}"; do
|
||||
local_file="$src$rel"
|
||||
remote_file="$remote_base/$rel"
|
||||
divider
|
||||
if ssh -n -o BatchMode=yes "$TARGET" "[ -f '$remote_file' ]" 2>/dev/null; then
|
||||
if ssh -n -o BatchMode=yes "$TARGET" "${REMOTE_PREFIX}test -f '$remote_file'" 2>/dev/null; then
|
||||
printf 'MODIFY %s\n' "$rel"
|
||||
diff -u --label "a/$rel (remote)" --label "b/$rel (local)" \
|
||||
<(ssh -n -o BatchMode=yes "$TARGET" "cat '$remote_file'" 2>/dev/null) \
|
||||
<(ssh -n -o BatchMode=yes "$TARGET" "${REMOTE_PREFIX}cat '$remote_file'" 2>/dev/null) \
|
||||
"$local_file" || true
|
||||
else
|
||||
printf 'ADD %s\n' "$rel"
|
||||
@@ -242,7 +255,7 @@ for entry in "${PAIRS[@]}"; do
|
||||
divider
|
||||
printf 'DELETE %s\n' "$rel"
|
||||
diff -u --label "a/$rel (remote)" --label /dev/null \
|
||||
<(ssh -n -o BatchMode=yes "$TARGET" "cat '$remote_file'" 2>/dev/null) \
|
||||
<(ssh -n -o BatchMode=yes "$TARGET" "${REMOTE_PREFIX}cat '$remote_file'" 2>/dev/null) \
|
||||
/dev/null || true
|
||||
done <<<"$deleted"
|
||||
done
|
||||
@@ -264,6 +277,7 @@ for entry in "${PAIRS[@]}"; do
|
||||
[ "$kind" = compose ] && extra+=(--exclude='conf/')
|
||||
printf 'pushing %s → %s\n' "$src" "$dest"
|
||||
rsync -az --delete \
|
||||
"${RSYNC_REMOTE[@]}" \
|
||||
"${EXCLUDES[@]}" "${extra[@]}" \
|
||||
"$src" "$dest"
|
||||
done
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# Beszel to althing
|
||||
|
||||
`beszel-althing.service` runs on nh3-dev as lkraven, listening at
|
||||
`10.100.10.50:8096`. Beszel sends Shoutrrr generic JSON to `/beszel`; the
|
||||
bridge invokes the supported `postbox send` command with the body on stdin.
|
||||
It uses the established infra-ops automation identity and sends to infra-ops.
|
||||
|
||||
Deploy from the repository root:
|
||||
|
||||
```sh
|
||||
scripts/elway infra-ops@10.100.10.50 --playbook playbooks/beszel-althing.yaml
|
||||
```
|
||||
|
||||
The service accepts requests only from ana-docker and local diagnostic
|
||||
addresses. There is no public listener or new althing handle. It reports
|
||||
success only after postbox returns a delivery receipt. Failures return HTTP
|
||||
502 and appear in the system journal; there is no hidden retry queue. A
|
||||
post-office outage can therefore lose an alert, and needs independent health
|
||||
monitoring if guaranteed delivery during such outages becomes a requirement.
|
||||
|
||||
```sh
|
||||
systemctl status beszel-althing
|
||||
sudo -n journalctl -u beszel-althing --since '1 hour ago'
|
||||
curl -fsS http://10.100.10.50:8096/healthz
|
||||
```
|
||||
|
||||
`/healthz` checks the bridge process, not the downstream inbox. End-to-end
|
||||
verification requires a real Beszel threshold transition plus its althing
|
||||
receipt. The first verified alert is recorded in `stacks/beszel/README.md`.
|
||||
|
||||
To reroute later, change `BESZEL_ALERT_RECIPIENT` in the canonical unit to
|
||||
`miranda`, deploy, and trigger another end-to-end test. Leave `ALTHING_HANDLE`
|
||||
as infra-ops so the sender remains identifiable as infrastructure automation.
|
||||
The operator explicitly chose infra-ops for now.
|
||||
|
||||
Run `python3 -m unittest discover -s services/beszel-althing -p 'test_*.py'`.
|
||||
@@ -0,0 +1,24 @@
|
||||
[Unit]
|
||||
Description=Beszel alerts to the althing infra-ops inbox
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
User=lkraven
|
||||
Group=lkraven
|
||||
Environment=PYTHONUNBUFFERED=1
|
||||
Environment=ALTHING_HANDLE=infra-ops
|
||||
Environment=ALTHING_POST_OFFICE=http://10.100.50.40:8390
|
||||
Environment=POSTBOX=/home/lkraven/.local/bin/postbox
|
||||
Environment=BESZEL_ALERT_RECIPIENT=infra-ops
|
||||
Environment=BESZEL_BIND_HOST=10.100.10.50
|
||||
Environment=BESZEL_ALLOWED_SOURCES=10.250.50.70,10.100.10.50,127.0.0.1
|
||||
ExecStart=/usr/bin/python3 /opt/beszel-althing/bridge.py
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
PrivateTmp=true
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,63 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Receive Beszel's Shoutrrr JSON and deliver through the supported postbox CLI."""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
|
||||
def deliver(payload):
|
||||
title = payload.get('title', 'Beszel fleet alert')
|
||||
message = payload.get('message')
|
||||
if not isinstance(title, str) or not isinstance(message, str) or not message.strip():
|
||||
raise ValueError('Expected a nonempty message and string title')
|
||||
result = subprocess.run(
|
||||
[os.environ['POSTBOX'], '--json', 'send', '--to', os.environ['BESZEL_ALERT_RECIPIENT'],
|
||||
'--subject', '[Beszel] ' + title],
|
||||
input=message + '\n\nHub: http://10.250.50.70:8090\n',
|
||||
text=True, capture_output=True, timeout=25,
|
||||
)
|
||||
if result.returncode:
|
||||
raise RuntimeError('postbox delivery failed: ' + result.stderr.strip())
|
||||
receipt = json.loads(result.stdout)
|
||||
print(json.dumps({'event': 'delivered', 'title': title, 'receipt': receipt}), flush=True)
|
||||
return receipt
|
||||
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def respond(self, status, body):
|
||||
data = json.dumps(body).encode()
|
||||
self.send_response(status)
|
||||
self.send_header('Content-Type', 'application/json')
|
||||
self.send_header('Content-Length', str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
def do_GET(self):
|
||||
self.respond(200 if self.path == '/healthz' else 404,
|
||||
{'service': 'beszel-althing', 'delivery': 'verified per POST'})
|
||||
|
||||
def do_POST(self):
|
||||
if self.path != '/beszel':
|
||||
return self.respond(404, {'error': 'Unknown route'})
|
||||
if self.client_address[0] not in os.environ['BESZEL_ALLOWED_SOURCES'].split(','):
|
||||
return self.respond(403, {'error': 'Source not allowed'})
|
||||
try:
|
||||
length = int(self.headers.get('Content-Length', '0'))
|
||||
if not 0 < length <= 65536:
|
||||
raise ValueError('Invalid body size')
|
||||
self.connection.settimeout(10)
|
||||
payload = json.loads(self.rfile.read(length))
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError('Expected JSON object')
|
||||
receipt = deliver(payload)
|
||||
except (ValueError, TypeError) as exc:
|
||||
return self.respond(400, {'error': str(exc)})
|
||||
except (OSError, RuntimeError, subprocess.TimeoutExpired) as exc:
|
||||
print(json.dumps({'event': 'delivery_failed', 'error': str(exc)}), flush=True)
|
||||
return self.respond(502, {'error': 'Althing delivery failed; inspect service journal'})
|
||||
self.respond(200, {'delivered': True, 'receipt': receipt})
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
ThreadingHTTPServer((os.environ['BESZEL_BIND_HOST'], int(os.environ.get('BESZEL_BIND_PORT', '8096'))), Handler).serve_forever()
|
||||
@@ -0,0 +1,30 @@
|
||||
import importlib.util
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
spec=importlib.util.spec_from_file_location('bridge',Path(__file__).with_name('bridge.py'))
|
||||
bridge=importlib.util.module_from_spec(spec);spec.loader.exec_module(bridge)
|
||||
|
||||
class DeliveryTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.env=patch.dict(os.environ,POSTBOX='/bin/postbox',BESZEL_ALERT_RECIPIENT='infra-ops')
|
||||
self.env.start();self.addCleanup(self.env.stop)
|
||||
def test_posts_body_as_stdin_and_returns_receipt(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id": 42}','')) as run:
|
||||
self.assertEqual(bridge.deliver({'title':'Disk 85%','message':'tank is full'})['message_id'],42)
|
||||
args,kw=run.call_args
|
||||
self.assertEqual(args[0],['/bin/postbox','--json','send','--to','infra-ops','--subject','[Beszel] Disk 85%'])
|
||||
self.assertIn('tank is full',kw['input'])
|
||||
def test_delivery_failure_is_not_success(self):
|
||||
with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],2,'','post office unavailable')):
|
||||
with self.assertRaises(RuntimeError):bridge.deliver({'message':'disk full'})
|
||||
def test_rejects_empty_message_without_sending(self):
|
||||
with patch.object(bridge.subprocess,'run') as run:
|
||||
with self.assertRaises(ValueError):bridge.deliver({'message':''})
|
||||
run.assert_not_called()
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -36,8 +36,13 @@ BESZEL_AGENT_PORT=45876
|
||||
# Grab it by clicking "Add System" → copy the key shown in the dialog.
|
||||
BESZEL_HUB_KEY=
|
||||
|
||||
# Extra filesystems to track beyond the root mount, comma-separated.
|
||||
# Examples:
|
||||
# on ana-ml2: /tank
|
||||
# on ana-docker: /mnt/backup,/mnt/compose
|
||||
# Docker must bind-mount extra filesystems; select this host's override.
|
||||
# Example for ana-ml2 (see README for the other hosts):
|
||||
# COMPOSE_FILE=compose.yaml:hosts/ana-ml2.yaml
|
||||
# BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home
|
||||
# Paths below are INSIDE the container, not unmounted host paths.
|
||||
BESZEL_EXTRA_FS=
|
||||
|
||||
# Optional outbound agent authentication (leave blank for SSH-key mode).
|
||||
HUB_URL=
|
||||
BESZEL_TOKEN=
|
||||
|
||||
+109
-70
@@ -1,89 +1,128 @@
|
||||
# beszel
|
||||
# Beszel fleet monitoring
|
||||
|
||||
Lightweight monitoring — CPU, memory, disk, network, and per-container stats for every Docker host, with alerts over email/webhook. Pairs with Dozzle (logs) on the same server.
|
||||
Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the
|
||||
2026-09-10 wiring. The hub also retains corviduo-dev's existing registration.
|
||||
|
||||
**Deploys to:**
|
||||
- **ana-docker** (hub + local agent) — UI at `http://10.250.50.70:8090`
|
||||
- **ana-ml2** (agent only) — listens on `10.250.50.54:45876`
|
||||
- **nh3-docker** (agent only, cross-site) — listens on `10.100.50.40:45876`
|
||||
- **corviduo-dev** (agent only) — listens on `10.250.50.152:45876`. Compose at `/home/vh/docker/compose/beszel/` (not `/opt/docker/compose/` — see `servers/corviduo-dev/README.md` for why)
|
||||
| Host | Compose directory under `/opt/docker/compose/` | Additional filesystems |
|
||||
|---|---|---|
|
||||
| ana-docker | beszel | /mnt/backup |
|
||||
| ana-ml2 | beszel-agent-ana | /tank, /home |
|
||||
| nh3-docker | beszel-agent-nh3 | none |
|
||||
| esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books |
|
||||
| irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy |
|
||||
| vm-esh-nas | beszel-agent-esh-nas | /mnt/books, /mnt/share, /mnt/music, /mnt/media |
|
||||
| nh3-dev | beszel | /mnt/backup, /mnt/smithy |
|
||||
|
||||
Same compose.yaml on each host. Per-host `.env` sets `COMPOSE_PROFILES` to bring up the right combination. Each agent host is added individually in the hub UI.
|
||||
Use `infra-ops@<ip>` with passwordless sudo, except vm-esh-nas:
|
||||
`lkraven@10.0.50.154` has Docker access. Irvine's hub address is
|
||||
`100.64.0.6`; its retired `10.100.79.3` address caused silent loss of monitoring.
|
||||
|
||||
## How hub ↔ agent auth works
|
||||
## Filesystems and deployment
|
||||
|
||||
Beszel uses SSH-key-based auth: the hub generates its own keypair on first boot, and each agent must be seeded with the hub's **public key** via the `KEY` env var. Agents listen on a port (default 45876); the hub pulls metrics by connecting to them with that key.
|
||||
The canonical source is `stacks/beszel/`. Keep existing Compose project
|
||||
names/directories and named volumes to preserve agent identity and history.
|
||||
The deployment helper supports `DEPLOY_DEST_STACK` for legacy stack names
|
||||
and `DEPLOY_SUDO=1` for root-owned directories. Example:
|
||||
|
||||
Operator flow on first deploy:
|
||||
|
||||
1. Bring up the **hub** on ana-docker with `BESZEL_HUB_KEY=` blank and the agent profile disabled.
|
||||
2. Open the UI, create the admin account, click **Add System** — Beszel shows the public key.
|
||||
3. Copy the key into `BESZEL_HUB_KEY` in the `.env` on both hosts.
|
||||
4. Re-deploy the hub with `COMPOSE_PROFILES=hub,agent` to add the local agent; deploy the agent on ana-ml2.
|
||||
5. Back in the UI, **Add System** with `host=127.0.0.1 port=45876` (local) and `host=10.250.50.54 port=45876` (ana-ml2).
|
||||
|
||||
## Deploy — hub + local agent (ana-docker)
|
||||
|
||||
```bash
|
||||
ssh ana-docker
|
||||
sudo mkdir -p /opt/docker/compose/beszel
|
||||
sudo chown $USER /opt/docker/compose/beszel
|
||||
cd /opt/docker/compose/beszel
|
||||
|
||||
# scp compose.yaml + .env.example, then:
|
||||
cp .env.example .env
|
||||
# First pass — hub only, no key yet:
|
||||
# COMPOSE_PROFILES=hub
|
||||
# BESZEL_PORT=8090
|
||||
docker compose up -d
|
||||
|
||||
# Open http://10.250.50.70:8090 → create admin → click "Add System" →
|
||||
# copy the displayed public key into BESZEL_HUB_KEY.
|
||||
|
||||
# Second pass — add the local agent:
|
||||
# COMPOSE_PROFILES=hub,agent
|
||||
# BESZEL_EXTRA_FS=/mnt/backup,/mnt/compose
|
||||
docker compose up -d
|
||||
```sh
|
||||
DEPLOY_SUDO=1 DEPLOY_DEST_STACK=beszel-agent-ana \
|
||||
scripts/deploy-stack.sh infra-ops@10.250.50.54 beszel --compose
|
||||
```
|
||||
|
||||
## Deploy — agent (ana-ml2)
|
||||
Agents use host-specific overrides selected by live `.env`:
|
||||
|
||||
```bash
|
||||
ssh ana-ml2
|
||||
sudo mkdir -p /opt/docker/compose/beszel
|
||||
sudo chown $USER /opt/docker/compose/beszel
|
||||
cd /opt/docker/compose/beszel
|
||||
|
||||
# scp the same compose.yaml + .env.example, then:
|
||||
cp .env.example .env
|
||||
# Edit to:
|
||||
# COMPOSE_PROFILES=agent
|
||||
# BESZEL_HUB_KEY=<same key as the hub>
|
||||
# BESZEL_EXTRA_FS=/tank
|
||||
|
||||
docker compose up -d
|
||||
docker compose logs -f
|
||||
```dotenv
|
||||
COMPOSE_PROFILES=agent
|
||||
COMPOSE_FILE=compose.yaml:hosts/ana-ml2.yaml
|
||||
BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home
|
||||
```
|
||||
|
||||
Then in the hub UI, **Add System** with `host=10.250.50.54`, `port=45876`.
|
||||
Docker agents need actual read-only bind mounts under `/extra-filesystems`.
|
||||
`EXTRA_FILESYSTEMS=/tank` alone does not expose the host filesystem. The
|
||||
overrides provide those mounts; the environment refers to the container paths.
|
||||
Mounts are observed before deployment. Network filesystems provide usage, not
|
||||
block-device I/O counters. Shared ZFS datasets expose their available quota,
|
||||
which differs from raw pool allocation and snapshot-inclusive usage.
|
||||
|
||||
## Verify
|
||||
After deployment, validate with `docker compose config --quiet`, then
|
||||
`docker compose up -d beszel-agent`; restart alone does not apply env or mounts.
|
||||
The reusable playbook is `playbooks/beszel-filesystems.yaml` with `stack_dir`,
|
||||
`host_name`, and `extra_fs` variables. Environment backups are kept in
|
||||
`.env.before-fleet-wiring-20260910` on each host.
|
||||
|
||||
```bash
|
||||
# Hub health
|
||||
curl -s http://10.250.50.70:8090/api/health
|
||||
nh3-dev has the older `docker-compose` command; use that spelling. It also
|
||||
requires the external `traefik-net` network to exist even for the agent profile.
|
||||
Only the Beszel agent is started there. On other hosts, use `docker compose`.
|
||||
|
||||
# Agent reachable
|
||||
ssh ana-docker 'nc -zv 10.250.50.54 45876'
|
||||
Auth supports a hub SSH public key (`BESZEL_HUB_KEY`) or outbound token mode
|
||||
(`BESZEL_TOKEN` and `HUB_URL`). Existing auth was preserved; nh3-dev uses the
|
||||
hub public key. Never copy live tokens into version control.
|
||||
|
||||
# Local agent reachable from hub container
|
||||
docker exec beszel nc -zv host.docker.internal 45876
|
||||
## GPU telemetry
|
||||
|
||||
ana-ml2 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA
|
||||
`utility` access to all GPUs. Both hosts already have NVIDIA Container Toolkit.
|
||||
This collects per-card utilization, VRAM, temperature, and power draw without
|
||||
changing the serving containers or GPU power limits. Verified hub samples
|
||||
include both RTX PRO 6000 Blackwell cards, the RTX 3090, and the RTX A6000.
|
||||
|
||||
Power charts are actual GPU watts, not total wall power or a PSU/circuit sizing
|
||||
recommendation. Other system components and workload peaks still matter.
|
||||
|
||||
## Homepage
|
||||
|
||||
The existing Docker-discovered Monitoring card carries the native Beszel
|
||||
widget, version 2. No manual Beszel entry is added to services.yaml.
|
||||
Leaving `systemId` unset gives the fleet overview (systems/up).
|
||||
|
||||
The dedicated PocketBase superuser is `beszel-monitoring@phasefinal.com`.
|
||||
Its credential is stored in Vaultwarden as `ana-docker/beszel-monitoring` and
|
||||
in Homepage's live `.env` as `HOMEPAGE_VAR_BESZEL_USERNAME` and
|
||||
`HOMEPAGE_VAR_BESZEL_PASSWORD`. Labels contain only Homepage placeholders.
|
||||
The operator's existing account was not reset.
|
||||
|
||||
Verify one card and its real widget response:
|
||||
|
||||
```sh
|
||||
curl -fsS http://10.0.50.45:5100/api/services |
|
||||
jq '[.[] | .services[]? | select(.name=="Beszel")] | length'
|
||||
curl -fsS 'http://10.0.50.45:5100/api/services/proxy?group=Monitoring&service=Beszel&endpoint=systems&index=0' |
|
||||
jq '{totalItems, systems: [.items[] | {name,status}]}'
|
||||
```
|
||||
|
||||
## Sizing / impact
|
||||
|
||||
The agent is ~10 MB RAM and negligible CPU — runs fine alongside anything on ana-ml2 including GPU workloads. Host-mode networking means it has no port conflicts with other stacks as long as `BESZEL_AGENT_PORT` stays unique.
|
||||
|
||||
## Alerts
|
||||
|
||||
Configured inside the hub UI (Settings → Notifications). Supports email (SMTP), Gotify, ntfy, Discord, Slack, and generic webhooks. Alert rules attach to per-system or global thresholds (CPU, memory, disk, container down, etc.).
|
||||
Thirty rules cover the seven hosts above under the existing operator user:
|
||||
|
||||
| Condition | Threshold | Duration |
|
||||
|---|---|---|
|
||||
| Disk (root or any extra filesystem) | >85% | 5 minutes |
|
||||
| CPU | >95% | 15 minutes |
|
||||
| Memory | >90% | 10 minutes |
|
||||
| Offline | down | 2 minutes |
|
||||
| Temperature (ana-ml2 and irv-ml1) | >85 C | 5 minutes |
|
||||
|
||||
CPU thresholds are sustained-load warnings; expected long-running compute may
|
||||
need tuning. GPU utilization alone is not an alarm because busy GPUs are normal.
|
||||
corviduo-dev remains monitored but its alert policy was not changed.
|
||||
|
||||
Notification URL:
|
||||
`generic://10.100.10.50:8096/beszel?disabletls=yes&template=json`
|
||||
|
||||
The bridge at `services/beszel-althing/` forwards through `postbox` to the
|
||||
**infra-ops inbox**, as the operator requested. Existing unused email delivery
|
||||
was replaced with this verified route. Miranda is a later cutover, not enabled.
|
||||
See that service's README for operation and recipient changes.
|
||||
|
||||
Acceptance on 2026-09-10: ana-ml2 Disk was temporarily lowered to 1%/1 minute;
|
||||
the real alert reached althing at 15:29:45Z, thread
|
||||
`01M25Z0WFDJM92GPTJQF769HJ7`. The threshold was then restored to 85%/5 minutes.
|
||||
Verification uses `postbox thread`, which does not consume the inbox.
|
||||
|
||||
This installed configuration monitors filesystem capacity. It does not yet
|
||||
wire ZFS pool degradation, SMART, scrubs, or an independent hub-down watchdog.
|
||||
Those require separate follow-up; a green usage chart does not attest to pool health.
|
||||
|
||||
References: [additional disks](https://beszel.dev/guide/additional-disks),
|
||||
[GPU telemetry](https://beszel.dev/guide/gpu),
|
||||
[Homepage widget](https://gethomepage.dev/widgets/services/beszel/).
|
||||
|
||||
@@ -41,6 +41,11 @@ services:
|
||||
- homepage.icon=mdi-chart-line
|
||||
- homepage.description=Server + container monitoring
|
||||
- homepage.href=http://10.250.50.70:${BESZEL_PORT}
|
||||
- homepage.widget.type=beszel
|
||||
- homepage.widget.url=http://10.250.50.70:${BESZEL_PORT}
|
||||
- homepage.widget.version=2
|
||||
- homepage.widget.username={{HOMEPAGE_VAR_BESZEL_USERNAME}}
|
||||
- homepage.widget.password={{HOMEPAGE_VAR_BESZEL_PASSWORD}}
|
||||
|
||||
beszel-agent:
|
||||
image: henrygd/beszel-agent:${BESZEL_VERSION}
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
services:
|
||||
beszel-agent:
|
||||
volumes:
|
||||
- /mnt/backup:/extra-filesystems/backup:ro
|
||||
@@ -0,0 +1,13 @@
|
||||
services:
|
||||
beszel-agent:
|
||||
image: henrygd/beszel-agent-nvidia:0.18.7
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [utility]
|
||||
volumes:
|
||||
- /tank:/extra-filesystems/tank:ro
|
||||
- /home:/extra-filesystems/home:ro
|
||||
@@ -0,0 +1,5 @@
|
||||
services:
|
||||
beszel-agent:
|
||||
volumes:
|
||||
- /mnt/backup:/extra-filesystems/backup:ro
|
||||
- /mnt/books:/extra-filesystems/books:ro
|
||||
@@ -0,0 +1,14 @@
|
||||
services:
|
||||
beszel-agent:
|
||||
image: henrygd/beszel-agent-nvidia:0.18.7
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [utility]
|
||||
volumes:
|
||||
- /worktank:/extra-filesystems/worktank:ro
|
||||
- /storetank:/extra-filesystems/storetank:ro
|
||||
- /mnt/smithy:/extra-filesystems/smithy:ro
|
||||
@@ -0,0 +1,5 @@
|
||||
services:
|
||||
beszel-agent:
|
||||
volumes:
|
||||
- /mnt/backup:/extra-filesystems/backup:ro
|
||||
- /mnt/smithy:/extra-filesystems/smithy:ro
|
||||
@@ -0,0 +1,7 @@
|
||||
services:
|
||||
beszel-agent:
|
||||
volumes:
|
||||
- /mnt/books:/extra-filesystems/books:ro
|
||||
- /mnt/share:/extra-filesystems/share:ro
|
||||
- /mnt/music:/extra-filesystems/music:ro
|
||||
- /mnt/media:/extra-filesystems/media:ro
|
||||
Reference in New Issue
Block a user