diff --git a/persistent-memory.d/2026-09-10-beszel-fleet-wiring.md b/persistent-memory.d/2026-09-10-beszel-fleet-wiring.md new file mode 100644 index 0000000..918eae1 --- /dev/null +++ b/persistent-memory.d/2026-09-10-beszel-fleet-wiring.md @@ -0,0 +1,46 @@ +# Beszel fleet wiring — 2026-09-10 + +Operator requested `/tmp/beszel.md` handoff execution, selected **infra-ops inbox** +for alerts (Miranda later), approved creation of a dedicated monitoring superuser, +and asked for GPU usage/power telemetry and the card's health detail. + +Completed: all seven requested hosts up, alongside previously registered +corviduo-dev (8/8). nh3-docker revived; nh3-dev added. vm-esh-nas was already up, +contrary to the handoff; access is lkraven, not infra-ops. Irvine's agent was +healthy but its hub record still pointed at retired 10.100.79.3; fixed to +100.64.0.6. althing-post-office container remained up throughout. + +Docker agents need bind mounts, not merely EXTRA_FILESYSTEMS=/tank. Host +overrides under stacks/beszel/hosts provide read-only mounts. Existing project +directories/volumes preserved using new deploy-stack options DEPLOY_DEST_STACK +and DEPLOY_SUDO=1. nh3-dev uses legacy docker-compose and needed the external +traefik-net network even with agent-only profile. No host Docker upgrade. + +ana-ml2 tank: 4548.68 / 8791.46 GiB (~51.7%). ana-docker root: ~83.1%, close +to 85% disk warning. irv-ml1 storetank ~77.4%. NVIDIA agent 0.18.7 on both +GPU hosts reports all four cards' utilization, VRAM and watts. No GPU power +limits or serving workloads changed. GPU watts do not size a whole-host PSU. + +Homepage uses existing discovery labels and version-2 widget; verified one +card and live authenticated data. This overview shows systems/up only; +reachability is not a degraded-health score. Per-system widget can expose +CPU/memory/root disk/network; hub charts contain the additional disks and GPUs. + +Approved dedicated PocketBase superuser beszel-monitoring@phasefinal.com, +Vaultwarden ana-docker/beszel-monitoring; Homepage live .env contains its +credential, labels only placeholders. Existing operator login unchanged. + +Thirty rules: disk >85% for 5m, CPU >95% for 15m, memory >90% for 10m, +offline 2m on all seven, temperature >85C for 5m on GPU hosts. Existing unused +email route replaced with verified webhook. nh3-dev system service +beszel-althing forwards JSON via supported postbox CLI, sender/recipient +infra-ops; configurable recipient for later Miranda move. See service README. + +Real alert test: ana-ml2 Disk 1%/1m fired at 15:29:45Z into althing thread +01M25Z0WFDJM92GPTJQF769HJ7, receipt confirmed infra-ops reachable. Restored +85%/5m afterward. Fixed hub appURL from localhost to 10.250.50.70:8090 for +clickable alert deep links. Inbox verification did not mark mail read. + +Still separate: ZFS degradation/SMART/scrubs and independent hub/bridge/post +office outage detection. Bridge deliberately has no hidden delivery queue; +downstream failure is logged and HTTP 502, not a claimed delivery. diff --git a/persistent-memory.md b/persistent-memory.md index 247d384..2eb6d75 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -9,6 +9,8 @@ _Last updated: 2026-09-10 07:05 PT (**Pfish-6** = run-6 NVFP4 is the standing se ## Repo purpose +- **2026-09-10 Beszel fleet wiring:** all seven requested hosts plus existing corviduo-dev report up. `/tank` and other data filesystems now have real usage metrics; NVIDIA telemetry covers ana-ml2 and irv-ml1. Thirty alerts deliver to **infra-ops**, explicitly chosen by operator; Miranda routing is deferred. A real low-threshold disk alert reached althing, then the threshold was restored to 85%/5 min. Homepage has one native overview widget (reachability counts, not degraded health). Dedicated superuser approved and stored in Vaultwarden. See `persistent-memory.d/2026-09-10-beszel-fleet-wiring.md` and `stacks/beszel/README.md`. + Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under diff --git a/playbooks/beszel-althing.yaml b/playbooks/beszel-althing.yaml new file mode 100644 index 0000000..36c2760 --- /dev/null +++ b/playbooks/beszel-althing.yaml @@ -0,0 +1,20 @@ +steps: + - name: Install Beszel alert bridge + sudo: true + upload: + src: services/beszel-althing/bridge.py + dest: /opt/beszel-althing/bridge.py + mode: '0644' + - name: Install Beszel alert bridge unit + sudo: true + upload: + src: services/beszel-althing/beszel-althing.service + dest: /etc/systemd/system/beszel-althing.service + mode: '0644' + - name: Start Beszel alert bridge + sudo: true + shell: systemctl daemon-reload && systemctl enable beszel-althing.service && systemctl restart beszel-althing.service +verify: + - name: Verify bridge process + shell: systemctl is-active beszel-althing.service && curl --retry 5 --retry-connrefused --retry-delay 1 -fsS http://10.100.10.50:8096/healthz + changed_when: 'false' diff --git a/playbooks/beszel-filesystems.yaml b/playbooks/beszel-filesystems.yaml new file mode 100644 index 0000000..6e1a476 --- /dev/null +++ b/playbooks/beszel-filesystems.yaml @@ -0,0 +1,22 @@ +steps: + - name: Back up agent environment before filesystem wiring + sudo: true + shell: cp -p {{ stack_dir }}/.env {{ stack_dir }}/.env.before-fleet-wiring-20260910 + creates: '{{ stack_dir }}/.env.before-fleet-wiring-20260910' + - name: Select canonical host filesystem override + sudo: true + shell: | + set -euo pipefail + cd {{ stack_dir }} + sed -i '/^COMPOSE_FILE=/d; /^BESZEL_EXTRA_FS=/d' .env + printf '\nCOMPOSE_FILE=compose.yaml:hosts/{{ host_name }}.yaml\nBESZEL_EXTRA_FS={{ extra_fs }}\n' >> .env + docker compose config --quiet + docker compose up -d --pull never beszel-agent +verify: + - name: Verify running mounts and process health + sudo: true + shell: | + docker inspect beszel-agent --format '{{json .Mounts}}' + docker inspect beszel-agent | python3 -c 'import json,sys; print([e for e in json.load(sys.stdin)[0]["Config"]["Env"] if e.startswith("EXTRA_FILESYSTEMS=")])' + docker exec beszel-agent /agent health + changed_when: 'false' diff --git a/scripts/deploy-stack.sh b/scripts/deploy-stack.sh index f638880..3bb8793 100755 --- a/scripts/deploy-stack.sh +++ b/scripts/deploy-stack.sh @@ -25,6 +25,9 @@ # scripts/deploy-stack.sh --yes # skip prompt (use sparingly) # scripts/deploy-stack.sh --compose # push only compose side # scripts/deploy-stack.sh --conf # push only conf side +# Optional environment: +# DEPLOY_DEST_STACK= retain a legacy remote stack directory/project +# DEPLOY_SUDO=1 use passwordless sudo for remote files and rsync set -euo pipefail @@ -67,6 +70,7 @@ STACK= ASSUME_YES=0 DO_COMPOSE=1 DO_CONF=1 +DEST_STACK=${DEPLOY_DEST_STACK:-} for a in "$@"; do case "$a" in --yes|-y) ASSUME_YES=1 ;; @@ -85,6 +89,8 @@ done [ -n "$HOST" ] || { echo "usage: $(basename "$0") " >&2; exit 2; } [ -n "$STACK" ] || { echo "usage: $(basename "$0") " >&2; exit 2; } +DEST_STACK=${DEST_STACK:-$STACK} +[[ "$DEST_STACK" =~ ^[a-zA-Z0-9][a-zA-Z0-9_-]*$ ]] || { echo "invalid DEPLOY_DEST_STACK" >&2; exit 2; } resolve_target() { # ssh-target file wins when present (may carry user@ or non-default port); @@ -105,6 +111,12 @@ resolve_target() { } TARGET=$(resolve_target "$HOST") +REMOTE_PREFIX= +RSYNC_REMOTE=() +if [ "${DEPLOY_SUDO:-0}" = 1 ]; then + REMOTE_PREFIX='sudo -n ' + RSYNC_REMOTE=(--rsync-path='sudo -n rsync') +fi STACK_DIR="$STACKS_DIR/$STACK" [ -d "$STACK_DIR" ] || { echo "error: $STACK_DIR not found — author the canonical stack first (see stacks// for examples)" >&2; exit 2; } @@ -112,10 +124,10 @@ STACK_DIR="$STACKS_DIR/$STACK" # Collect the two src/dest pairs we need to consider. PAIRS=() # each entry: "||" if [ "$DO_COMPOSE" -eq 1 ]; then - PAIRS+=("compose|$STACK_DIR/|$TARGET:/opt/docker/compose/$STACK/") + PAIRS+=("compose|$STACK_DIR/|$TARGET:/opt/docker/compose/$DEST_STACK/") fi if [ "$DO_CONF" -eq 1 ] && [ -d "$STACK_DIR/conf" ]; then - PAIRS+=("conf|$STACK_DIR/conf/|$TARGET:/opt/docker/conf/$STACK/") + PAIRS+=("conf|$STACK_DIR/conf/|$TARGET:/opt/docker/conf/$DEST_STACK/") fi [ "${#PAIRS[@]}" -gt 0 ] || { echo "nothing to deploy"; exit 0; } @@ -133,9 +145,9 @@ for entry in "${PAIRS[@]}"; do # Pre-create the remote dir. Without this, rsync against a nonexistent # destination can fail in ways the dry-run doesn't surface cleanly. - remote_path="/opt/docker/$kind/$STACK/" + remote_path="/opt/docker/$kind/$DEST_STACK/" if ! ssh -n -o BatchMode=yes -o ConnectTimeout=10 "$TARGET" \ - "mkdir -p '$remote_path'" 2>/dev/null; then + "${REMOTE_PREFIX}mkdir -p '$remote_path'" 2>/dev/null; then echo "error: could not create $remote_path on $TARGET (check perms / ssh)" >&2 exit 2 fi @@ -143,6 +155,7 @@ for entry in "${PAIRS[@]}"; do tmp_out=$(mktemp) tmp_err=$(mktemp) rc=0 rsync -az --delete --dry-run \ + "${RSYNC_REMOTE[@]}" \ --out-format='%i %n' \ "${EXCLUDES[@]}" "${extra[@]}" \ "$src" "$dest" >"$tmp_out" 2>"$tmp_err" || rc=$? @@ -190,9 +203,9 @@ if [ "$any_change" -eq 0 ]; then for entry in "${PAIRS[@]}"; do IFS='|' read -r kind _ _ <<<"$entry" raw=${RAW_RSYNC_OUT_BY_KIND[$kind]:-} - remote_path="/opt/docker/$kind/$STACK/" + remote_path="/opt/docker/$kind/$DEST_STACK/" remote_count=$(ssh -n -o BatchMode=yes "$TARGET" \ - "find '$remote_path' -mindepth 1 -maxdepth 1 2>/dev/null | wc -l" \ + "${REMOTE_PREFIX}find '$remote_path' -mindepth 1 -maxdepth 1 2>/dev/null | wc -l" \ 2>/dev/null || echo "?") printf ' %s: remote has %s entries, rsync itemize output:\n' "$kind" "$remote_count" if [ -z "$raw" ]; then @@ -209,7 +222,7 @@ divider() { printf '\n%s\n' "--------------------------------------------------- for entry in "${PAIRS[@]}"; do IFS='|' read -r kind src dest <<<"$entry" - remote_base="/opt/docker/$kind/$STACK" + remote_base="/opt/docker/$kind/$DEST_STACK" changed=${CHANGED_FILES_BY_KIND[$kind]:-} deleted=${DELETED_FILES_BY_KIND[$kind]:-} [ -z "$changed$deleted" ] && continue @@ -224,10 +237,10 @@ for entry in "${PAIRS[@]}"; do local_file="$src$rel" remote_file="$remote_base/$rel" divider - if ssh -n -o BatchMode=yes "$TARGET" "[ -f '$remote_file' ]" 2>/dev/null; then + if ssh -n -o BatchMode=yes "$TARGET" "${REMOTE_PREFIX}test -f '$remote_file'" 2>/dev/null; then printf 'MODIFY %s\n' "$rel" diff -u --label "a/$rel (remote)" --label "b/$rel (local)" \ - <(ssh -n -o BatchMode=yes "$TARGET" "cat '$remote_file'" 2>/dev/null) \ + <(ssh -n -o BatchMode=yes "$TARGET" "${REMOTE_PREFIX}cat '$remote_file'" 2>/dev/null) \ "$local_file" || true else printf 'ADD %s\n' "$rel" @@ -242,7 +255,7 @@ for entry in "${PAIRS[@]}"; do divider printf 'DELETE %s\n' "$rel" diff -u --label "a/$rel (remote)" --label /dev/null \ - <(ssh -n -o BatchMode=yes "$TARGET" "cat '$remote_file'" 2>/dev/null) \ + <(ssh -n -o BatchMode=yes "$TARGET" "${REMOTE_PREFIX}cat '$remote_file'" 2>/dev/null) \ /dev/null || true done <<<"$deleted" done @@ -264,6 +277,7 @@ for entry in "${PAIRS[@]}"; do [ "$kind" = compose ] && extra+=(--exclude='conf/') printf 'pushing %s → %s\n' "$src" "$dest" rsync -az --delete \ + "${RSYNC_REMOTE[@]}" \ "${EXCLUDES[@]}" "${extra[@]}" \ "$src" "$dest" done diff --git a/services/beszel-althing/README.md b/services/beszel-althing/README.md new file mode 100644 index 0000000..af0709b --- /dev/null +++ b/services/beszel-althing/README.md @@ -0,0 +1,36 @@ +# Beszel to althing + +`beszel-althing.service` runs on nh3-dev as lkraven, listening at +`10.100.10.50:8096`. Beszel sends Shoutrrr generic JSON to `/beszel`; the +bridge invokes the supported `postbox send` command with the body on stdin. +It uses the established infra-ops automation identity and sends to infra-ops. + +Deploy from the repository root: + +```sh +scripts/elway infra-ops@10.100.10.50 --playbook playbooks/beszel-althing.yaml +``` + +The service accepts requests only from ana-docker and local diagnostic +addresses. There is no public listener or new althing handle. It reports +success only after postbox returns a delivery receipt. Failures return HTTP +502 and appear in the system journal; there is no hidden retry queue. A +post-office outage can therefore lose an alert, and needs independent health +monitoring if guaranteed delivery during such outages becomes a requirement. + +```sh +systemctl status beszel-althing +sudo -n journalctl -u beszel-althing --since '1 hour ago' +curl -fsS http://10.100.10.50:8096/healthz +``` + +`/healthz` checks the bridge process, not the downstream inbox. End-to-end +verification requires a real Beszel threshold transition plus its althing +receipt. The first verified alert is recorded in `stacks/beszel/README.md`. + +To reroute later, change `BESZEL_ALERT_RECIPIENT` in the canonical unit to +`miranda`, deploy, and trigger another end-to-end test. Leave `ALTHING_HANDLE` +as infra-ops so the sender remains identifiable as infrastructure automation. +The operator explicitly chose infra-ops for now. + +Run `python3 -m unittest discover -s services/beszel-althing -p 'test_*.py'`. diff --git a/services/beszel-althing/beszel-althing.service b/services/beszel-althing/beszel-althing.service new file mode 100644 index 0000000..9e4d541 --- /dev/null +++ b/services/beszel-althing/beszel-althing.service @@ -0,0 +1,24 @@ +[Unit] +Description=Beszel alerts to the althing infra-ops inbox +After=network-online.target +Wants=network-online.target + +[Service] +User=lkraven +Group=lkraven +Environment=PYTHONUNBUFFERED=1 +Environment=ALTHING_HANDLE=infra-ops +Environment=ALTHING_POST_OFFICE=http://10.100.50.40:8390 +Environment=POSTBOX=/home/lkraven/.local/bin/postbox +Environment=BESZEL_ALERT_RECIPIENT=infra-ops +Environment=BESZEL_BIND_HOST=10.100.10.50 +Environment=BESZEL_ALLOWED_SOURCES=10.250.50.70,10.100.10.50,127.0.0.1 +ExecStart=/usr/bin/python3 /opt/beszel-althing/bridge.py +Restart=on-failure +RestartSec=5 +NoNewPrivileges=true +ProtectSystem=strict +PrivateTmp=true + +[Install] +WantedBy=multi-user.target diff --git a/services/beszel-althing/bridge.py b/services/beszel-althing/bridge.py new file mode 100644 index 0000000..22d389b --- /dev/null +++ b/services/beszel-althing/bridge.py @@ -0,0 +1,63 @@ +#!/usr/bin/env python3 +"""Receive Beszel's Shoutrrr JSON and deliver through the supported postbox CLI.""" +import json +import os +import subprocess +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + + +def deliver(payload): + title = payload.get('title', 'Beszel fleet alert') + message = payload.get('message') + if not isinstance(title, str) or not isinstance(message, str) or not message.strip(): + raise ValueError('Expected a nonempty message and string title') + result = subprocess.run( + [os.environ['POSTBOX'], '--json', 'send', '--to', os.environ['BESZEL_ALERT_RECIPIENT'], + '--subject', '[Beszel] ' + title], + input=message + '\n\nHub: http://10.250.50.70:8090\n', + text=True, capture_output=True, timeout=25, + ) + if result.returncode: + raise RuntimeError('postbox delivery failed: ' + result.stderr.strip()) + receipt = json.loads(result.stdout) + print(json.dumps({'event': 'delivered', 'title': title, 'receipt': receipt}), flush=True) + return receipt + + +class Handler(BaseHTTPRequestHandler): + def respond(self, status, body): + data = json.dumps(body).encode() + self.send_response(status) + self.send_header('Content-Type', 'application/json') + self.send_header('Content-Length', str(len(data))) + self.end_headers() + self.wfile.write(data) + + def do_GET(self): + self.respond(200 if self.path == '/healthz' else 404, + {'service': 'beszel-althing', 'delivery': 'verified per POST'}) + + def do_POST(self): + if self.path != '/beszel': + return self.respond(404, {'error': 'Unknown route'}) + if self.client_address[0] not in os.environ['BESZEL_ALLOWED_SOURCES'].split(','): + return self.respond(403, {'error': 'Source not allowed'}) + try: + length = int(self.headers.get('Content-Length', '0')) + if not 0 < length <= 65536: + raise ValueError('Invalid body size') + self.connection.settimeout(10) + payload = json.loads(self.rfile.read(length)) + if not isinstance(payload, dict): + raise ValueError('Expected JSON object') + receipt = deliver(payload) + except (ValueError, TypeError) as exc: + return self.respond(400, {'error': str(exc)}) + except (OSError, RuntimeError, subprocess.TimeoutExpired) as exc: + print(json.dumps({'event': 'delivery_failed', 'error': str(exc)}), flush=True) + return self.respond(502, {'error': 'Althing delivery failed; inspect service journal'}) + self.respond(200, {'delivered': True, 'receipt': receipt}) + + +if __name__ == '__main__': + ThreadingHTTPServer((os.environ['BESZEL_BIND_HOST'], int(os.environ.get('BESZEL_BIND_PORT', '8096'))), Handler).serve_forever() diff --git a/services/beszel-althing/test_bridge.py b/services/beszel-althing/test_bridge.py new file mode 100644 index 0000000..bd859dd --- /dev/null +++ b/services/beszel-althing/test_bridge.py @@ -0,0 +1,30 @@ +import importlib.util +import os +from pathlib import Path +import subprocess +import unittest +from unittest.mock import patch + +spec=importlib.util.spec_from_file_location('bridge',Path(__file__).with_name('bridge.py')) +bridge=importlib.util.module_from_spec(spec);spec.loader.exec_module(bridge) + +class DeliveryTests(unittest.TestCase): + def setUp(self): + self.env=patch.dict(os.environ,POSTBOX='/bin/postbox',BESZEL_ALERT_RECIPIENT='infra-ops') + self.env.start();self.addCleanup(self.env.stop) + def test_posts_body_as_stdin_and_returns_receipt(self): + with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id": 42}','')) as run: + self.assertEqual(bridge.deliver({'title':'Disk 85%','message':'tank is full'})['message_id'],42) + args,kw=run.call_args + self.assertEqual(args[0],['/bin/postbox','--json','send','--to','infra-ops','--subject','[Beszel] Disk 85%']) + self.assertIn('tank is full',kw['input']) + def test_delivery_failure_is_not_success(self): + with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],2,'','post office unavailable')): + with self.assertRaises(RuntimeError):bridge.deliver({'message':'disk full'}) + def test_rejects_empty_message_without_sending(self): + with patch.object(bridge.subprocess,'run') as run: + with self.assertRaises(ValueError):bridge.deliver({'message':''}) + run.assert_not_called() + +if __name__ == '__main__': + unittest.main() diff --git a/stacks/beszel/.env.example b/stacks/beszel/.env.example index d7e4b86..a53c39c 100644 --- a/stacks/beszel/.env.example +++ b/stacks/beszel/.env.example @@ -36,8 +36,13 @@ BESZEL_AGENT_PORT=45876 # Grab it by clicking "Add System" → copy the key shown in the dialog. BESZEL_HUB_KEY= -# Extra filesystems to track beyond the root mount, comma-separated. -# Examples: -# on ana-ml2: /tank -# on ana-docker: /mnt/backup,/mnt/compose +# Docker must bind-mount extra filesystems; select this host's override. +# Example for ana-ml2 (see README for the other hosts): +# COMPOSE_FILE=compose.yaml:hosts/ana-ml2.yaml +# BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home +# Paths below are INSIDE the container, not unmounted host paths. BESZEL_EXTRA_FS= + +# Optional outbound agent authentication (leave blank for SSH-key mode). +HUB_URL= +BESZEL_TOKEN= diff --git a/stacks/beszel/README.md b/stacks/beszel/README.md index 44b51f3..3573b45 100644 --- a/stacks/beszel/README.md +++ b/stacks/beszel/README.md @@ -1,89 +1,128 @@ -# beszel +# Beszel fleet monitoring -Lightweight monitoring — CPU, memory, disk, network, and per-container stats for every Docker host, with alerts over email/webhook. Pairs with Dozzle (logs) on the same server. +Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the +2026-09-10 wiring. The hub also retains corviduo-dev's existing registration. -**Deploys to:** -- **ana-docker** (hub + local agent) — UI at `http://10.250.50.70:8090` -- **ana-ml2** (agent only) — listens on `10.250.50.54:45876` -- **nh3-docker** (agent only, cross-site) — listens on `10.100.50.40:45876` -- **corviduo-dev** (agent only) — listens on `10.250.50.152:45876`. Compose at `/home/vh/docker/compose/beszel/` (not `/opt/docker/compose/` — see `servers/corviduo-dev/README.md` for why) +| Host | Compose directory under `/opt/docker/compose/` | Additional filesystems | +|---|---|---| +| ana-docker | beszel | /mnt/backup | +| ana-ml2 | beszel-agent-ana | /tank, /home | +| nh3-docker | beszel-agent-nh3 | none | +| esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books | +| irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy | +| vm-esh-nas | beszel-agent-esh-nas | /mnt/books, /mnt/share, /mnt/music, /mnt/media | +| nh3-dev | beszel | /mnt/backup, /mnt/smithy | -Same compose.yaml on each host. Per-host `.env` sets `COMPOSE_PROFILES` to bring up the right combination. Each agent host is added individually in the hub UI. +Use `infra-ops@` with passwordless sudo, except vm-esh-nas: +`lkraven@10.0.50.154` has Docker access. Irvine's hub address is +`100.64.0.6`; its retired `10.100.79.3` address caused silent loss of monitoring. -## How hub ↔ agent auth works +## Filesystems and deployment -Beszel uses SSH-key-based auth: the hub generates its own keypair on first boot, and each agent must be seeded with the hub's **public key** via the `KEY` env var. Agents listen on a port (default 45876); the hub pulls metrics by connecting to them with that key. +The canonical source is `stacks/beszel/`. Keep existing Compose project +names/directories and named volumes to preserve agent identity and history. +The deployment helper supports `DEPLOY_DEST_STACK` for legacy stack names +and `DEPLOY_SUDO=1` for root-owned directories. Example: -Operator flow on first deploy: - -1. Bring up the **hub** on ana-docker with `BESZEL_HUB_KEY=` blank and the agent profile disabled. -2. Open the UI, create the admin account, click **Add System** — Beszel shows the public key. -3. Copy the key into `BESZEL_HUB_KEY` in the `.env` on both hosts. -4. Re-deploy the hub with `COMPOSE_PROFILES=hub,agent` to add the local agent; deploy the agent on ana-ml2. -5. Back in the UI, **Add System** with `host=127.0.0.1 port=45876` (local) and `host=10.250.50.54 port=45876` (ana-ml2). - -## Deploy — hub + local agent (ana-docker) - -```bash -ssh ana-docker -sudo mkdir -p /opt/docker/compose/beszel -sudo chown $USER /opt/docker/compose/beszel -cd /opt/docker/compose/beszel - -# scp compose.yaml + .env.example, then: -cp .env.example .env -# First pass — hub only, no key yet: -# COMPOSE_PROFILES=hub -# BESZEL_PORT=8090 -docker compose up -d - -# Open http://10.250.50.70:8090 → create admin → click "Add System" → -# copy the displayed public key into BESZEL_HUB_KEY. - -# Second pass — add the local agent: -# COMPOSE_PROFILES=hub,agent -# BESZEL_EXTRA_FS=/mnt/backup,/mnt/compose -docker compose up -d +```sh +DEPLOY_SUDO=1 DEPLOY_DEST_STACK=beszel-agent-ana \ + scripts/deploy-stack.sh infra-ops@10.250.50.54 beszel --compose ``` -## Deploy — agent (ana-ml2) +Agents use host-specific overrides selected by live `.env`: -```bash -ssh ana-ml2 -sudo mkdir -p /opt/docker/compose/beszel -sudo chown $USER /opt/docker/compose/beszel -cd /opt/docker/compose/beszel - -# scp the same compose.yaml + .env.example, then: -cp .env.example .env -# Edit to: -# COMPOSE_PROFILES=agent -# BESZEL_HUB_KEY= -# BESZEL_EXTRA_FS=/tank - -docker compose up -d -docker compose logs -f +```dotenv +COMPOSE_PROFILES=agent +COMPOSE_FILE=compose.yaml:hosts/ana-ml2.yaml +BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home ``` -Then in the hub UI, **Add System** with `host=10.250.50.54`, `port=45876`. +Docker agents need actual read-only bind mounts under `/extra-filesystems`. +`EXTRA_FILESYSTEMS=/tank` alone does not expose the host filesystem. The +overrides provide those mounts; the environment refers to the container paths. +Mounts are observed before deployment. Network filesystems provide usage, not +block-device I/O counters. Shared ZFS datasets expose their available quota, +which differs from raw pool allocation and snapshot-inclusive usage. -## Verify +After deployment, validate with `docker compose config --quiet`, then +`docker compose up -d beszel-agent`; restart alone does not apply env or mounts. +The reusable playbook is `playbooks/beszel-filesystems.yaml` with `stack_dir`, +`host_name`, and `extra_fs` variables. Environment backups are kept in +`.env.before-fleet-wiring-20260910` on each host. -```bash -# Hub health -curl -s http://10.250.50.70:8090/api/health +nh3-dev has the older `docker-compose` command; use that spelling. It also +requires the external `traefik-net` network to exist even for the agent profile. +Only the Beszel agent is started there. On other hosts, use `docker compose`. -# Agent reachable -ssh ana-docker 'nc -zv 10.250.50.54 45876' +Auth supports a hub SSH public key (`BESZEL_HUB_KEY`) or outbound token mode +(`BESZEL_TOKEN` and `HUB_URL`). Existing auth was preserved; nh3-dev uses the +hub public key. Never copy live tokens into version control. -# Local agent reachable from hub container -docker exec beszel nc -zv host.docker.internal 45876 +## GPU telemetry + +ana-ml2 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA +`utility` access to all GPUs. Both hosts already have NVIDIA Container Toolkit. +This collects per-card utilization, VRAM, temperature, and power draw without +changing the serving containers or GPU power limits. Verified hub samples +include both RTX PRO 6000 Blackwell cards, the RTX 3090, and the RTX A6000. + +Power charts are actual GPU watts, not total wall power or a PSU/circuit sizing +recommendation. Other system components and workload peaks still matter. + +## Homepage + +The existing Docker-discovered Monitoring card carries the native Beszel +widget, version 2. No manual Beszel entry is added to services.yaml. +Leaving `systemId` unset gives the fleet overview (systems/up). + +The dedicated PocketBase superuser is `beszel-monitoring@phasefinal.com`. +Its credential is stored in Vaultwarden as `ana-docker/beszel-monitoring` and +in Homepage's live `.env` as `HOMEPAGE_VAR_BESZEL_USERNAME` and +`HOMEPAGE_VAR_BESZEL_PASSWORD`. Labels contain only Homepage placeholders. +The operator's existing account was not reset. + +Verify one card and its real widget response: + +```sh +curl -fsS http://10.0.50.45:5100/api/services | + jq '[.[] | .services[]? | select(.name=="Beszel")] | length' +curl -fsS 'http://10.0.50.45:5100/api/services/proxy?group=Monitoring&service=Beszel&endpoint=systems&index=0' | + jq '{totalItems, systems: [.items[] | {name,status}]}' ``` -## Sizing / impact - -The agent is ~10 MB RAM and negligible CPU — runs fine alongside anything on ana-ml2 including GPU workloads. Host-mode networking means it has no port conflicts with other stacks as long as `BESZEL_AGENT_PORT` stays unique. - ## Alerts -Configured inside the hub UI (Settings → Notifications). Supports email (SMTP), Gotify, ntfy, Discord, Slack, and generic webhooks. Alert rules attach to per-system or global thresholds (CPU, memory, disk, container down, etc.). +Thirty rules cover the seven hosts above under the existing operator user: + +| Condition | Threshold | Duration | +|---|---|---| +| Disk (root or any extra filesystem) | >85% | 5 minutes | +| CPU | >95% | 15 minutes | +| Memory | >90% | 10 minutes | +| Offline | down | 2 minutes | +| Temperature (ana-ml2 and irv-ml1) | >85 C | 5 minutes | + +CPU thresholds are sustained-load warnings; expected long-running compute may +need tuning. GPU utilization alone is not an alarm because busy GPUs are normal. +corviduo-dev remains monitored but its alert policy was not changed. + +Notification URL: +`generic://10.100.10.50:8096/beszel?disabletls=yes&template=json` + +The bridge at `services/beszel-althing/` forwards through `postbox` to the +**infra-ops inbox**, as the operator requested. Existing unused email delivery +was replaced with this verified route. Miranda is a later cutover, not enabled. +See that service's README for operation and recipient changes. + +Acceptance on 2026-09-10: ana-ml2 Disk was temporarily lowered to 1%/1 minute; +the real alert reached althing at 15:29:45Z, thread +`01M25Z0WFDJM92GPTJQF769HJ7`. The threshold was then restored to 85%/5 minutes. +Verification uses `postbox thread`, which does not consume the inbox. + +This installed configuration monitors filesystem capacity. It does not yet +wire ZFS pool degradation, SMART, scrubs, or an independent hub-down watchdog. +Those require separate follow-up; a green usage chart does not attest to pool health. + +References: [additional disks](https://beszel.dev/guide/additional-disks), +[GPU telemetry](https://beszel.dev/guide/gpu), +[Homepage widget](https://gethomepage.dev/widgets/services/beszel/). diff --git a/stacks/beszel/compose.yaml b/stacks/beszel/compose.yaml index 8249178..9989b9a 100644 --- a/stacks/beszel/compose.yaml +++ b/stacks/beszel/compose.yaml @@ -41,6 +41,11 @@ services: - homepage.icon=mdi-chart-line - homepage.description=Server + container monitoring - homepage.href=http://10.250.50.70:${BESZEL_PORT} + - homepage.widget.type=beszel + - homepage.widget.url=http://10.250.50.70:${BESZEL_PORT} + - homepage.widget.version=2 + - homepage.widget.username={{HOMEPAGE_VAR_BESZEL_USERNAME}} + - homepage.widget.password={{HOMEPAGE_VAR_BESZEL_PASSWORD}} beszel-agent: image: henrygd/beszel-agent:${BESZEL_VERSION} diff --git a/stacks/beszel/hosts/ana-docker.yaml b/stacks/beszel/hosts/ana-docker.yaml new file mode 100644 index 0000000..2ee7903 --- /dev/null +++ b/stacks/beszel/hosts/ana-docker.yaml @@ -0,0 +1,4 @@ +services: + beszel-agent: + volumes: + - /mnt/backup:/extra-filesystems/backup:ro diff --git a/stacks/beszel/hosts/ana-ml2.yaml b/stacks/beszel/hosts/ana-ml2.yaml new file mode 100644 index 0000000..036d5e5 --- /dev/null +++ b/stacks/beszel/hosts/ana-ml2.yaml @@ -0,0 +1,13 @@ +services: + beszel-agent: + image: henrygd/beszel-agent-nvidia:0.18.7 + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: [utility] + volumes: + - /tank:/extra-filesystems/tank:ro + - /home:/extra-filesystems/home:ro diff --git a/stacks/beszel/hosts/esh-docker-vm.yaml b/stacks/beszel/hosts/esh-docker-vm.yaml new file mode 100644 index 0000000..e4c95c1 --- /dev/null +++ b/stacks/beszel/hosts/esh-docker-vm.yaml @@ -0,0 +1,5 @@ +services: + beszel-agent: + volumes: + - /mnt/backup:/extra-filesystems/backup:ro + - /mnt/books:/extra-filesystems/books:ro diff --git a/stacks/beszel/hosts/irv-ml1.yaml b/stacks/beszel/hosts/irv-ml1.yaml new file mode 100644 index 0000000..a51091b --- /dev/null +++ b/stacks/beszel/hosts/irv-ml1.yaml @@ -0,0 +1,14 @@ +services: + beszel-agent: + image: henrygd/beszel-agent-nvidia:0.18.7 + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: all + capabilities: [utility] + volumes: + - /worktank:/extra-filesystems/worktank:ro + - /storetank:/extra-filesystems/storetank:ro + - /mnt/smithy:/extra-filesystems/smithy:ro diff --git a/stacks/beszel/hosts/nh3-dev.yaml b/stacks/beszel/hosts/nh3-dev.yaml new file mode 100644 index 0000000..8bdd1ac --- /dev/null +++ b/stacks/beszel/hosts/nh3-dev.yaml @@ -0,0 +1,5 @@ +services: + beszel-agent: + volumes: + - /mnt/backup:/extra-filesystems/backup:ro + - /mnt/smithy:/extra-filesystems/smithy:ro diff --git a/stacks/beszel/hosts/vm-esh-nas.yaml b/stacks/beszel/hosts/vm-esh-nas.yaml new file mode 100644 index 0000000..0bce9c9 --- /dev/null +++ b/stacks/beszel/hosts/vm-esh-nas.yaml @@ -0,0 +1,7 @@ +services: + beszel-agent: + volumes: + - /mnt/books:/extra-filesystems/books:ro + - /mnt/share:/extra-filesystems/share:ro + - /mnt/music:/extra-filesystems/music:ro + - /mnt/media:/extra-filesystems/media:ro