Wire Beszel fleet filesystems, GPU telemetry, dashboard and alerts

2026-09-10 08:35:28 -07:00
parent 20bbb95113
commit eb75713c1b
18 changed files with 438 additions and 84 deletions
@@ -0,0 +1,46 @@
 # Beszel fleet wiring — 2026-09-10
 Operator requested `/tmp/beszel.md` handoff execution, selected **infra-ops inbox**
 for alerts (Miranda later), approved creation of a dedicated monitoring superuser,
 and asked for GPU usage/power telemetry and the card's health detail.
 Completed: all seven requested hosts up, alongside previously registered
 corviduo-dev (8/8). nh3-docker revived; nh3-dev added. vm-esh-nas was already up,
 contrary to the handoff; access is lkraven, not infra-ops. Irvine's agent was
 healthy but its hub record still pointed at retired 10.100.79.3; fixed to
 100.64.0.6. althing-post-office container remained up throughout.
 Docker agents need bind mounts, not merely EXTRA_FILESYSTEMS=/tank. Host
 overrides under stacks/beszel/hosts provide read-only mounts. Existing project
 directories/volumes preserved using new deploy-stack options DEPLOY_DEST_STACK
 and DEPLOY_SUDO=1. nh3-dev uses legacy docker-compose and needed the external
 traefik-net network even with agent-only profile. No host Docker upgrade.
 ana-ml2 tank: 4548.68 / 8791.46 GiB (~51.7%). ana-docker root: ~83.1%, close
 to 85% disk warning. irv-ml1 storetank ~77.4%. NVIDIA agent 0.18.7 on both
 GPU hosts reports all four cards' utilization, VRAM and watts. No GPU power
 limits or serving workloads changed. GPU watts do not size a whole-host PSU.
 Homepage uses existing discovery labels and version-2 widget; verified one
 card and live authenticated data. This overview shows systems/up only;
 reachability is not a degraded-health score. Per-system widget can expose
 CPU/memory/root disk/network; hub charts contain the additional disks and GPUs.
 Approved dedicated PocketBase superuser beszel-monitoring@phasefinal.com,
 Vaultwarden ana-docker/beszel-monitoring; Homepage live .env contains its
 credential, labels only placeholders. Existing operator login unchanged.
 Thirty rules: disk >85% for 5m, CPU >95% for 15m, memory >90% for 10m,
 offline 2m on all seven, temperature >85C for 5m on GPU hosts. Existing unused
 email route replaced with verified webhook. nh3-dev system service
 beszel-althing forwards JSON via supported postbox CLI, sender/recipient
 infra-ops; configurable recipient for later Miranda move. See service README.
 Real alert test: ana-ml2 Disk 1%/1m fired at 15:29:45Z into althing thread
 01M25Z0WFDJM92GPTJQF769HJ7, receipt confirmed infra-ops reachable. Restored
 85%/5m afterward. Fixed hub appURL from localhost to 10.250.50.70:8090 for
 clickable alert deep links. Inbox verification did not mark mail read.
 Still separate: ZFS degradation/SMART/scrubs and independent hub/bridge/post
 office outage detection. Bridge deliberately has no hidden delivery queue;
 downstream failure is logged and HTTP 502, not a claimed delivery.
@@ -9,6 +9,8 @@ _Last updated: 2026-09-10 07:05 PT (**Pfish-6** = run-6 NVFP4 is the standing se
 ## Repo purpose
 - **2026-09-10 Beszel fleet wiring:** all seven requested hosts plus existing corviduo-dev report up. `/tank` and other data filesystems now have real usage metrics; NVIDIA telemetry covers ana-ml2 and irv-ml1. Thirty alerts deliver to **infra-ops**, explicitly chosen by operator; Miranda routing is deferred. A real low-threshold disk alert reached althing, then the threshold was restored to 85%/5 min. Homepage has one native overview widget (reachability counts, not degraded health). Dedicated superuser approved and stored in Vaultwarden. See `persistent-memory.d/2026-09-10-beszel-fleet-wiring.md` and `stacks/beszel/README.md`.
 Reference workspace for PFI infrastructure: server inventory, canonical
 Docker Compose stacks, ops playbooks, and conventions. Authoritative
 copies of compose files live on the servers under
@@ -0,0 +1,20 @@
 steps:
  - name: Install Beszel alert bridge
    sudo: true
    upload:
      src: services/beszel-althing/bridge.py
      dest: /opt/beszel-althing/bridge.py
      mode: '0644'
  - name: Install Beszel alert bridge unit
    sudo: true
    upload:
      src: services/beszel-althing/beszel-althing.service
      dest: /etc/systemd/system/beszel-althing.service
      mode: '0644'
  - name: Start Beszel alert bridge
    sudo: true
    shell: systemctl daemon-reload && systemctl enable beszel-althing.service && systemctl restart beszel-althing.service
 verify:
  - name: Verify bridge process
    shell: systemctl is-active beszel-althing.service && curl --retry 5 --retry-connrefused --retry-delay 1 -fsS http://10.100.10.50:8096/healthz
    changed_when: 'false'
@@ -0,0 +1,22 @@
 steps:
  - name: Back up agent environment before filesystem wiring
    sudo: true
    shell: cp -p {{ stack_dir }}/.env {{ stack_dir }}/.env.before-fleet-wiring-20260910
    creates: '{{ stack_dir }}/.env.before-fleet-wiring-20260910'
  - name: Select canonical host filesystem override
    sudo: true
    shell: |
      set -euo pipefail
      cd {{ stack_dir }}
      sed -i '/^COMPOSE_FILE=/d; /^BESZEL_EXTRA_FS=/d' .env
      printf '\nCOMPOSE_FILE=compose.yaml:hosts/{{ host_name }}.yaml\nBESZEL_EXTRA_FS={{ extra_fs }}\n' >> .env
      docker compose config --quiet
      docker compose up -d --pull never beszel-agent
 verify:
  - name: Verify running mounts and process health
    sudo: true
    shell: |
      docker inspect beszel-agent --format '{{json .Mounts}}'
      docker inspect beszel-agent | python3 -c 'import json,sys; print([e for e in json.load(sys.stdin)[0]["Config"]["Env"] if e.startswith("EXTRA_FILESYSTEMS=")])'
      docker exec beszel-agent /agent health
    changed_when: 'false'
@@ -25,6 +25,9 @@
 #   scripts/deploy-stack.sh <host> <stack> --yes     # skip prompt (use sparingly)
 #   scripts/deploy-stack.sh <host> <stack> --compose # push only compose side
 #   scripts/deploy-stack.sh <host> <stack> --conf    # push only conf side
 # Optional environment:
 #   DEPLOY_DEST_STACK=<name>  retain a legacy remote stack directory/project
 #   DEPLOY_SUDO=1            use passwordless sudo for remote files and rsync
 set -euo pipefail
@@ -67,6 +70,7 @@ STACK=
 ASSUME_YES=0
 DO_COMPOSE=1
 DO_CONF=1
 DEST_STACK=${DEPLOY_DEST_STACK:-}
 for a in "$@"; do
  case "$a" in
    --yes|-y)   ASSUME_YES=1 ;;
@@ -85,6 +89,8 @@ done
 [ -n "$HOST"  ] || { echo "usage: $(basename "$0") <host> <stack>" >&2; exit 2; }
 [ -n "$STACK" ] || { echo "usage: $(basename "$0") <host> <stack>" >&2; exit 2; }
 DEST_STACK=${DEST_STACK:-$STACK}
 [[ "$DEST_STACK" =~ ^[a-zA-Z0-9][a-zA-Z0-9_-]*$ ]] || { echo "invalid DEPLOY_DEST_STACK" >&2; exit 2; }
 resolve_target() {
  # ssh-target file wins when present (may carry user@ or non-default port);
@@ -105,6 +111,12 @@ resolve_target() {
 }
 TARGET=$(resolve_target "$HOST")
 REMOTE_PREFIX=
 RSYNC_REMOTE=()
 if [ "${DEPLOY_SUDO:-0}" = 1 ]; then
  REMOTE_PREFIX='sudo -n '
  RSYNC_REMOTE=(--rsync-path='sudo -n rsync')
 fi
 STACK_DIR="$STACKS_DIR/$STACK"
 [ -d "$STACK_DIR" ] || { echo "error: $STACK_DIR not found — author the canonical stack first (see stacks/<other>/ for examples)" >&2; exit 2; }
@@ -112,10 +124,10 @@ STACK_DIR="$STACKS_DIR/$STACK"
 # Collect the two src/dest pairs we need to consider.
 PAIRS=()      # each entry: "<kind>|<src>|<dest>"
 if [ "$DO_COMPOSE" -eq 1 ]; then
-  PAIRS+=("compose|$STACK_DIR/|$TARGET:/opt/docker/compose/$STACK/")
+  PAIRS+=("compose|$STACK_DIR/|$TARGET:/opt/docker/compose/$DEST_STACK/")
 fi
 if [ "$DO_CONF" -eq 1 ] && [ -d "$STACK_DIR/conf" ]; then
-  PAIRS+=("conf|$STACK_DIR/conf/|$TARGET:/opt/docker/conf/$STACK/")
+  PAIRS+=("conf|$STACK_DIR/conf/|$TARGET:/opt/docker/conf/$DEST_STACK/")
 fi
 [ "${#PAIRS[@]}" -gt 0 ] || { echo "nothing to deploy"; exit 0; }
@@ -133,9 +145,9 @@ for entry in "${PAIRS[@]}"; do
  # Pre-create the remote dir. Without this, rsync against a nonexistent
  # destination can fail in ways the dry-run doesn't surface cleanly.
-  remote_path="/opt/docker/$kind/$STACK/"
+  remote_path="/opt/docker/$kind/$DEST_STACK/"
  if ! ssh -n -o BatchMode=yes -o ConnectTimeout=10 "$TARGET" \
-       "mkdir -p '$remote_path'" 2>/dev/null; then
+       "${REMOTE_PREFIX}mkdir -p '$remote_path'" 2>/dev/null; then
    echo "error: could not create $remote_path on $TARGET (check perms / ssh)" >&2
    exit 2
  fi
@@ -143,6 +155,7 @@ for entry in "${PAIRS[@]}"; do
  tmp_out=$(mktemp) tmp_err=$(mktemp)
  rc=0
  rsync -az --delete --dry-run \
      "${RSYNC_REMOTE[@]}" \
      --out-format='%i %n' \
      "${EXCLUDES[@]}" "${extra[@]}" \
      "$src" "$dest" >"$tmp_out" 2>"$tmp_err" || rc=$?
@@ -190,9 +203,9 @@ if [ "$any_change" -eq 0 ]; then
  for entry in "${PAIRS[@]}"; do
    IFS='|' read -r kind _ _ <<<"$entry"
    raw=${RAW_RSYNC_OUT_BY_KIND[$kind]:-}
-    remote_path="/opt/docker/$kind/$STACK/"
+    remote_path="/opt/docker/$kind/$DEST_STACK/"
    remote_count=$(ssh -n -o BatchMode=yes "$TARGET" \
-      "find '$remote_path' -mindepth 1 -maxdepth 1 2>/dev/null | wc -l" \
+      "${REMOTE_PREFIX}find '$remote_path' -mindepth 1 -maxdepth 1 2>/dev/null | wc -l" \
      2>/dev/null || echo "?")
    printf '  %s: remote has %s entries, rsync itemize output:\n' "$kind" "$remote_count"
    if [ -z "$raw" ]; then
@@ -209,7 +222,7 @@ divider() { printf '\n%s\n' "---------------------------------------------------
 for entry in "${PAIRS[@]}"; do
  IFS='|' read -r kind src dest <<<"$entry"
-  remote_base="/opt/docker/$kind/$STACK"
+  remote_base="/opt/docker/$kind/$DEST_STACK"
  changed=${CHANGED_FILES_BY_KIND[$kind]:-}
  deleted=${DELETED_FILES_BY_KIND[$kind]:-}
  [ -z "$changed$deleted" ] && continue
@@ -224,10 +237,10 @@ for entry in "${PAIRS[@]}"; do
    local_file="$src$rel"
    remote_file="$remote_base/$rel"
    divider
-    if ssh -n -o BatchMode=yes "$TARGET" "[ -f '$remote_file' ]" 2>/dev/null; then
+    if ssh -n -o BatchMode=yes "$TARGET" "${REMOTE_PREFIX}test -f '$remote_file'" 2>/dev/null; then
      printf 'MODIFY  %s\n' "$rel"
      diff -u --label "a/$rel (remote)" --label "b/$rel (local)" \
-        <(ssh -n -o BatchMode=yes "$TARGET" "cat '$remote_file'" 2>/dev/null) \
+        <(ssh -n -o BatchMode=yes "$TARGET" "${REMOTE_PREFIX}cat '$remote_file'" 2>/dev/null) \
        "$local_file" || true
    else
      printf 'ADD     %s\n' "$rel"
@@ -242,7 +255,7 @@ for entry in "${PAIRS[@]}"; do
    divider
    printf 'DELETE  %s\n' "$rel"
    diff -u --label "a/$rel (remote)" --label /dev/null \
-      <(ssh -n -o BatchMode=yes "$TARGET" "cat '$remote_file'" 2>/dev/null) \
+      <(ssh -n -o BatchMode=yes "$TARGET" "${REMOTE_PREFIX}cat '$remote_file'" 2>/dev/null) \
      /dev/null || true
  done <<<"$deleted"
 done
@@ -264,6 +277,7 @@ for entry in "${PAIRS[@]}"; do
  [ "$kind" = compose ] && extra+=(--exclude='conf/')
  printf 'pushing %s → %s\n' "$src" "$dest"
  rsync -az --delete \
    "${RSYNC_REMOTE[@]}" \
    "${EXCLUDES[@]}" "${extra[@]}" \
    "$src" "$dest"
 done
@@ -0,0 +1,36 @@
 # Beszel to althing
 `beszel-althing.service` runs on nh3-dev as lkraven, listening at
 `10.100.10.50:8096`. Beszel sends Shoutrrr generic JSON to `/beszel`; the
 bridge invokes the supported `postbox send` command with the body on stdin.
 It uses the established infra-ops automation identity and sends to infra-ops.
 Deploy from the repository root:
 ```sh
 scripts/elway infra-ops@10.100.10.50 --playbook playbooks/beszel-althing.yaml
 ```
 The service accepts requests only from ana-docker and local diagnostic
 addresses. There is no public listener or new althing handle. It reports
 success only after postbox returns a delivery receipt. Failures return HTTP
 502 and appear in the system journal; there is no hidden retry queue. A
 post-office outage can therefore lose an alert, and needs independent health
 monitoring if guaranteed delivery during such outages becomes a requirement.
 ```sh
 systemctl status beszel-althing
 sudo -n journalctl -u beszel-althing --since '1 hour ago'
 curl -fsS http://10.100.10.50:8096/healthz
 ```
 `/healthz` checks the bridge process, not the downstream inbox. End-to-end
 verification requires a real Beszel threshold transition plus its althing
 receipt. The first verified alert is recorded in `stacks/beszel/README.md`.
 To reroute later, change `BESZEL_ALERT_RECIPIENT` in the canonical unit to
 `miranda`, deploy, and trigger another end-to-end test. Leave `ALTHING_HANDLE`
 as infra-ops so the sender remains identifiable as infrastructure automation.
 The operator explicitly chose infra-ops for now.
 Run `python3 -m unittest discover -s services/beszel-althing -p 'test_*.py'`.
@@ -0,0 +1,24 @@
 [Unit]
 Description=Beszel alerts to the althing infra-ops inbox
 After=network-online.target
 Wants=network-online.target
 [Service]
 User=lkraven
 Group=lkraven
 Environment=PYTHONUNBUFFERED=1
 Environment=ALTHING_HANDLE=infra-ops
 Environment=ALTHING_POST_OFFICE=http://10.100.50.40:8390
 Environment=POSTBOX=/home/lkraven/.local/bin/postbox
 Environment=BESZEL_ALERT_RECIPIENT=infra-ops
 Environment=BESZEL_BIND_HOST=10.100.10.50
 Environment=BESZEL_ALLOWED_SOURCES=10.250.50.70,10.100.10.50,127.0.0.1
 ExecStart=/usr/bin/python3 /opt/beszel-althing/bridge.py
 Restart=on-failure
 RestartSec=5
 NoNewPrivileges=true
 ProtectSystem=strict
 PrivateTmp=true
 [Install]
 WantedBy=multi-user.target
@@ -0,0 +1,63 @@
 #!/usr/bin/env python3
 """Receive Beszel's Shoutrrr JSON and deliver through the supported postbox CLI."""
 import json
 import os
 import subprocess
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 def deliver(payload):
    title = payload.get('title', 'Beszel fleet alert')
    message = payload.get('message')
    if not isinstance(title, str) or not isinstance(message, str) or not message.strip():
        raise ValueError('Expected a nonempty message and string title')
    result = subprocess.run(
        [os.environ['POSTBOX'], '--json', 'send', '--to', os.environ['BESZEL_ALERT_RECIPIENT'],
         '--subject', '[Beszel] ' + title],
        input=message + '\n\nHub: http://10.250.50.70:8090\n',
        text=True, capture_output=True, timeout=25,
    )
    if result.returncode:
        raise RuntimeError('postbox delivery failed: ' + result.stderr.strip())
    receipt = json.loads(result.stdout)
    print(json.dumps({'event': 'delivered', 'title': title, 'receipt': receipt}), flush=True)
    return receipt
 class Handler(BaseHTTPRequestHandler):
    def respond(self, status, body):
        data = json.dumps(body).encode()
        self.send_response(status)
        self.send_header('Content-Type', 'application/json')
        self.send_header('Content-Length', str(len(data)))
        self.end_headers()
        self.wfile.write(data)
    def do_GET(self):
        self.respond(200 if self.path == '/healthz' else 404,
                     {'service': 'beszel-althing', 'delivery': 'verified per POST'})
    def do_POST(self):
        if self.path != '/beszel':
            return self.respond(404, {'error': 'Unknown route'})
        if self.client_address[0] not in os.environ['BESZEL_ALLOWED_SOURCES'].split(','):
            return self.respond(403, {'error': 'Source not allowed'})
        try:
            length = int(self.headers.get('Content-Length', '0'))
            if not 0 < length <= 65536:
                raise ValueError('Invalid body size')
            self.connection.settimeout(10)
            payload = json.loads(self.rfile.read(length))
            if not isinstance(payload, dict):
                raise ValueError('Expected JSON object')
            receipt = deliver(payload)
        except (ValueError, TypeError) as exc:
            return self.respond(400, {'error': str(exc)})
        except (OSError, RuntimeError, subprocess.TimeoutExpired) as exc:
            print(json.dumps({'event': 'delivery_failed', 'error': str(exc)}), flush=True)
            return self.respond(502, {'error': 'Althing delivery failed; inspect service journal'})
        self.respond(200, {'delivered': True, 'receipt': receipt})
 if __name__ == '__main__':
    ThreadingHTTPServer((os.environ['BESZEL_BIND_HOST'], int(os.environ.get('BESZEL_BIND_PORT', '8096'))), Handler).serve_forever()
@@ -0,0 +1,30 @@
 import importlib.util
 import os
 from pathlib import Path
 import subprocess
 import unittest
 from unittest.mock import patch
 spec=importlib.util.spec_from_file_location('bridge',Path(__file__).with_name('bridge.py'))
 bridge=importlib.util.module_from_spec(spec);spec.loader.exec_module(bridge)
 class DeliveryTests(unittest.TestCase):
    def setUp(self):
        self.env=patch.dict(os.environ,POSTBOX='/bin/postbox',BESZEL_ALERT_RECIPIENT='infra-ops')
        self.env.start();self.addCleanup(self.env.stop)
    def test_posts_body_as_stdin_and_returns_receipt(self):
        with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],0,'{"message_id": 42}','')) as run:
            self.assertEqual(bridge.deliver({'title':'Disk 85%','message':'tank is full'})['message_id'],42)
            args,kw=run.call_args
            self.assertEqual(args[0],['/bin/postbox','--json','send','--to','infra-ops','--subject','[Beszel] Disk 85%'])
            self.assertIn('tank is full',kw['input'])
    def test_delivery_failure_is_not_success(self):
        with patch.object(bridge.subprocess,'run',return_value=subprocess.CompletedProcess([],2,'','post office unavailable')):
            with self.assertRaises(RuntimeError):bridge.deliver({'message':'disk full'})
    def test_rejects_empty_message_without_sending(self):
        with patch.object(bridge.subprocess,'run') as run:
            with self.assertRaises(ValueError):bridge.deliver({'message':''})
            run.assert_not_called()
 if __name__ == '__main__':
    unittest.main()
@@ -36,8 +36,13 @@ BESZEL_AGENT_PORT=45876
 # Grab it by clicking "Add System" → copy the key shown in the dialog.
 BESZEL_HUB_KEY=
-# Extra filesystems to track beyond the root mount, comma-separated.
+# Docker must bind-mount extra filesystems; select this host's override.
-# Examples:
+# Example for ana-ml2 (see README for the other hosts):
-#   on ana-ml2: /tank
+# COMPOSE_FILE=compose.yaml:hosts/ana-ml2.yaml
-#   on ana-docker: /mnt/backup,/mnt/compose
+# BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home
 # Paths below are INSIDE the container, not unmounted host paths.
 BESZEL_EXTRA_FS=
 # Optional outbound agent authentication (leave blank for SSH-key mode).
 HUB_URL=
 BESZEL_TOKEN=
@@ -1,89 +1,128 @@
-# beszel
+# Beszel fleet monitoring
-Lightweight monitoring — CPU, memory, disk, network, and per-container stats for every Docker host, with alerts over email/webhook. Pairs with Dozzle (logs) on the same server.
+Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the
 2026-09-10 wiring. The hub also retains corviduo-dev's existing registration.
-**Deploys to:**
+| Host | Compose directory under `/opt/docker/compose/` | Additional filesystems |
- **ana-docker** (hub + local agent) — UI at `http://10.250.50.70:8090`
+|---|---|---|
- **ana-ml2** (agent only) — listens on `10.250.50.54:45876`
+| ana-docker | beszel | /mnt/backup |
- **nh3-docker** (agent only, cross-site) — listens on `10.100.50.40:45876`
+| ana-ml2 | beszel-agent-ana | /tank, /home |
- **corviduo-dev** (agent only) — listens on `10.250.50.152:45876`. Compose at `/home/vh/docker/compose/beszel/` (not `/opt/docker/compose/` — see `servers/corviduo-dev/README.md` for why)
+| nh3-docker | beszel-agent-nh3 | none |
 | esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books |
 | irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy |
 | vm-esh-nas | beszel-agent-esh-nas | /mnt/books, /mnt/share, /mnt/music, /mnt/media |
 | nh3-dev | beszel | /mnt/backup, /mnt/smithy |
-Same compose.yaml on each host. Per-host `.env` sets `COMPOSE_PROFILES` to bring up the right combination. Each agent host is added individually in the hub UI.
+Use `infra-ops@<ip>` with passwordless sudo, except vm-esh-nas:
 `lkraven@10.0.50.154` has Docker access. Irvine's hub address is
 `100.64.0.6`; its retired `10.100.79.3` address caused silent loss of monitoring.
-## How hub ↔ agent auth works
+## Filesystems and deployment
-Beszel uses SSH-key-based auth: the hub generates its own keypair on first boot, and each agent must be seeded with the hub's **public key** via the `KEY` env var. Agents listen on a port (default 45876); the hub pulls metrics by connecting to them with that key.
+The canonical source is `stacks/beszel/`. Keep existing Compose project
 names/directories and named volumes to preserve agent identity and history.
 The deployment helper supports `DEPLOY_DEST_STACK` for legacy stack names
 and `DEPLOY_SUDO=1` for root-owned directories. Example:
-Operator flow on first deploy:
+```sh
-
+DEPLOY_SUDO=1 DEPLOY_DEST_STACK=beszel-agent-ana \
-1. Bring up the **hub** on ana-docker with `BESZEL_HUB_KEY=` blank and the agent profile disabled.
+  scripts/deploy-stack.sh infra-ops@10.250.50.54 beszel --compose
 2. Open the UI, create the admin account, click **Add System** — Beszel shows the public key.
 3. Copy the key into `BESZEL_HUB_KEY` in the `.env` on both hosts.
 4. Re-deploy the hub with `COMPOSE_PROFILES=hub,agent` to add the local agent; deploy the agent on ana-ml2.
 5. Back in the UI, **Add System** with `host=127.0.0.1 port=45876` (local) and `host=10.250.50.54 port=45876` (ana-ml2).
 ## Deploy — hub + local agent (ana-docker)
 ```bash
 ssh ana-docker
 sudo mkdir -p /opt/docker/compose/beszel
 sudo chown $USER /opt/docker/compose/beszel
 cd /opt/docker/compose/beszel
 # scp compose.yaml + .env.example, then:
 cp .env.example .env
 # First pass — hub only, no key yet:
 #   COMPOSE_PROFILES=hub
 #   BESZEL_PORT=8090
 docker compose up -d
 # Open http://10.250.50.70:8090 → create admin → click "Add System" →
 # copy the displayed public key into BESZEL_HUB_KEY.
 # Second pass — add the local agent:
 #   COMPOSE_PROFILES=hub,agent
 #   BESZEL_EXTRA_FS=/mnt/backup,/mnt/compose
 docker compose up -d
 ```
-## Deploy — agent (ana-ml2)
+Agents use host-specific overrides selected by live `.env`:
-```bash
+```dotenv
-ssh ana-ml2
+COMPOSE_PROFILES=agent
-sudo mkdir -p /opt/docker/compose/beszel
+COMPOSE_FILE=compose.yaml:hosts/ana-ml2.yaml
-sudo chown $USER /opt/docker/compose/beszel
+BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home
 cd /opt/docker/compose/beszel
 # scp the same compose.yaml + .env.example, then:
 cp .env.example .env
 # Edit to:
 #   COMPOSE_PROFILES=agent
 #   BESZEL_HUB_KEY=<same key as the hub>
 #   BESZEL_EXTRA_FS=/tank
 docker compose up -d
 docker compose logs -f
 ```
-Then in the hub UI, **Add System** with `host=10.250.50.54`, `port=45876`.
+Docker agents need actual read-only bind mounts under `/extra-filesystems`.
 `EXTRA_FILESYSTEMS=/tank` alone does not expose the host filesystem. The
 overrides provide those mounts; the environment refers to the container paths.
 Mounts are observed before deployment. Network filesystems provide usage, not
 block-device I/O counters. Shared ZFS datasets expose their available quota,
 which differs from raw pool allocation and snapshot-inclusive usage.
-## Verify
+After deployment, validate with `docker compose config --quiet`, then
 `docker compose up -d beszel-agent`; restart alone does not apply env or mounts.
 The reusable playbook is `playbooks/beszel-filesystems.yaml` with `stack_dir`,
 `host_name`, and `extra_fs` variables. Environment backups are kept in
 `.env.before-fleet-wiring-20260910` on each host.
-```bash
+nh3-dev has the older `docker-compose` command; use that spelling. It also
-# Hub health
+requires the external `traefik-net` network to exist even for the agent profile.
-curl -s http://10.250.50.70:8090/api/health
+Only the Beszel agent is started there. On other hosts, use `docker compose`.
-# Agent reachable
+Auth supports a hub SSH public key (`BESZEL_HUB_KEY`) or outbound token mode
-ssh ana-docker 'nc -zv 10.250.50.54 45876'
+(`BESZEL_TOKEN` and `HUB_URL`). Existing auth was preserved; nh3-dev uses the
 hub public key. Never copy live tokens into version control.
-# Local agent reachable from hub container
+## GPU telemetry
-docker exec beszel nc -zv host.docker.internal 45876
+
 ana-ml2 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA
 `utility` access to all GPUs. Both hosts already have NVIDIA Container Toolkit.
 This collects per-card utilization, VRAM, temperature, and power draw without
 changing the serving containers or GPU power limits. Verified hub samples
 include both RTX PRO 6000 Blackwell cards, the RTX 3090, and the RTX A6000.
 Power charts are actual GPU watts, not total wall power or a PSU/circuit sizing
 recommendation. Other system components and workload peaks still matter.
 ## Homepage
 The existing Docker-discovered Monitoring card carries the native Beszel
 widget, version 2. No manual Beszel entry is added to services.yaml.
 Leaving `systemId` unset gives the fleet overview (systems/up).
 The dedicated PocketBase superuser is `beszel-monitoring@phasefinal.com`.
 Its credential is stored in Vaultwarden as `ana-docker/beszel-monitoring` and
 in Homepage's live `.env` as `HOMEPAGE_VAR_BESZEL_USERNAME` and
 `HOMEPAGE_VAR_BESZEL_PASSWORD`. Labels contain only Homepage placeholders.
 The operator's existing account was not reset.
 Verify one card and its real widget response:
 ```sh
 curl -fsS http://10.0.50.45:5100/api/services |
  jq '[.[] | .services[]? | select(.name=="Beszel")] | length'
 curl -fsS 'http://10.0.50.45:5100/api/services/proxy?group=Monitoring&service=Beszel&endpoint=systems&index=0' |
  jq '{totalItems, systems: [.items[] | {name,status}]}'
 ```
 ## Sizing / impact
 The agent is ~10 MB RAM and negligible CPU — runs fine alongside anything on ana-ml2 including GPU workloads. Host-mode networking means it has no port conflicts with other stacks as long as `BESZEL_AGENT_PORT` stays unique.
 ## Alerts
-Configured inside the hub UI (Settings → Notifications). Supports email (SMTP), Gotify, ntfy, Discord, Slack, and generic webhooks. Alert rules attach to per-system or global thresholds (CPU, memory, disk, container down, etc.).
+Thirty rules cover the seven hosts above under the existing operator user:
 | Condition | Threshold | Duration |
 |---|---|---|
 | Disk (root or any extra filesystem) | >85% | 5 minutes |
 | CPU | >95% | 15 minutes |
 | Memory | >90% | 10 minutes |
 | Offline | down | 2 minutes |
 | Temperature (ana-ml2 and irv-ml1) | >85 C | 5 minutes |
 CPU thresholds are sustained-load warnings; expected long-running compute may
 need tuning. GPU utilization alone is not an alarm because busy GPUs are normal.
 corviduo-dev remains monitored but its alert policy was not changed.
 Notification URL:
 `generic://10.100.10.50:8096/beszel?disabletls=yes&template=json`
 The bridge at `services/beszel-althing/` forwards through `postbox` to the
 **infra-ops inbox**, as the operator requested. Existing unused email delivery
 was replaced with this verified route. Miranda is a later cutover, not enabled.
 See that service's README for operation and recipient changes.
 Acceptance on 2026-09-10: ana-ml2 Disk was temporarily lowered to 1%/1 minute;
 the real alert reached althing at 15:29:45Z, thread
 `01M25Z0WFDJM92GPTJQF769HJ7`. The threshold was then restored to 85%/5 minutes.
 Verification uses `postbox thread`, which does not consume the inbox.
 This installed configuration monitors filesystem capacity. It does not yet
 wire ZFS pool degradation, SMART, scrubs, or an independent hub-down watchdog.
 Those require separate follow-up; a green usage chart does not attest to pool health.
 References: [additional disks](https://beszel.dev/guide/additional-disks),
 [GPU telemetry](https://beszel.dev/guide/gpu),
 [Homepage widget](https://gethomepage.dev/widgets/services/beszel/).
@@ -41,6 +41,11 @@ services:
      - homepage.icon=mdi-chart-line
      - homepage.description=Server + container monitoring
      - homepage.href=http://10.250.50.70:${BESZEL_PORT}
      - homepage.widget.type=beszel
      - homepage.widget.url=http://10.250.50.70:${BESZEL_PORT}
      - homepage.widget.version=2
      - homepage.widget.username={{HOMEPAGE_VAR_BESZEL_USERNAME}}
      - homepage.widget.password={{HOMEPAGE_VAR_BESZEL_PASSWORD}}
  beszel-agent:
    image: henrygd/beszel-agent:${BESZEL_VERSION}
@@ -0,0 +1,4 @@
 services:
  beszel-agent:
    volumes:
      - /mnt/backup:/extra-filesystems/backup:ro
@@ -0,0 +1,13 @@
 services:
  beszel-agent:
    image: henrygd/beszel-agent-nvidia:0.18.7
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [utility]
    volumes:
      - /tank:/extra-filesystems/tank:ro
      - /home:/extra-filesystems/home:ro
@@ -0,0 +1,5 @@
 services:
  beszel-agent:
    volumes:
      - /mnt/backup:/extra-filesystems/backup:ro
      - /mnt/books:/extra-filesystems/books:ro
@@ -0,0 +1,14 @@
 services:
  beszel-agent:
    image: henrygd/beszel-agent-nvidia:0.18.7
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [utility]
    volumes:
      - /worktank:/extra-filesystems/worktank:ro
      - /storetank:/extra-filesystems/storetank:ro
      - /mnt/smithy:/extra-filesystems/smithy:ro
@@ -0,0 +1,5 @@
 services:
  beszel-agent:
    volumes:
      - /mnt/backup:/extra-filesystems/backup:ro
      - /mnt/smithy:/extra-filesystems/smithy:ro
@@ -0,0 +1,7 @@
 services:
  beszel-agent:
    volumes:
      - /mnt/books:/extra-filesystems/books:ro
      - /mnt/share:/extra-filesystems/share:ro
      - /mnt/music:/extra-filesystems/music:ro
      - /mnt/media:/extra-filesystems/media:ro