|
|
|
@@ -1,89 +1,128 @@
|
|
|
|
|
# beszel
|
|
|
|
|
# Beszel fleet monitoring
|
|
|
|
|
|
|
|
|
|
Lightweight monitoring — CPU, memory, disk, network, and per-container stats for every Docker host, with alerts over email/webhook. Pairs with Dozzle (logs) on the same server.
|
|
|
|
|
Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the
|
|
|
|
|
2026-09-10 wiring. The hub also retains corviduo-dev's existing registration.
|
|
|
|
|
|
|
|
|
|
**Deploys to:**
|
|
|
|
|
- **ana-docker** (hub + local agent) — UI at `http://10.250.50.70:8090`
|
|
|
|
|
- **ana-ml2** (agent only) — listens on `10.250.50.54:45876`
|
|
|
|
|
- **nh3-docker** (agent only, cross-site) — listens on `10.100.50.40:45876`
|
|
|
|
|
- **corviduo-dev** (agent only) — listens on `10.250.50.152:45876`. Compose at `/home/vh/docker/compose/beszel/` (not `/opt/docker/compose/` — see `servers/corviduo-dev/README.md` for why)
|
|
|
|
|
| Host | Compose directory under `/opt/docker/compose/` | Additional filesystems |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| ana-docker | beszel | /mnt/backup |
|
|
|
|
|
| ana-ml2 | beszel-agent-ana | /tank, /home |
|
|
|
|
|
| nh3-docker | beszel-agent-nh3 | none |
|
|
|
|
|
| esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books |
|
|
|
|
|
| irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy |
|
|
|
|
|
| vm-esh-nas | beszel-agent-esh-nas | /mnt/books, /mnt/share, /mnt/music, /mnt/media |
|
|
|
|
|
| nh3-dev | beszel | /mnt/backup, /mnt/smithy |
|
|
|
|
|
|
|
|
|
|
Same compose.yaml on each host. Per-host `.env` sets `COMPOSE_PROFILES` to bring up the right combination. Each agent host is added individually in the hub UI.
|
|
|
|
|
Use `infra-ops@<ip>` with passwordless sudo, except vm-esh-nas:
|
|
|
|
|
`lkraven@10.0.50.154` has Docker access. Irvine's hub address is
|
|
|
|
|
`100.64.0.6`; its retired `10.100.79.3` address caused silent loss of monitoring.
|
|
|
|
|
|
|
|
|
|
## How hub ↔ agent auth works
|
|
|
|
|
## Filesystems and deployment
|
|
|
|
|
|
|
|
|
|
Beszel uses SSH-key-based auth: the hub generates its own keypair on first boot, and each agent must be seeded with the hub's **public key** via the `KEY` env var. Agents listen on a port (default 45876); the hub pulls metrics by connecting to them with that key.
|
|
|
|
|
The canonical source is `stacks/beszel/`. Keep existing Compose project
|
|
|
|
|
names/directories and named volumes to preserve agent identity and history.
|
|
|
|
|
The deployment helper supports `DEPLOY_DEST_STACK` for legacy stack names
|
|
|
|
|
and `DEPLOY_SUDO=1` for root-owned directories. Example:
|
|
|
|
|
|
|
|
|
|
Operator flow on first deploy:
|
|
|
|
|
|
|
|
|
|
1. Bring up the **hub** on ana-docker with `BESZEL_HUB_KEY=` blank and the agent profile disabled.
|
|
|
|
|
2. Open the UI, create the admin account, click **Add System** — Beszel shows the public key.
|
|
|
|
|
3. Copy the key into `BESZEL_HUB_KEY` in the `.env` on both hosts.
|
|
|
|
|
4. Re-deploy the hub with `COMPOSE_PROFILES=hub,agent` to add the local agent; deploy the agent on ana-ml2.
|
|
|
|
|
5. Back in the UI, **Add System** with `host=127.0.0.1 port=45876` (local) and `host=10.250.50.54 port=45876` (ana-ml2).
|
|
|
|
|
|
|
|
|
|
## Deploy — hub + local agent (ana-docker)
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
ssh ana-docker
|
|
|
|
|
sudo mkdir -p /opt/docker/compose/beszel
|
|
|
|
|
sudo chown $USER /opt/docker/compose/beszel
|
|
|
|
|
cd /opt/docker/compose/beszel
|
|
|
|
|
|
|
|
|
|
# scp compose.yaml + .env.example, then:
|
|
|
|
|
cp .env.example .env
|
|
|
|
|
# First pass — hub only, no key yet:
|
|
|
|
|
# COMPOSE_PROFILES=hub
|
|
|
|
|
# BESZEL_PORT=8090
|
|
|
|
|
docker compose up -d
|
|
|
|
|
|
|
|
|
|
# Open http://10.250.50.70:8090 → create admin → click "Add System" →
|
|
|
|
|
# copy the displayed public key into BESZEL_HUB_KEY.
|
|
|
|
|
|
|
|
|
|
# Second pass — add the local agent:
|
|
|
|
|
# COMPOSE_PROFILES=hub,agent
|
|
|
|
|
# BESZEL_EXTRA_FS=/mnt/backup,/mnt/compose
|
|
|
|
|
docker compose up -d
|
|
|
|
|
```sh
|
|
|
|
|
DEPLOY_SUDO=1 DEPLOY_DEST_STACK=beszel-agent-ana \
|
|
|
|
|
scripts/deploy-stack.sh infra-ops@10.250.50.54 beszel --compose
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Deploy — agent (ana-ml2)
|
|
|
|
|
Agents use host-specific overrides selected by live `.env`:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
ssh ana-ml2
|
|
|
|
|
sudo mkdir -p /opt/docker/compose/beszel
|
|
|
|
|
sudo chown $USER /opt/docker/compose/beszel
|
|
|
|
|
cd /opt/docker/compose/beszel
|
|
|
|
|
|
|
|
|
|
# scp the same compose.yaml + .env.example, then:
|
|
|
|
|
cp .env.example .env
|
|
|
|
|
# Edit to:
|
|
|
|
|
# COMPOSE_PROFILES=agent
|
|
|
|
|
# BESZEL_HUB_KEY=<same key as the hub>
|
|
|
|
|
# BESZEL_EXTRA_FS=/tank
|
|
|
|
|
|
|
|
|
|
docker compose up -d
|
|
|
|
|
docker compose logs -f
|
|
|
|
|
```dotenv
|
|
|
|
|
COMPOSE_PROFILES=agent
|
|
|
|
|
COMPOSE_FILE=compose.yaml:hosts/ana-ml2.yaml
|
|
|
|
|
BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Then in the hub UI, **Add System** with `host=10.250.50.54`, `port=45876`.
|
|
|
|
|
Docker agents need actual read-only bind mounts under `/extra-filesystems`.
|
|
|
|
|
`EXTRA_FILESYSTEMS=/tank` alone does not expose the host filesystem. The
|
|
|
|
|
overrides provide those mounts; the environment refers to the container paths.
|
|
|
|
|
Mounts are observed before deployment. Network filesystems provide usage, not
|
|
|
|
|
block-device I/O counters. Shared ZFS datasets expose their available quota,
|
|
|
|
|
which differs from raw pool allocation and snapshot-inclusive usage.
|
|
|
|
|
|
|
|
|
|
## Verify
|
|
|
|
|
After deployment, validate with `docker compose config --quiet`, then
|
|
|
|
|
`docker compose up -d beszel-agent`; restart alone does not apply env or mounts.
|
|
|
|
|
The reusable playbook is `playbooks/beszel-filesystems.yaml` with `stack_dir`,
|
|
|
|
|
`host_name`, and `extra_fs` variables. Environment backups are kept in
|
|
|
|
|
`.env.before-fleet-wiring-20260910` on each host.
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
# Hub health
|
|
|
|
|
curl -s http://10.250.50.70:8090/api/health
|
|
|
|
|
nh3-dev has the older `docker-compose` command; use that spelling. It also
|
|
|
|
|
requires the external `traefik-net` network to exist even for the agent profile.
|
|
|
|
|
Only the Beszel agent is started there. On other hosts, use `docker compose`.
|
|
|
|
|
|
|
|
|
|
# Agent reachable
|
|
|
|
|
ssh ana-docker 'nc -zv 10.250.50.54 45876'
|
|
|
|
|
Auth supports a hub SSH public key (`BESZEL_HUB_KEY`) or outbound token mode
|
|
|
|
|
(`BESZEL_TOKEN` and `HUB_URL`). Existing auth was preserved; nh3-dev uses the
|
|
|
|
|
hub public key. Never copy live tokens into version control.
|
|
|
|
|
|
|
|
|
|
# Local agent reachable from hub container
|
|
|
|
|
docker exec beszel nc -zv host.docker.internal 45876
|
|
|
|
|
## GPU telemetry
|
|
|
|
|
|
|
|
|
|
ana-ml2 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA
|
|
|
|
|
`utility` access to all GPUs. Both hosts already have NVIDIA Container Toolkit.
|
|
|
|
|
This collects per-card utilization, VRAM, temperature, and power draw without
|
|
|
|
|
changing the serving containers or GPU power limits. Verified hub samples
|
|
|
|
|
include both RTX PRO 6000 Blackwell cards, the RTX 3090, and the RTX A6000.
|
|
|
|
|
|
|
|
|
|
Power charts are actual GPU watts, not total wall power or a PSU/circuit sizing
|
|
|
|
|
recommendation. Other system components and workload peaks still matter.
|
|
|
|
|
|
|
|
|
|
## Homepage
|
|
|
|
|
|
|
|
|
|
The existing Docker-discovered Monitoring card carries the native Beszel
|
|
|
|
|
widget, version 2. No manual Beszel entry is added to services.yaml.
|
|
|
|
|
Leaving `systemId` unset gives the fleet overview (systems/up).
|
|
|
|
|
|
|
|
|
|
The dedicated PocketBase superuser is `beszel-monitoring@phasefinal.com`.
|
|
|
|
|
Its credential is stored in Vaultwarden as `ana-docker/beszel-monitoring` and
|
|
|
|
|
in Homepage's live `.env` as `HOMEPAGE_VAR_BESZEL_USERNAME` and
|
|
|
|
|
`HOMEPAGE_VAR_BESZEL_PASSWORD`. Labels contain only Homepage placeholders.
|
|
|
|
|
The operator's existing account was not reset.
|
|
|
|
|
|
|
|
|
|
Verify one card and its real widget response:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
curl -fsS http://10.0.50.45:5100/api/services |
|
|
|
|
|
jq '[.[] | .services[]? | select(.name=="Beszel")] | length'
|
|
|
|
|
curl -fsS 'http://10.0.50.45:5100/api/services/proxy?group=Monitoring&service=Beszel&endpoint=systems&index=0' |
|
|
|
|
|
jq '{totalItems, systems: [.items[] | {name,status}]}'
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Sizing / impact
|
|
|
|
|
|
|
|
|
|
The agent is ~10 MB RAM and negligible CPU — runs fine alongside anything on ana-ml2 including GPU workloads. Host-mode networking means it has no port conflicts with other stacks as long as `BESZEL_AGENT_PORT` stays unique.
|
|
|
|
|
|
|
|
|
|
## Alerts
|
|
|
|
|
|
|
|
|
|
Configured inside the hub UI (Settings → Notifications). Supports email (SMTP), Gotify, ntfy, Discord, Slack, and generic webhooks. Alert rules attach to per-system or global thresholds (CPU, memory, disk, container down, etc.).
|
|
|
|
|
Thirty rules cover the seven hosts above under the existing operator user:
|
|
|
|
|
|
|
|
|
|
| Condition | Threshold | Duration |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| Disk (root or any extra filesystem) | >85% | 5 minutes |
|
|
|
|
|
| CPU | >95% | 15 minutes |
|
|
|
|
|
| Memory | >90% | 10 minutes |
|
|
|
|
|
| Offline | down | 2 minutes |
|
|
|
|
|
| Temperature (ana-ml2 and irv-ml1) | >85 C | 5 minutes |
|
|
|
|
|
|
|
|
|
|
CPU thresholds are sustained-load warnings; expected long-running compute may
|
|
|
|
|
need tuning. GPU utilization alone is not an alarm because busy GPUs are normal.
|
|
|
|
|
corviduo-dev remains monitored but its alert policy was not changed.
|
|
|
|
|
|
|
|
|
|
Notification URL:
|
|
|
|
|
`generic://10.100.10.50:8096/beszel?disabletls=yes&template=json`
|
|
|
|
|
|
|
|
|
|
The bridge at `services/beszel-althing/` forwards through `postbox` to the
|
|
|
|
|
**infra-ops inbox**, as the operator requested. Existing unused email delivery
|
|
|
|
|
was replaced with this verified route. Miranda is a later cutover, not enabled.
|
|
|
|
|
See that service's README for operation and recipient changes.
|
|
|
|
|
|
|
|
|
|
Acceptance on 2026-09-10: ana-ml2 Disk was temporarily lowered to 1%/1 minute;
|
|
|
|
|
the real alert reached althing at 15:29:45Z, thread
|
|
|
|
|
`01M25Z0WFDJM92GPTJQF769HJ7`. The threshold was then restored to 85%/5 minutes.
|
|
|
|
|
Verification uses `postbox thread`, which does not consume the inbox.
|
|
|
|
|
|
|
|
|
|
This installed configuration monitors filesystem capacity. It does not yet
|
|
|
|
|
wire ZFS pool degradation, SMART, scrubs, or an independent hub-down watchdog.
|
|
|
|
|
Those require separate follow-up; a green usage chart does not attest to pool health.
|
|
|
|
|
|
|
|
|
|
References: [additional disks](https://beszel.dev/guide/additional-disks),
|
|
|
|
|
[GPU telemetry](https://beszel.dev/guide/gpu),
|
|
|
|
|
[Homepage widget](https://gethomepage.dev/widgets/services/beszel/).
|
|
|
|
|
