129 lines
5.8 KiB
Markdown
129 lines
5.8 KiB
Markdown
# Beszel fleet monitoring
|
|
|
|
Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the
|
|
2026-09-10 wiring. The hub also retains corviduo-dev's existing registration.
|
|
|
|
| Host | Compose directory under `/opt/docker/compose/` | Additional filesystems |
|
|
|---|---|---|
|
|
| ana-docker | beszel | /mnt/backup |
|
|
| ana-ml2 | beszel-agent-ana | /tank, /home |
|
|
| nh3-docker | beszel-agent-nh3 | none |
|
|
| esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books |
|
|
| irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy |
|
|
| vm-esh-nas | beszel-agent-esh-nas | /mnt/books, /mnt/share, /mnt/music, /mnt/media |
|
|
| nh3-dev | beszel | /mnt/backup, /mnt/smithy |
|
|
|
|
Use `infra-ops@<ip>` with passwordless sudo, except vm-esh-nas:
|
|
`lkraven@10.0.50.154` has Docker access. Irvine's hub address is
|
|
`100.64.0.6`; its retired `10.100.79.3` address caused silent loss of monitoring.
|
|
|
|
## Filesystems and deployment
|
|
|
|
The canonical source is `stacks/beszel/`. Keep existing Compose project
|
|
names/directories and named volumes to preserve agent identity and history.
|
|
The deployment helper supports `DEPLOY_DEST_STACK` for legacy stack names
|
|
and `DEPLOY_SUDO=1` for root-owned directories. Example:
|
|
|
|
```sh
|
|
DEPLOY_SUDO=1 DEPLOY_DEST_STACK=beszel-agent-ana \
|
|
scripts/deploy-stack.sh infra-ops@10.250.50.54 beszel --compose
|
|
```
|
|
|
|
Agents use host-specific overrides selected by live `.env`:
|
|
|
|
```dotenv
|
|
COMPOSE_PROFILES=agent
|
|
COMPOSE_FILE=compose.yaml:hosts/ana-ml2.yaml
|
|
BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home
|
|
```
|
|
|
|
Docker agents need actual read-only bind mounts under `/extra-filesystems`.
|
|
`EXTRA_FILESYSTEMS=/tank` alone does not expose the host filesystem. The
|
|
overrides provide those mounts; the environment refers to the container paths.
|
|
Mounts are observed before deployment. Network filesystems provide usage, not
|
|
block-device I/O counters. Shared ZFS datasets expose their available quota,
|
|
which differs from raw pool allocation and snapshot-inclusive usage.
|
|
|
|
After deployment, validate with `docker compose config --quiet`, then
|
|
`docker compose up -d beszel-agent`; restart alone does not apply env or mounts.
|
|
The reusable playbook is `playbooks/beszel-filesystems.yaml` with `stack_dir`,
|
|
`host_name`, and `extra_fs` variables. Environment backups are kept in
|
|
`.env.before-fleet-wiring-20260910` on each host.
|
|
|
|
nh3-dev has the older `docker-compose` command; use that spelling. It also
|
|
requires the external `traefik-net` network to exist even for the agent profile.
|
|
Only the Beszel agent is started there. On other hosts, use `docker compose`.
|
|
|
|
Auth supports a hub SSH public key (`BESZEL_HUB_KEY`) or outbound token mode
|
|
(`BESZEL_TOKEN` and `HUB_URL`). Existing auth was preserved; nh3-dev uses the
|
|
hub public key. Never copy live tokens into version control.
|
|
|
|
## GPU telemetry
|
|
|
|
ana-ml2 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA
|
|
`utility` access to all GPUs. Both hosts already have NVIDIA Container Toolkit.
|
|
This collects per-card utilization, VRAM, temperature, and power draw without
|
|
changing the serving containers or GPU power limits. Verified hub samples
|
|
include both RTX PRO 6000 Blackwell cards, the RTX 3090, and the RTX A6000.
|
|
|
|
Power charts are actual GPU watts, not total wall power or a PSU/circuit sizing
|
|
recommendation. Other system components and workload peaks still matter.
|
|
|
|
## Homepage
|
|
|
|
The existing Docker-discovered Monitoring card carries the native Beszel
|
|
widget, version 2. No manual Beszel entry is added to services.yaml.
|
|
Leaving `systemId` unset gives the fleet overview (systems/up).
|
|
|
|
The dedicated PocketBase superuser is `beszel-monitoring@phasefinal.com`.
|
|
Its credential is stored in Vaultwarden as `ana-docker/beszel-monitoring` and
|
|
in Homepage's live `.env` as `HOMEPAGE_VAR_BESZEL_USERNAME` and
|
|
`HOMEPAGE_VAR_BESZEL_PASSWORD`. Labels contain only Homepage placeholders.
|
|
The operator's existing account was not reset.
|
|
|
|
Verify one card and its real widget response:
|
|
|
|
```sh
|
|
curl -fsS http://10.0.50.45:5100/api/services |
|
|
jq '[.[] | .services[]? | select(.name=="Beszel")] | length'
|
|
curl -fsS 'http://10.0.50.45:5100/api/services/proxy?group=Monitoring&service=Beszel&endpoint=systems&index=0' |
|
|
jq '{totalItems, systems: [.items[] | {name,status}]}'
|
|
```
|
|
|
|
## Alerts
|
|
|
|
Thirty rules cover the seven hosts above under the existing operator user:
|
|
|
|
| Condition | Threshold | Duration |
|
|
|---|---|---|
|
|
| Disk (root or any extra filesystem) | >85% | 5 minutes |
|
|
| CPU | >95% | 15 minutes |
|
|
| Memory | >90% | 10 minutes |
|
|
| Offline | down | 2 minutes |
|
|
| Temperature (ana-ml2 and irv-ml1) | >85 C | 5 minutes |
|
|
|
|
CPU thresholds are sustained-load warnings; expected long-running compute may
|
|
need tuning. GPU utilization alone is not an alarm because busy GPUs are normal.
|
|
corviduo-dev remains monitored but its alert policy was not changed.
|
|
|
|
Notification URL:
|
|
`generic://10.100.10.50:8096/beszel?disabletls=yes&template=json`
|
|
|
|
The bridge at `services/beszel-althing/` forwards through `postbox` to the
|
|
**infra-ops inbox**, as the operator requested. Existing unused email delivery
|
|
was replaced with this verified route. Miranda is a later cutover, not enabled.
|
|
See that service's README for operation and recipient changes.
|
|
|
|
Acceptance on 2026-09-10: ana-ml2 Disk was temporarily lowered to 1%/1 minute;
|
|
the real alert reached althing at 15:29:45Z, thread
|
|
`01M25Z0WFDJM92GPTJQF769HJ7`. The threshold was then restored to 85%/5 minutes.
|
|
Verification uses `postbox thread`, which does not consume the inbox.
|
|
|
|
This installed configuration monitors filesystem capacity. It does not yet
|
|
wire ZFS pool degradation, SMART, scrubs, or an independent hub-down watchdog.
|
|
Those require separate follow-up; a green usage chart does not attest to pool health.
|
|
|
|
References: [additional disks](https://beszel.dev/guide/additional-disks),
|
|
[GPU telemetry](https://beszel.dev/guide/gpu),
|
|
[Homepage widget](https://gethomepage.dev/widgets/services/beszel/).
|