vh
6d901c0e8d
nh3-ml1's first Temperature alert (2026-09-25 2105, 88.8 C) was nh3-pve's CPU
package during the nightly vzdump; the GPU sat at 47 C. hwmon is not namespaced,
so an LXC's agent reports its host's coretemp sensors under the LXC's name.
- hosts/{nh3,esh}-ml1.yaml: SENSORS=-coretemp_*,acpitz (blacklist); GPU and
NVMe temperatures verified still reported.
- Hub: Temperature >95 C / 5 m on nh3-pve and esh-pve for the CPUs (i9-13900H
TjMax 100 C; nh3-pve 2-hour averages reach 88 C on busy days).
- nh3-pve README: runs warmer than esh-pve; workload confounds it, check
airflow on the next visit.
160 lines
8.0 KiB
Markdown
160 lines
8.0 KiB
Markdown
# Beszel fleet monitoring
|
|
|
|
Priority 2 completed 2026-09-11: pfi-postgres, esh-vm-db, pbs-ana and pbs-nh3
|
|
added with 16 alerts. Live samples verified, including both PBS datastores.
|
|
Fleet 17/18 up (known fv-ml1 outage). See `configs/beszel-agent/PRIORITY2.md`.
|
|
|
|
Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the
|
|
2026-09-10 wiring. The hub also retains corviduo-dev's existing registration.
|
|
|
|
2026-09-11: five native agents added (ana-nas plus pfi-pve, nh3-pve,
|
|
esh-pve and esh-pve-nas); 13 systems total and 20 additional alert rules.
|
|
See `configs/beszel-agent/README.md`. Subsequently nh3-nas was installed via
|
|
DSM Docker and registered with four alerts (14 registrations total). Recovery
|
|
verification at 2026-09-12 01:51Z: all six priority-1 hosts up, Synology live
|
|
filesystem samples verified; fleet 13/14 up with known fv-ml1 outage.
|
|
|
|
| Host | Compose directory under `/opt/docker/compose/` | Additional filesystems |
|
|
|---|---|---|
|
|
| ana-docker | beszel | /mnt/backup |
|
|
| fv-ml1 | beszel-agent-ana | /tank, /home |
|
|
| nh3-docker | beszel-agent-nh3 | none |
|
|
| esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books |
|
|
| irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy |
|
|
| vm-esh-nas | beszel-agent-esh-nas | /mnt/books, /mnt/share, /mnt/music, /mnt/media |
|
|
| nh3-dev | beszel | none since 2026-09-25 (was /mnt/backup, /mnt/smithy — see below) |
|
|
| esh-ml1 (added 2026-09-25) | beszel | none — NVIDIA image (`hosts/esh-ml1.yaml`) for the RTX 2000E Ada |
|
|
| nh3-ml1 (added 2026-09-25) | beszel | none — NVIDIA image (`hosts/nh3-ml1.yaml`) for the RTX 2000E Ada; hub system `1feeeq61g4mkqre`, same 5 alerts as esh-ml1 |
|
|
|
|
⚠ **nh3-dev's agent was DOWN from the 2026-09-24 NH3 power recovery until
|
|
2026-09-25.** Docker could not bind `/mnt/smithy` at boot ("no such device"): since
|
|
`1cbde50` the NAS shares are automounted, and nh3-nas was not up yet. Docker does not
|
|
retry a container that fails to *create*, so `unless-stopped` never brought it back.
|
|
Started by hand, then **fixed the same day**: `hosts/nh3-dev.yaml` no longer binds
|
|
the NAS shares and `BESZEL_EXTRA_FS` is empty. Their capacity is nh3-nas's own
|
|
volume, which nh3-nas's agent reports.
|
|
|
|
Use `infra-ops@<ip>` with passwordless sudo, except vm-esh-nas:
|
|
`lkraven@10.0.50.154` has Docker access. Irvine's hub address is
|
|
`100.64.0.6`; its retired `10.100.79.3` address caused silent loss of monitoring.
|
|
|
|
## Filesystems and deployment
|
|
|
|
The canonical source is `stacks/beszel/`. Keep existing Compose project
|
|
names/directories and named volumes to preserve agent identity and history.
|
|
The deployment helper supports `DEPLOY_DEST_STACK` for legacy stack names
|
|
and `DEPLOY_SUDO=1` for root-owned directories. Example:
|
|
|
|
```sh
|
|
DEPLOY_SUDO=1 DEPLOY_DEST_STACK=beszel-agent-ana \
|
|
scripts/deploy-stack.sh infra-ops@10.251.50.54 beszel --compose
|
|
```
|
|
|
|
Agents use host-specific overrides selected by live `.env`:
|
|
|
|
```dotenv
|
|
COMPOSE_PROFILES=agent
|
|
COMPOSE_FILE=compose.yaml:hosts/fv-ml1.yaml
|
|
BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home
|
|
```
|
|
|
|
Docker agents need actual read-only bind mounts under `/extra-filesystems`.
|
|
`EXTRA_FILESYSTEMS=/tank` alone does not expose the host filesystem. The
|
|
overrides provide those mounts; the environment refers to the container paths.
|
|
Mounts are observed before deployment. Network filesystems provide usage, not
|
|
block-device I/O counters. Shared ZFS datasets expose their available quota,
|
|
which differs from raw pool allocation and snapshot-inclusive usage.
|
|
|
|
After deployment, validate with `docker compose config --quiet`, then
|
|
`docker compose up -d beszel-agent`; restart alone does not apply env or mounts.
|
|
The reusable playbook is `playbooks/beszel-filesystems.yaml` with `stack_dir`,
|
|
`host_name`, and `extra_fs` variables. Environment backups are kept in
|
|
`.env.before-fleet-wiring-20260910` on each host.
|
|
|
|
nh3-dev now has the `docker compose` plugin (v5.5.1, 2026-09-25); the old `docker-compose` binary is gone. It also
|
|
requires the external `traefik-net` network to exist even for the agent profile.
|
|
Only the Beszel agent is started there. On other hosts, use `docker compose`.
|
|
|
|
Auth supports a hub SSH public key (`BESZEL_HUB_KEY`) or outbound token mode
|
|
(`BESZEL_TOKEN` and `HUB_URL`). Existing auth was preserved; nh3-dev uses the
|
|
hub public key. Never copy live tokens into version control.
|
|
|
|
## GPU telemetry
|
|
|
|
fv-ml1 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA
|
|
`utility` access to all GPUs. Both hosts already have NVIDIA Container Toolkit.
|
|
This collects per-card utilization, VRAM, temperature, and power draw without
|
|
changing the serving containers or GPU power limits. Verified hub samples
|
|
include both RTX PRO 6000 Blackwell cards, the RTX 3090, and the RTX A6000.
|
|
|
|
⚠ **A GPU LXC reads its hypervisor's CPU sensors.** hwmon is not namespaced, so
|
|
esh-ml1 and nh3-ml1 would report esh-pve's / nh3-pve's `coretemp_*`, and their
|
|
Temperature alert fired on the HOST CPU under the GPU box's name (nh3-ml1,
|
|
2026-09-25 2105: 88.8 °C = nh3-pve's package during the 2100 vzdump, GPU at 47 °C).
|
|
Their overrides set `SENSORS=-coretemp_*,acpitz` (blacklist mode), which keeps
|
|
the GPU and NVMe temperatures. The CPU alert lives on the hypervisor's own system
|
|
at >95 °C: the i9-13900H's TjMax is 100 °C, and nh3-pve's 2-hour averages have
|
|
reached 88 °C on busy days, so 85 °C would fire on normal work.
|
|
|
|
Power charts are actual GPU watts, not total wall power or a PSU/circuit sizing
|
|
recommendation. Other system components and workload peaks still matter.
|
|
|
|
## Homepage
|
|
|
|
The existing Docker-discovered Monitoring card carries the native Beszel
|
|
widget, version 2. No manual Beszel entry is added to services.yaml.
|
|
Leaving `systemId` unset gives the fleet overview (systems/up).
|
|
|
|
The dedicated PocketBase superuser is `beszel-monitoring@phasefinal.com`.
|
|
Its credential is stored in Vaultwarden as `ana-docker/beszel-monitoring` and
|
|
in Homepage's live `.env` as `HOMEPAGE_VAR_BESZEL_USERNAME` and
|
|
`HOMEPAGE_VAR_BESZEL_PASSWORD`. Labels contain only Homepage placeholders.
|
|
The operator's existing account was not reset.
|
|
|
|
Verify one card and its real widget response:
|
|
|
|
```sh
|
|
curl -fsS http://10.0.50.45:5100/api/services |
|
|
jq '[.[] | .services[]? | select(.name=="Beszel")] | length'
|
|
curl -fsS 'http://10.0.50.45:5100/api/services/proxy?group=Monitoring&service=Beszel&endpoint=systems&index=0' |
|
|
jq '{totalItems, systems: [.items[] | {name,status}]}'
|
|
```
|
|
|
|
## Alerts
|
|
|
|
Thirty rules cover the seven hosts above under the existing operator user:
|
|
|
|
| Condition | Threshold | Duration |
|
|
|---|---|---|
|
|
| Disk (root or any extra filesystem) | >85% | 5 minutes |
|
|
| CPU | >95% | 15 minutes |
|
|
| Memory | >90% | 10 minutes |
|
|
| Offline | down | 2 minutes |
|
|
| Temperature (fv-ml1, irv-ml1, esh-ml1, nh3-ml1) | >85 C | 5 minutes |
|
|
| Temperature, hypervisor CPU (nh3-pve, esh-pve; added 2026-09-25) | >95 C | 5 minutes |
|
|
|
|
CPU thresholds are sustained-load warnings; expected long-running compute may
|
|
need tuning. GPU utilization alone is not an alarm because busy GPUs are normal.
|
|
corviduo-dev remains monitored but its alert policy was not changed.
|
|
|
|
Notification URL:
|
|
`generic://10.100.10.50:8096/beszel?disabletls=yes&template=json`
|
|
|
|
The bridge at `services/althing-alert-bridge/` forwards through `postbox` to the
|
|
**infra-ops inbox**, as the operator requested. Existing unused email delivery
|
|
was replaced with this verified route. Miranda is a later cutover, not enabled.
|
|
See that service's README for operation and recipient changes.
|
|
|
|
Acceptance on 2026-09-10: fv-ml1 Disk was temporarily lowered to 1%/1 minute;
|
|
the real alert reached althing at 15:29:45Z, thread
|
|
`01M25Z0WFDJM92GPTJQF769HJ7`. The threshold was then restored to 85%/5 minutes.
|
|
Verification uses `postbox thread`, which does not consume the inbox.
|
|
|
|
This installed configuration monitors filesystem capacity. It does not yet
|
|
wire ZFS pool degradation, SMART, scrubs, or an independent hub-down watchdog.
|
|
Those require separate follow-up; a green usage chart does not attest to pool health.
|
|
|
|
References: [additional disks](https://beszel.dev/guide/additional-disks),
|
|
[GPU telemetry](https://beszel.dev/guide/gpu),
|
|
[Homepage widget](https://gethomepage.dev/widgets/services/beszel/).
|