vh
65dc586497
- Beszel: NVIDIA agent (stacks/beszel hosts/esh-ml1.yaml), hub system registered with Status/Disk/CPU/Memory/Temperature alerts; GPU samples verified (RTX 2000E Ada util, VRAM, power, temp). - Uptime Kuma: health monitors for the TEI embed (:8001) and rerank (:8013) services — the sole backends behind the gateway, so the one exception to "seats are out of Kuma's lane". Reward seat excluded (no consumer). - Homepage: dockerd on tcp/2375 bound to 10.0.50.80 (playbook step); esh-ml1-docker added to docker.yaml. - Dozzle: agent v10.4.1 on esh-ml1. The hub's DOZZLE_REMOTE_AGENT still named ana-ml2's 10.250.50.54 and irv-ml1's retired 10.100.79.3; repointed to 10.251.50.54 / 10.6.110.50 (fv-ml1 and irv-ml1 logs were missing). - Documented nh3-dev's Beszel agent failing on NH3 cold start (bind of an automounted NAS share); revived by hand, fix still open.
150 lines
7.1 KiB
Markdown
150 lines
7.1 KiB
Markdown
# Beszel fleet monitoring
|
|
|
|
Priority 2 completed 2026-09-11: pfi-postgres, esh-vm-db, pbs-ana and pbs-nh3
|
|
added with 16 alerts. Live samples verified, including both PBS datastores.
|
|
Fleet 17/18 up (known fv-ml1 outage). See `configs/beszel-agent/PRIORITY2.md`.
|
|
|
|
Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the
|
|
2026-09-10 wiring. The hub also retains corviduo-dev's existing registration.
|
|
|
|
2026-09-11: five native agents added (ana-nas plus pfi-pve, nh3-pve,
|
|
esh-pve and esh-pve-nas); 13 systems total and 20 additional alert rules.
|
|
See `configs/beszel-agent/README.md`. Subsequently nh3-nas was installed via
|
|
DSM Docker and registered with four alerts (14 registrations total). Recovery
|
|
verification at 2026-09-12 01:51Z: all six priority-1 hosts up, Synology live
|
|
filesystem samples verified; fleet 13/14 up with known fv-ml1 outage.
|
|
|
|
| Host | Compose directory under `/opt/docker/compose/` | Additional filesystems |
|
|
|---|---|---|
|
|
| ana-docker | beszel | /mnt/backup |
|
|
| fv-ml1 | beszel-agent-ana | /tank, /home |
|
|
| nh3-docker | beszel-agent-nh3 | none |
|
|
| esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books |
|
|
| irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy |
|
|
| vm-esh-nas | beszel-agent-esh-nas | /mnt/books, /mnt/share, /mnt/music, /mnt/media |
|
|
| nh3-dev | beszel | /mnt/backup, /mnt/smithy |
|
|
| esh-ml1 (added 2026-09-25) | beszel | none — NVIDIA image (`hosts/esh-ml1.yaml`) for the RTX 2000E Ada |
|
|
|
|
⚠ **nh3-dev's agent was DOWN from the 2026-09-24 NH3 power recovery until
|
|
2026-09-25.** Docker could not bind `/mnt/smithy` at boot ("no such device"): since
|
|
`1cbde50` the NAS shares are automounted, and nh3-nas was not up yet. Docker does not
|
|
retry a container that fails to *create*, so `unless-stopped` never brought it back.
|
|
Started by hand. **It will recur on the next NH3 cold start** until the agent stops
|
|
depending on NAS mounts at boot. The NAS capacity is already reported by
|
|
nh3-nas's own agent, so dropping the two extra filesystems here is the simplest fix.
|
|
Open item.
|
|
|
|
Use `infra-ops@<ip>` with passwordless sudo, except vm-esh-nas:
|
|
`lkraven@10.0.50.154` has Docker access. Irvine's hub address is
|
|
`100.64.0.6`; its retired `10.100.79.3` address caused silent loss of monitoring.
|
|
|
|
## Filesystems and deployment
|
|
|
|
The canonical source is `stacks/beszel/`. Keep existing Compose project
|
|
names/directories and named volumes to preserve agent identity and history.
|
|
The deployment helper supports `DEPLOY_DEST_STACK` for legacy stack names
|
|
and `DEPLOY_SUDO=1` for root-owned directories. Example:
|
|
|
|
```sh
|
|
DEPLOY_SUDO=1 DEPLOY_DEST_STACK=beszel-agent-ana \
|
|
scripts/deploy-stack.sh infra-ops@10.251.50.54 beszel --compose
|
|
```
|
|
|
|
Agents use host-specific overrides selected by live `.env`:
|
|
|
|
```dotenv
|
|
COMPOSE_PROFILES=agent
|
|
COMPOSE_FILE=compose.yaml:hosts/fv-ml1.yaml
|
|
BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home
|
|
```
|
|
|
|
Docker agents need actual read-only bind mounts under `/extra-filesystems`.
|
|
`EXTRA_FILESYSTEMS=/tank` alone does not expose the host filesystem. The
|
|
overrides provide those mounts; the environment refers to the container paths.
|
|
Mounts are observed before deployment. Network filesystems provide usage, not
|
|
block-device I/O counters. Shared ZFS datasets expose their available quota,
|
|
which differs from raw pool allocation and snapshot-inclusive usage.
|
|
|
|
After deployment, validate with `docker compose config --quiet`, then
|
|
`docker compose up -d beszel-agent`; restart alone does not apply env or mounts.
|
|
The reusable playbook is `playbooks/beszel-filesystems.yaml` with `stack_dir`,
|
|
`host_name`, and `extra_fs` variables. Environment backups are kept in
|
|
`.env.before-fleet-wiring-20260910` on each host.
|
|
|
|
nh3-dev has the older `docker-compose` command; use that spelling. It also
|
|
requires the external `traefik-net` network to exist even for the agent profile.
|
|
Only the Beszel agent is started there. On other hosts, use `docker compose`.
|
|
|
|
Auth supports a hub SSH public key (`BESZEL_HUB_KEY`) or outbound token mode
|
|
(`BESZEL_TOKEN` and `HUB_URL`). Existing auth was preserved; nh3-dev uses the
|
|
hub public key. Never copy live tokens into version control.
|
|
|
|
## GPU telemetry
|
|
|
|
fv-ml1 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA
|
|
`utility` access to all GPUs. Both hosts already have NVIDIA Container Toolkit.
|
|
This collects per-card utilization, VRAM, temperature, and power draw without
|
|
changing the serving containers or GPU power limits. Verified hub samples
|
|
include both RTX PRO 6000 Blackwell cards, the RTX 3090, and the RTX A6000.
|
|
|
|
Power charts are actual GPU watts, not total wall power or a PSU/circuit sizing
|
|
recommendation. Other system components and workload peaks still matter.
|
|
|
|
## Homepage
|
|
|
|
The existing Docker-discovered Monitoring card carries the native Beszel
|
|
widget, version 2. No manual Beszel entry is added to services.yaml.
|
|
Leaving `systemId` unset gives the fleet overview (systems/up).
|
|
|
|
The dedicated PocketBase superuser is `beszel-monitoring@phasefinal.com`.
|
|
Its credential is stored in Vaultwarden as `ana-docker/beszel-monitoring` and
|
|
in Homepage's live `.env` as `HOMEPAGE_VAR_BESZEL_USERNAME` and
|
|
`HOMEPAGE_VAR_BESZEL_PASSWORD`. Labels contain only Homepage placeholders.
|
|
The operator's existing account was not reset.
|
|
|
|
Verify one card and its real widget response:
|
|
|
|
```sh
|
|
curl -fsS http://10.0.50.45:5100/api/services |
|
|
jq '[.[] | .services[]? | select(.name=="Beszel")] | length'
|
|
curl -fsS 'http://10.0.50.45:5100/api/services/proxy?group=Monitoring&service=Beszel&endpoint=systems&index=0' |
|
|
jq '{totalItems, systems: [.items[] | {name,status}]}'
|
|
```
|
|
|
|
## Alerts
|
|
|
|
Thirty rules cover the seven hosts above under the existing operator user:
|
|
|
|
| Condition | Threshold | Duration |
|
|
|---|---|---|
|
|
| Disk (root or any extra filesystem) | >85% | 5 minutes |
|
|
| CPU | >95% | 15 minutes |
|
|
| Memory | >90% | 10 minutes |
|
|
| Offline | down | 2 minutes |
|
|
| Temperature (fv-ml1 and irv-ml1) | >85 C | 5 minutes |
|
|
|
|
CPU thresholds are sustained-load warnings; expected long-running compute may
|
|
need tuning. GPU utilization alone is not an alarm because busy GPUs are normal.
|
|
corviduo-dev remains monitored but its alert policy was not changed.
|
|
|
|
Notification URL:
|
|
`generic://10.100.10.50:8096/beszel?disabletls=yes&template=json`
|
|
|
|
The bridge at `services/althing-alert-bridge/` forwards through `postbox` to the
|
|
**infra-ops inbox**, as the operator requested. Existing unused email delivery
|
|
was replaced with this verified route. Miranda is a later cutover, not enabled.
|
|
See that service's README for operation and recipient changes.
|
|
|
|
Acceptance on 2026-09-10: fv-ml1 Disk was temporarily lowered to 1%/1 minute;
|
|
the real alert reached althing at 15:29:45Z, thread
|
|
`01M25Z0WFDJM92GPTJQF769HJ7`. The threshold was then restored to 85%/5 minutes.
|
|
Verification uses `postbox thread`, which does not consume the inbox.
|
|
|
|
This installed configuration monitors filesystem capacity. It does not yet
|
|
wire ZFS pool degradation, SMART, scrubs, or an independent hub-down watchdog.
|
|
Those require separate follow-up; a green usage chart does not attest to pool health.
|
|
|
|
References: [additional disks](https://beszel.dev/guide/additional-disks),
|
|
[GPU telemetry](https://beszel.dev/guide/gpu),
|
|
[Homepage widget](https://gethomepage.dev/widgets/services/beszel/).
|