vh
30f2c977b7
Prime bootstrapped infra-ops on vm-esh-nas with playbooks/bootstrap-infra-ops-user.yaml. It got the fleet-pinned uid/gid 850, NOPASSWD sudo with log_output, the docker group and a 0700 home. That let playbooks/restic-repository-file.yaml migrate the last restic host: the live profile matched the repo's pre-change sha, its units no longer carry the URL, its secrets are vaulted, and the live and repo profiles now match (a5ea75ea). All eight restic hosts are clean. The staged helper script is gone, both from Prime's home on the host and from the repo. The docs that described vm-esh-nas as lkraven-only are updated.
160 lines
8.1 KiB
Markdown
160 lines
8.1 KiB
Markdown
# Beszel fleet monitoring
|
|
|
|
Priority 2 completed 2026-09-11: pfi-postgres, esh-vm-db, pbs-ana and pbs-nh3
|
|
added with 16 alerts. Live samples verified, including both PBS datastores.
|
|
Fleet 17/18 up (known fv-ml1 outage). See `configs/beszel-agent/PRIORITY2.md`.
|
|
|
|
Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the
|
|
2026-09-10 wiring. The hub also retains corviduo-dev's existing registration.
|
|
|
|
2026-09-11: five native agents added (ana-nas plus pfi-pve, nh3-pve,
|
|
esh-pve and esh-pve-nas); 13 systems total and 20 additional alert rules.
|
|
See `configs/beszel-agent/README.md`. Subsequently nh3-nas was installed via
|
|
DSM Docker and registered with four alerts (14 registrations total). Recovery
|
|
verification at 2026-09-12 01:51Z: all six priority-1 hosts up, Synology live
|
|
filesystem samples verified; fleet 13/14 up with known fv-ml1 outage.
|
|
|
|
| Host | Compose directory under `/opt/docker/compose/` | Additional filesystems |
|
|
|---|---|---|
|
|
| ana-docker | beszel | /mnt/backup |
|
|
| fv-ml1 | beszel-agent-ana | /tank, /home |
|
|
| nh3-docker | beszel-agent-nh3 | none |
|
|
| esh-docker-vm (hub name esh-vm-docker) | beszel-agent-esh | /mnt/backup, /mnt/books |
|
|
| irv-ml1 | beszel-agent-irv | /worktank, /storetank, /mnt/smithy |
|
|
| vm-esh-nas | beszel-agent-esh-nas | /mnt/books, /mnt/share, /mnt/music, /mnt/media |
|
|
| nh3-dev | beszel | none since 2026-09-25 (was /mnt/backup, /mnt/smithy — see below) |
|
|
| esh-ml1 (added 2026-09-25) | beszel | none — NVIDIA image (`hosts/esh-ml1.yaml`) for the RTX 2000E Ada |
|
|
| nh3-ml1 (added 2026-09-25) | beszel | none — NVIDIA image (`hosts/nh3-ml1.yaml`) for the RTX 2000E Ada; hub system `1feeeq61g4mkqre`, same 5 alerts as esh-ml1 |
|
|
|
|
⚠ **nh3-dev's agent was DOWN from the 2026-09-24 NH3 power recovery until
|
|
2026-09-25.** Docker could not bind `/mnt/smithy` at boot ("no such device"): since
|
|
`1cbde50` the NAS shares are automounted, and nh3-nas was not up yet. Docker does not
|
|
retry a container that fails to *create*, so `unless-stopped` never brought it back.
|
|
Started by hand, then **fixed the same day**: `hosts/nh3-dev.yaml` no longer binds
|
|
the NAS shares and `BESZEL_EXTRA_FS` is empty. Their capacity is nh3-nas's own
|
|
volume, which nh3-nas's agent reports.
|
|
|
|
Use `infra-ops@<ip>` with passwordless sudo. vm-esh-nas has it too since 2026-09-27.
|
|
Before that date only `lkraven@10.0.50.154`, with Docker access, worked there. Irvine's hub address is
|
|
`100.64.0.6`; its retired `10.100.79.3` address caused silent loss of monitoring.
|
|
|
|
## Filesystems and deployment
|
|
|
|
The canonical source is `stacks/beszel/`. Keep existing Compose project
|
|
names/directories and named volumes to preserve agent identity and history.
|
|
The deployment helper supports `DEPLOY_DEST_STACK` for legacy stack names
|
|
and `DEPLOY_SUDO=1` for root-owned directories. Example:
|
|
|
|
```sh
|
|
DEPLOY_SUDO=1 DEPLOY_DEST_STACK=beszel-agent-ana \
|
|
scripts/deploy-stack.sh infra-ops@10.251.50.54 beszel --compose
|
|
```
|
|
|
|
Agents use host-specific overrides selected by live `.env`:
|
|
|
|
```dotenv
|
|
COMPOSE_PROFILES=agent
|
|
COMPOSE_FILE=compose.yaml:hosts/fv-ml1.yaml
|
|
BESZEL_EXTRA_FS=/extra-filesystems/tank,/extra-filesystems/home
|
|
```
|
|
|
|
Docker agents need actual read-only bind mounts under `/extra-filesystems`.
|
|
`EXTRA_FILESYSTEMS=/tank` alone does not expose the host filesystem. The
|
|
overrides provide those mounts; the environment refers to the container paths.
|
|
Mounts are observed before deployment. Network filesystems provide usage, not
|
|
block-device I/O counters. Shared ZFS datasets expose their available quota,
|
|
which differs from raw pool allocation and snapshot-inclusive usage.
|
|
|
|
After deployment, validate with `docker compose config --quiet`, then
|
|
`docker compose up -d beszel-agent`; restart alone does not apply env or mounts.
|
|
The reusable playbook is `playbooks/beszel-filesystems.yaml` with `stack_dir`,
|
|
`host_name`, and `extra_fs` variables. Environment backups are kept in
|
|
`.env.before-fleet-wiring-20260910` on each host.
|
|
|
|
nh3-dev now has the `docker compose` plugin (v5.5.1, 2026-09-25); the old `docker-compose` binary is gone. It also
|
|
requires the external `traefik-net` network to exist even for the agent profile.
|
|
Only the Beszel agent is started there. On other hosts, use `docker compose`.
|
|
|
|
Auth supports a hub SSH public key (`BESZEL_HUB_KEY`) or outbound token mode
|
|
(`BESZEL_TOKEN` and `HUB_URL`). Existing auth was preserved; nh3-dev uses the
|
|
hub public key. Never copy live tokens into version control.
|
|
|
|
## GPU telemetry
|
|
|
|
fv-ml1 and irv-ml1 use `henrygd/beszel-agent-nvidia:0.18.7` with NVIDIA
|
|
`utility` access to all GPUs. Both hosts already have NVIDIA Container Toolkit.
|
|
This collects per-card utilization, VRAM, temperature, and power draw without
|
|
changing the serving containers or GPU power limits. Verified hub samples
|
|
include both RTX PRO 6000 Blackwell cards, the RTX 3090, and the RTX A6000.
|
|
|
|
⚠ **A GPU LXC reads its hypervisor's CPU sensors.** hwmon is not namespaced, so
|
|
esh-ml1 and nh3-ml1 would report esh-pve's / nh3-pve's `coretemp_*`, and their
|
|
Temperature alert fired on the HOST CPU under the GPU box's name (nh3-ml1,
|
|
2026-09-25 2105: 88.8 °C = nh3-pve's package during the 2100 vzdump, GPU at 47 °C).
|
|
Their overrides set `SENSORS=-coretemp_*,acpitz` (blacklist mode), which keeps
|
|
the GPU and NVMe temperatures. The CPU alert lives on the hypervisor's own system
|
|
at >95 °C: the i9-13900H's TjMax is 100 °C, and nh3-pve's 2-hour averages have
|
|
reached 88 °C on busy days, so 85 °C would fire on normal work.
|
|
|
|
Power charts are actual GPU watts, not total wall power or a PSU/circuit sizing
|
|
recommendation. Other system components and workload peaks still matter.
|
|
|
|
## Homepage
|
|
|
|
The existing Docker-discovered Monitoring card carries the native Beszel
|
|
widget, version 2. No manual Beszel entry is added to services.yaml.
|
|
Leaving `systemId` unset gives the fleet overview (systems/up).
|
|
|
|
The dedicated PocketBase superuser is `beszel-monitoring@phasefinal.com`.
|
|
Its credential is stored in Vaultwarden as `ana-docker/beszel-monitoring` and
|
|
in Homepage's live `.env` as `HOMEPAGE_VAR_BESZEL_USERNAME` and
|
|
`HOMEPAGE_VAR_BESZEL_PASSWORD`. Labels contain only Homepage placeholders.
|
|
The operator's existing account was not reset.
|
|
|
|
Verify one card and its real widget response:
|
|
|
|
```sh
|
|
curl -fsS http://10.0.50.45:5100/api/services |
|
|
jq '[.[] | .services[]? | select(.name=="Beszel")] | length'
|
|
curl -fsS 'http://10.0.50.45:5100/api/services/proxy?group=Monitoring&service=Beszel&endpoint=systems&index=0' |
|
|
jq '{totalItems, systems: [.items[] | {name,status}]}'
|
|
```
|
|
|
|
## Alerts
|
|
|
|
Thirty rules cover the seven hosts above under the existing operator user:
|
|
|
|
| Condition | Threshold | Duration |
|
|
|---|---|---|
|
|
| Disk (root or any extra filesystem) | >85% | 5 minutes |
|
|
| CPU | >95% | 15 minutes |
|
|
| Memory | >90% | 10 minutes |
|
|
| Offline | down | 2 minutes |
|
|
| Temperature (fv-ml1, irv-ml1, esh-ml1, nh3-ml1) | >85 C | 5 minutes |
|
|
| Temperature, hypervisor CPU (nh3-pve, esh-pve; added 2026-09-25) | >95 C | 5 minutes |
|
|
|
|
CPU thresholds are sustained-load warnings; expected long-running compute may
|
|
need tuning. GPU utilization alone is not an alarm because busy GPUs are normal.
|
|
corviduo-dev remains monitored but its alert policy was not changed.
|
|
|
|
Notification URL:
|
|
`generic://10.100.10.50:8096/beszel?disabletls=yes&template=json`
|
|
|
|
The bridge at `services/althing-alert-bridge/` forwards through `postbox` to the
|
|
**infra-ops inbox**, as the operator requested. Existing unused email delivery
|
|
was replaced with this verified route. Miranda is a later cutover, not enabled.
|
|
See that service's README for operation and recipient changes.
|
|
|
|
Acceptance on 2026-09-10: fv-ml1 Disk was temporarily lowered to 1%/1 minute;
|
|
the real alert reached althing at 15:29:45Z, thread
|
|
`01M25Z0WFDJM92GPTJQF769HJ7`. The threshold was then restored to 85%/5 minutes.
|
|
Verification uses `postbox thread`, which does not consume the inbox.
|
|
|
|
This installed configuration monitors filesystem capacity. It does not yet
|
|
wire ZFS pool degradation, SMART, scrubs, or an independent hub-down watchdog.
|
|
Those require separate follow-up; a green usage chart does not attest to pool health.
|
|
|
|
References: [additional disks](https://beszel.dev/guide/additional-disks),
|
|
[GPU telemetry](https://beszel.dev/guide/gpu),
|
|
[Homepage widget](https://gethomepage.dev/widgets/services/beszel/).
|