feat(beszel): commit the fleet agent rollout — configs, playbooks and runbooks
Beszel agents are installed and verified across the fleet but the artifacts that produced them were never committed, so the deployment existed only on the hosts. Adds the per-host agent environment files (PORT, NICS, EXTRA_FILESYSTEMS and the hub's PUBLIC key), the systemd unit, the guest install script, the Synology compose, and the elway playbooks for native, guest-stage, guest-install and Synology paths. The two dated memory detail files covering the priority-1 and priority-2 waves ship alongside, per the convention that memory lands with the work it describes. No credentials here. The KEY= value in every host env is the Beszel hub's public ed25519 key, identical across all nine and public by design; the agent README says so explicitly. The nh3-nas sudo password referenced in the runbook prose lives in Vaultwarden and the helper scripts named there never contained it. ⚠ Overlapping VMIDs across hypervisors are a standing trap and are recorded in the priority-2 notes: pfi-pve 105=postgres and 100=pbs-ana, nh3-pve 105=pbs-nh3. ⚠ PBS-NH3's export was ~75.5% used at capture; resource checks are not job success monitoring and should not be read as such.
This commit is contained in:
@@ -0,0 +1,47 @@
|
||||
# Priority 2 — verified 2026-09-11
|
||||
|
||||
Four native unprivileged Beszel 0.18.7 agents installed, active, boot-enabled,
|
||||
and reporting fresh hub samples at 2026-09-12 01:56Z:
|
||||
|
||||
| Host | Address | Filesystems | Root usage |
|
||||
|---|---|---|---|
|
||||
| pfi-postgres | 10.250.50.80 | root (local database storage) | 6.31% |
|
||||
| esh-vm-db | 10.0.50.60 | root, /mnt/backup | 2.22% |
|
||||
| pbs-ana | 10.250.50.90 | root, /mnt/pbs-datastore | 10.43% |
|
||||
| pbs-nh3 | 10.100.50.90 | root, /mnt/pbs-datastore | 7.32% |
|
||||
|
||||
PBS-ANA datastore filesystem: 1426.62/21749.13 GiB (~6.6%).
|
||||
PBS-NH3 datastore filesystem: 32365.07/42879.36 GiB (~75.5%). These measure
|
||||
the exported filesystem's capacity, not bytes belonging exclusively to PBS.
|
||||
NFS provides capacity, not local block-device I/O counters.
|
||||
|
||||
Each has Disk >85%/5m, CPU >95%/15m, Memory >90%/10m and offline/2m:
|
||||
16 new rules read back successfully. Existing operator-user infra-ops bridge
|
||||
route retained. No notification destination changes or induced alert test.
|
||||
Fleet readback: 18 registered, 17 up; only known ana-ml2 outage remains.
|
||||
PostgreSQL accepts connections on both DB hosts; MongoDB running on ESH;
|
||||
both PBS API and proxy services running. No DB/PBS/VM restarts performed.
|
||||
Resource monitoring does NOT verify backup/sync-job success or DB query health.
|
||||
|
||||
## Administrative path
|
||||
|
||||
esh-vm-db uses infra-ops NOPASSWD sudo with playbooks/beszel-native.yaml.
|
||||
Other three allow documented lkraven SSH but require sudo password; no matching
|
||||
vault item found. Existing Proxmox guest-agent admin execution was verified and
|
||||
used instead, WITHOUT changing accounts, sudoers or SSH settings:
|
||||
|
||||
| VM | Hypervisor | VMID |
|
||||
|---|---|---|
|
||||
| pfi-postgres | pfi-pve (10.250.250.31) | 105 |
|
||||
| pbs-ana | pfi-pve (10.250.250.31) | 100 |
|
||||
| pbs-nh3 | nh3-pve (10.100.250.60) | 105 |
|
||||
|
||||
First run `playbooks/beszel-guest-stage.yaml` against the guest's SSH alias with
|
||||
`env_file=configs/beszel-agent/hosts/<name>.env`; then run
|
||||
`playbooks/beszel-guest-install.yaml` against infra-ops@hypervisor with `vmid`.
|
||||
Always verify host/VMID mapping first (IDs overlap across hypervisors).
|
||||
Uses the previously checksum-verified binary in /tmp/beszel-priority1/;
|
||||
see README.md for download/checksum. Stage files contain public key only.
|
||||
Install script preserves existing application services and only restarts Beszel.
|
||||
Hub registration/alert creation is separate from these deployment playbooks.
|
||||
Changes saved locally, not committed, per outstanding main-branch approval.
|
||||
@@ -0,0 +1,63 @@
|
||||
# Native Beszel agents — priority 1
|
||||
|
||||
Deployed 2026-09-11 to ana-nas (LXC), pfi-pve, nh3-pve, esh-pve and
|
||||
esh-pve-nas. No Docker was installed on hypervisors. Agents run as dedicated
|
||||
unprivileged `beszel`, with a systemd service enabled at boot. Hub SSH-polls
|
||||
port 45876 using its existing public key; no new private credential is needed.
|
||||
The committed key is PUBLIC. Do not commit API credentials or agent tokens.
|
||||
|
||||
Version pinned to 0.18.7 to match the existing hub. Download
|
||||
`beszel-agent_linux_amd64.tar.gz` from the official GitHub v0.18.7 release;
|
||||
verified SHA256: `4ae327aac5ad5a231845b0ef613066d555bbe52f7ecb2f28a53d07c04e689aff`.
|
||||
Extract the executable to `/tmp/beszel-priority1/beszel-agent`, then run:
|
||||
|
||||
```sh
|
||||
scripts/elway infra-ops@10.250.50.50 --playbook playbooks/beszel-native.yaml \
|
||||
--var env_file=configs/beszel-agent/hosts/ana-nas.env
|
||||
```
|
||||
|
||||
Use the matching address and env file for each host; `binary_file` overrides
|
||||
the local executable path. Hub registration and four alert records are separate
|
||||
from this host deployment playbook and already exist for all five.
|
||||
|
||||
Host environments list observed mountpoints. Hypervisor NICS explicitly select
|
||||
physical Ethernet ports, avoiding bridge/tap/veth double counting; revisit if
|
||||
hardware changes. Root is automatically monitored. ZFS extra-filesystem labels
|
||||
can be backing-device names or dataset basenames rather than mountpoint names.
|
||||
A parent dataset's usage does NOT include every child's allocation. These are
|
||||
filesystem-capacity charts, not whole-pool capacity/degradation guarantees.
|
||||
LXC host-level CPU/kernel metrics may reflect the shared host; ana-nas is not
|
||||
an independent physical NAS.
|
||||
|
||||
Acceptance: all five services enabled and active; hub samples contain all
|
||||
configured extra filesystems; Homepage reports 13/13 systems up. Added four
|
||||
rules each: Disk >85%/5m, CPU >95%/15m, Memory >90%/10m, offline/2m. Existing
|
||||
operator user's infra-ops bridge route retained. No induced alert test this
|
||||
pass; that route was end-to-end tested on 2026-09-10.
|
||||
|
||||
## Synology sixth host — installed and verified
|
||||
|
||||
Operator pointed out the existing Vaultwarden `nh3-nas/infra-ops-password`.
|
||||
Password-based SSH AND sudo as infra-ops both work. Key-only SSH rejection was
|
||||
not a need to bootstrap access; always check the vault before asking operator.
|
||||
No permissions, keys or account settings changed.
|
||||
|
||||
Deployed 2026-09-11 at 21:50Z via existing DSM Container Manager:
|
||||
`/volume1/compose/beszel/compose.yaml`, canonical
|
||||
`stacks/beszel/synology/compose.yaml`. Restart policy unless-stopped, 0.18.7.
|
||||
Read-only zoneinfo mount measures DSM root (`md0`); read-only stack-directory
|
||||
mount measures `/volume1` (`cachedev_0`). Agent logs confirm both. Existing
|
||||
rest-server stayed healthy with four-week uptime. DSM root ~67%, volume1 ~76%.
|
||||
|
||||
Hub registration `gru841k5cuwb5z2` and four standard alerts created successfully
|
||||
before a connectivity failure. Recovery verified at 2026-09-12 01:51Z:
|
||||
Homepage shows 13/14 up, only ana-ml2 down (operator aware). All six priority-1
|
||||
hosts up. Synology sample at 01:50:32Z reports CPU 1.11%, memory 7.05%, DSM
|
||||
root 66.02%, volume1 32365.07/42879.36 GiB (~75.5%). All four alert rules
|
||||
read back correctly. No network or ana-ml2 changes attempted.
|
||||
|
||||
Initial-install playbook `playbooks/beszel-synology.yaml` refuses to overwrite
|
||||
an existing compose file. DSM requires legacy SCP (`scp -O`). This session used
|
||||
`/tmp/nh3-nas-elway.py` to supply vaulted sudo password in memory, legacy SCP,
|
||||
and `/tmp/nh3-nas-askpass.py` for SSH; neither script contains the password.
|
||||
The native Debian unit is NOT intended for DSM's systemd 219.
|
||||
@@ -0,0 +1,23 @@
|
||||
[Unit]
|
||||
Description=Beszel fleet monitoring agent
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
|
||||
[Service]
|
||||
User=beszel
|
||||
Group=beszel
|
||||
EnvironmentFile=/etc/beszel-agent/environment
|
||||
ExecStart=/usr/local/bin/beszel-agent
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
StateDirectory=beszel-agent
|
||||
NoNewPrivileges=yes
|
||||
ProtectSystem=strict
|
||||
ProtectHome=read-only
|
||||
PrivateTmp=yes
|
||||
ProtectKernelTunables=yes
|
||||
ProtectControlGroups=yes
|
||||
RestrictSUIDSGID=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,14 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
test "$(id -u)" = 0
|
||||
getent passwd beszel >/dev/null || useradd --system --user-group --home-dir /var/lib/beszel-agent --shell /usr/sbin/nologin beszel
|
||||
install -d -o root -g root -m 0755 /etc/beszel-agent
|
||||
install -o root -g root -m 0755 /tmp/beszel-priority2-agent /usr/local/bin/beszel-agent
|
||||
install -o root -g root -m 0600 /tmp/beszel-priority2.env /etc/beszel-agent/environment
|
||||
install -o root -g root -m 0644 /tmp/beszel-priority2.service /etc/systemd/system/beszel-agent.service
|
||||
systemd-analyze verify /etc/systemd/system/beszel-agent.service
|
||||
systemctl daemon-reload
|
||||
systemctl enable beszel-agent
|
||||
systemctl restart beszel-agent
|
||||
systemctl is-active beszel-agent
|
||||
systemctl is-enabled beszel-agent
|
||||
@@ -0,0 +1,3 @@
|
||||
PORT=45876
|
||||
KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
|
||||
EXTRA_FILESYSTEMS=/mnt/backup,/mnt/docker,/mnt/pve-VMStorage,/mnt/ospool-backup
|
||||
@@ -0,0 +1,4 @@
|
||||
PORT=45876
|
||||
NICS=enp9s0,enp10s0,enp11s0,enp12s0,enp5s0f0,enp5s0f1
|
||||
KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
|
||||
EXTRA_FILESYSTEMS=/boot,/nvme,/ssd,/tank,/tank/media,/tank/share,/tank/vmbu,/tank/books,/tank/backup
|
||||
@@ -0,0 +1,4 @@
|
||||
PORT=45876
|
||||
NICS=enp88s0,enp89s0,enp3s0f0np0,enp3s0f1np1
|
||||
KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
|
||||
EXTRA_FILESYSTEMS=/mnt/pve/esh-nas,/mnt/pve/tank-vmbu
|
||||
@@ -0,0 +1,3 @@
|
||||
PORT=45876
|
||||
KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
|
||||
EXTRA_FILESYSTEMS=/mnt/backup
|
||||
@@ -0,0 +1,4 @@
|
||||
PORT=45876
|
||||
NICS=enp87s0,enp88s0,enp2s0f0np0,enp2s0f1np1
|
||||
KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
|
||||
EXTRA_FILESYSTEMS=/rpool,/var/lib/vz,/mnt/pve/pfi-nh3-nas
|
||||
@@ -0,0 +1,3 @@
|
||||
PORT=45876
|
||||
KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
|
||||
EXTRA_FILESYSTEMS=/mnt/pbs-datastore
|
||||
@@ -0,0 +1,3 @@
|
||||
PORT=45876
|
||||
KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
|
||||
EXTRA_FILESYSTEMS=/mnt/pbs-datastore
|
||||
@@ -0,0 +1,2 @@
|
||||
PORT=45876
|
||||
KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
|
||||
@@ -0,0 +1,4 @@
|
||||
PORT=45876
|
||||
NICS=eno8303,eno8403,eno12399np0,eno12409np1
|
||||
KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
|
||||
EXTRA_FILESYSTEMS=/NASPool,/NASPool/backupStore,/NASPool/pve-VMStorage,/ospool
|
||||
Reference in New Issue
Block a user