diff --git a/configs/beszel-agent/PRIORITY2.md b/configs/beszel-agent/PRIORITY2.md new file mode 100644 index 0000000..40bb5b6 --- /dev/null +++ b/configs/beszel-agent/PRIORITY2.md @@ -0,0 +1,47 @@ +# Priority 2 — verified 2026-09-11 + +Four native unprivileged Beszel 0.18.7 agents installed, active, boot-enabled, +and reporting fresh hub samples at 2026-09-12 01:56Z: + +| Host | Address | Filesystems | Root usage | +|---|---|---|---| +| pfi-postgres | 10.250.50.80 | root (local database storage) | 6.31% | +| esh-vm-db | 10.0.50.60 | root, /mnt/backup | 2.22% | +| pbs-ana | 10.250.50.90 | root, /mnt/pbs-datastore | 10.43% | +| pbs-nh3 | 10.100.50.90 | root, /mnt/pbs-datastore | 7.32% | + +PBS-ANA datastore filesystem: 1426.62/21749.13 GiB (~6.6%). +PBS-NH3 datastore filesystem: 32365.07/42879.36 GiB (~75.5%). These measure +the exported filesystem's capacity, not bytes belonging exclusively to PBS. +NFS provides capacity, not local block-device I/O counters. + +Each has Disk >85%/5m, CPU >95%/15m, Memory >90%/10m and offline/2m: +16 new rules read back successfully. Existing operator-user infra-ops bridge +route retained. No notification destination changes or induced alert test. +Fleet readback: 18 registered, 17 up; only known ana-ml2 outage remains. +PostgreSQL accepts connections on both DB hosts; MongoDB running on ESH; +both PBS API and proxy services running. No DB/PBS/VM restarts performed. +Resource monitoring does NOT verify backup/sync-job success or DB query health. + +## Administrative path + +esh-vm-db uses infra-ops NOPASSWD sudo with playbooks/beszel-native.yaml. +Other three allow documented lkraven SSH but require sudo password; no matching +vault item found. Existing Proxmox guest-agent admin execution was verified and +used instead, WITHOUT changing accounts, sudoers or SSH settings: + +| VM | Hypervisor | VMID | +|---|---|---| +| pfi-postgres | pfi-pve (10.250.250.31) | 105 | +| pbs-ana | pfi-pve (10.250.250.31) | 100 | +| pbs-nh3 | nh3-pve (10.100.250.60) | 105 | + +First run `playbooks/beszel-guest-stage.yaml` against the guest's SSH alias with +`env_file=configs/beszel-agent/hosts/.env`; then run +`playbooks/beszel-guest-install.yaml` against infra-ops@hypervisor with `vmid`. +Always verify host/VMID mapping first (IDs overlap across hypervisors). +Uses the previously checksum-verified binary in /tmp/beszel-priority1/; +see README.md for download/checksum. Stage files contain public key only. +Install script preserves existing application services and only restarts Beszel. +Hub registration/alert creation is separate from these deployment playbooks. +Changes saved locally, not committed, per outstanding main-branch approval. diff --git a/configs/beszel-agent/README.md b/configs/beszel-agent/README.md new file mode 100644 index 0000000..e20e9d8 --- /dev/null +++ b/configs/beszel-agent/README.md @@ -0,0 +1,63 @@ +# Native Beszel agents — priority 1 + +Deployed 2026-09-11 to ana-nas (LXC), pfi-pve, nh3-pve, esh-pve and +esh-pve-nas. No Docker was installed on hypervisors. Agents run as dedicated +unprivileged `beszel`, with a systemd service enabled at boot. Hub SSH-polls +port 45876 using its existing public key; no new private credential is needed. +The committed key is PUBLIC. Do not commit API credentials or agent tokens. + +Version pinned to 0.18.7 to match the existing hub. Download +`beszel-agent_linux_amd64.tar.gz` from the official GitHub v0.18.7 release; +verified SHA256: `4ae327aac5ad5a231845b0ef613066d555bbe52f7ecb2f28a53d07c04e689aff`. +Extract the executable to `/tmp/beszel-priority1/beszel-agent`, then run: + +```sh +scripts/elway infra-ops@10.250.50.50 --playbook playbooks/beszel-native.yaml \ + --var env_file=configs/beszel-agent/hosts/ana-nas.env +``` + +Use the matching address and env file for each host; `binary_file` overrides +the local executable path. Hub registration and four alert records are separate +from this host deployment playbook and already exist for all five. + +Host environments list observed mountpoints. Hypervisor NICS explicitly select +physical Ethernet ports, avoiding bridge/tap/veth double counting; revisit if +hardware changes. Root is automatically monitored. ZFS extra-filesystem labels +can be backing-device names or dataset basenames rather than mountpoint names. +A parent dataset's usage does NOT include every child's allocation. These are +filesystem-capacity charts, not whole-pool capacity/degradation guarantees. +LXC host-level CPU/kernel metrics may reflect the shared host; ana-nas is not +an independent physical NAS. + +Acceptance: all five services enabled and active; hub samples contain all +configured extra filesystems; Homepage reports 13/13 systems up. Added four +rules each: Disk >85%/5m, CPU >95%/15m, Memory >90%/10m, offline/2m. Existing +operator user's infra-ops bridge route retained. No induced alert test this +pass; that route was end-to-end tested on 2026-09-10. + +## Synology sixth host — installed and verified + +Operator pointed out the existing Vaultwarden `nh3-nas/infra-ops-password`. +Password-based SSH AND sudo as infra-ops both work. Key-only SSH rejection was +not a need to bootstrap access; always check the vault before asking operator. +No permissions, keys or account settings changed. + +Deployed 2026-09-11 at 21:50Z via existing DSM Container Manager: +`/volume1/compose/beszel/compose.yaml`, canonical +`stacks/beszel/synology/compose.yaml`. Restart policy unless-stopped, 0.18.7. +Read-only zoneinfo mount measures DSM root (`md0`); read-only stack-directory +mount measures `/volume1` (`cachedev_0`). Agent logs confirm both. Existing +rest-server stayed healthy with four-week uptime. DSM root ~67%, volume1 ~76%. + +Hub registration `gru841k5cuwb5z2` and four standard alerts created successfully +before a connectivity failure. Recovery verified at 2026-09-12 01:51Z: +Homepage shows 13/14 up, only ana-ml2 down (operator aware). All six priority-1 +hosts up. Synology sample at 01:50:32Z reports CPU 1.11%, memory 7.05%, DSM +root 66.02%, volume1 32365.07/42879.36 GiB (~75.5%). All four alert rules +read back correctly. No network or ana-ml2 changes attempted. + +Initial-install playbook `playbooks/beszel-synology.yaml` refuses to overwrite +an existing compose file. DSM requires legacy SCP (`scp -O`). This session used +`/tmp/nh3-nas-elway.py` to supply vaulted sudo password in memory, legacy SCP, +and `/tmp/nh3-nas-askpass.py` for SSH; neither script contains the password. +The native Debian unit is NOT intended for DSM's systemd 219. diff --git a/configs/beszel-agent/beszel-agent.service b/configs/beszel-agent/beszel-agent.service new file mode 100644 index 0000000..49c17c8 --- /dev/null +++ b/configs/beszel-agent/beszel-agent.service @@ -0,0 +1,23 @@ +[Unit] +Description=Beszel fleet monitoring agent +Wants=network-online.target +After=network-online.target + +[Service] +User=beszel +Group=beszel +EnvironmentFile=/etc/beszel-agent/environment +ExecStart=/usr/local/bin/beszel-agent +Restart=on-failure +RestartSec=5 +StateDirectory=beszel-agent +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=read-only +PrivateTmp=yes +ProtectKernelTunables=yes +ProtectControlGroups=yes +RestrictSUIDSGID=yes + +[Install] +WantedBy=multi-user.target diff --git a/configs/beszel-agent/guest-install.sh b/configs/beszel-agent/guest-install.sh new file mode 100644 index 0000000..80efc66 --- /dev/null +++ b/configs/beszel-agent/guest-install.sh @@ -0,0 +1,14 @@ +#!/bin/bash +set -euo pipefail +test "$(id -u)" = 0 +getent passwd beszel >/dev/null || useradd --system --user-group --home-dir /var/lib/beszel-agent --shell /usr/sbin/nologin beszel +install -d -o root -g root -m 0755 /etc/beszel-agent +install -o root -g root -m 0755 /tmp/beszel-priority2-agent /usr/local/bin/beszel-agent +install -o root -g root -m 0600 /tmp/beszel-priority2.env /etc/beszel-agent/environment +install -o root -g root -m 0644 /tmp/beszel-priority2.service /etc/systemd/system/beszel-agent.service +systemd-analyze verify /etc/systemd/system/beszel-agent.service +systemctl daemon-reload +systemctl enable beszel-agent +systemctl restart beszel-agent +systemctl is-active beszel-agent +systemctl is-enabled beszel-agent diff --git a/configs/beszel-agent/hosts/ana-nas.env b/configs/beszel-agent/hosts/ana-nas.env new file mode 100644 index 0000000..c5379fb --- /dev/null +++ b/configs/beszel-agent/hosts/ana-nas.env @@ -0,0 +1,3 @@ +PORT=45876 +KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt" +EXTRA_FILESYSTEMS=/mnt/backup,/mnt/docker,/mnt/pve-VMStorage,/mnt/ospool-backup diff --git a/configs/beszel-agent/hosts/esh-pve-nas.env b/configs/beszel-agent/hosts/esh-pve-nas.env new file mode 100644 index 0000000..b415f4f --- /dev/null +++ b/configs/beszel-agent/hosts/esh-pve-nas.env @@ -0,0 +1,4 @@ +PORT=45876 +NICS=enp9s0,enp10s0,enp11s0,enp12s0,enp5s0f0,enp5s0f1 +KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt" +EXTRA_FILESYSTEMS=/boot,/nvme,/ssd,/tank,/tank/media,/tank/share,/tank/vmbu,/tank/books,/tank/backup diff --git a/configs/beszel-agent/hosts/esh-pve.env b/configs/beszel-agent/hosts/esh-pve.env new file mode 100644 index 0000000..3e342f9 --- /dev/null +++ b/configs/beszel-agent/hosts/esh-pve.env @@ -0,0 +1,4 @@ +PORT=45876 +NICS=enp88s0,enp89s0,enp3s0f0np0,enp3s0f1np1 +KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt" +EXTRA_FILESYSTEMS=/mnt/pve/esh-nas,/mnt/pve/tank-vmbu diff --git a/configs/beszel-agent/hosts/esh-vm-db.env b/configs/beszel-agent/hosts/esh-vm-db.env new file mode 100644 index 0000000..074dd72 --- /dev/null +++ b/configs/beszel-agent/hosts/esh-vm-db.env @@ -0,0 +1,3 @@ +PORT=45876 +KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt" +EXTRA_FILESYSTEMS=/mnt/backup diff --git a/configs/beszel-agent/hosts/nh3-pve.env b/configs/beszel-agent/hosts/nh3-pve.env new file mode 100644 index 0000000..16109b7 --- /dev/null +++ b/configs/beszel-agent/hosts/nh3-pve.env @@ -0,0 +1,4 @@ +PORT=45876 +NICS=enp87s0,enp88s0,enp2s0f0np0,enp2s0f1np1 +KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt" +EXTRA_FILESYSTEMS=/rpool,/var/lib/vz,/mnt/pve/pfi-nh3-nas diff --git a/configs/beszel-agent/hosts/pbs-ana.env b/configs/beszel-agent/hosts/pbs-ana.env new file mode 100644 index 0000000..9f00391 --- /dev/null +++ b/configs/beszel-agent/hosts/pbs-ana.env @@ -0,0 +1,3 @@ +PORT=45876 +KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt" +EXTRA_FILESYSTEMS=/mnt/pbs-datastore diff --git a/configs/beszel-agent/hosts/pbs-nh3.env b/configs/beszel-agent/hosts/pbs-nh3.env new file mode 100644 index 0000000..9f00391 --- /dev/null +++ b/configs/beszel-agent/hosts/pbs-nh3.env @@ -0,0 +1,3 @@ +PORT=45876 +KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt" +EXTRA_FILESYSTEMS=/mnt/pbs-datastore diff --git a/configs/beszel-agent/hosts/pfi-postgres.env b/configs/beszel-agent/hosts/pfi-postgres.env new file mode 100644 index 0000000..32d92eb --- /dev/null +++ b/configs/beszel-agent/hosts/pfi-postgres.env @@ -0,0 +1,2 @@ +PORT=45876 +KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt" diff --git a/configs/beszel-agent/hosts/pfi-pve.env b/configs/beszel-agent/hosts/pfi-pve.env new file mode 100644 index 0000000..0d4fc70 --- /dev/null +++ b/configs/beszel-agent/hosts/pfi-pve.env @@ -0,0 +1,4 @@ +PORT=45876 +NICS=eno8303,eno8403,eno12399np0,eno12409np1 +KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt" +EXTRA_FILESYSTEMS=/NASPool,/NASPool/backupStore,/NASPool/pve-VMStorage,/ospool diff --git a/persistent-memory.d/2026-09-11-beszel-priority1.md b/persistent-memory.d/2026-09-11-beszel-priority1.md new file mode 100644 index 0000000..b139902 --- /dev/null +++ b/persistent-memory.d/2026-09-11-beszel-priority1.md @@ -0,0 +1,30 @@ +# Beszel priority 1 — six deployed and verified + +Operator requested two NAS plus four hypervisors. Native 0.18.7 agents enabled +and verified on ana-nas, pfi-pve, nh3-pve, esh-pve, esh-pve-nas; dedicated +unprivileged beszel user, no Docker installed on hypervisors. Existing hub key +authentication and operator account. All five have Disk/CPU/Memory/Status +alerts using existing infra-ops webhook: 20 new rules, 50 total from our wiring. +Homepage 13/13 up; live samples verify extra filesystem capacity. Physical +NICS filters avoid double-counting guest virtual networking. No VM workloads, +storage configuration, existing agents or hub versions changed. + +Canonical configs/beszel-agent/, playbooks/beszel-native.yaml. Agent archive +checksum and exact mount configuration in README/env files. ZFS parents' df +usage is not pool allocation; degradation alerts still not configured. + +RESOLVED access: operator pointed to Vaultwarden nh3-nas/infra-ops-password; +password SSH and sudo both succeeded. Earlier key-only failure did not mean +privileged access was unavailable. No account/permission changes needed. +Synology Docker agent installed 21:50Z, md0 root and volume1 detected; existing +rest-server healthy/unrestarted. Registered gru841k5cuwb5z2 + 4 alerts, 54 total +from our rollout. Canonical stacks/beszel/synology/compose.yaml and initial +playbook beszel-synology.yaml. Vault-fed /tmp/nh3-nas-elway.py handles sudo and +DSM scp -O without logging password. Configs saved but NOT committed: previous +auto-review rejected main-branch commit; no bypass/retry attempted. + +RESOLVED at 2026-09-12 01:51Z after operator reported Anaheim back except +ana-ml2: Homepage 13/14 up, all six priority-1 hosts up, only ana-ml2 down. +Synology live sample 01:50:32Z: CPU 1.11%, memory 7.05%, root 66.02%, volume1 +32365.07/42879.36 GiB (~75.5%). Four alerts verified. Priority-1 rollout complete. +No networking or ana-ml2 changes made; configs remain uncommitted. diff --git a/persistent-memory.d/2026-09-11-beszel-priority2.md b/persistent-memory.d/2026-09-11-beszel-priority2.md new file mode 100644 index 0000000..d741367 --- /dev/null +++ b/persistent-memory.d/2026-09-11-beszel-priority2.md @@ -0,0 +1,15 @@ +# Priority 2 complete + +pfi-postgres, esh-vm-db, pbs-ana, pbs-nh3 have native unprivileged 0.18.7 agents, +enabled at boot and fresh hub samples verified 2026-09-12 01:56Z. Sixteen new +Disk/CPU/Memory/Status rules verified, existing infra-ops bridge route retained. +Fleet 17/18 up; only known ana-ml2 outage. DB/PBS services verified running, +Postgres accepts connections; no application or VM restarts performed. + +ESH uses existing infra-ops sudo. Other three: lkraven key SSH works, sudo needs +password, no matching vault entries. Deployed through EXISTING Proxmox guest +agents (no account/access changes): pfi-pve 105=postgres, 100=pbs-ana; +nh3-pve 105=pbs-nh3. Never confuse overlapping VMIDs. Canonical stage/install +playbooks and host envs saved; runbook configs/beszel-agent/PRIORITY2.md. +PBS-NH3 export ~75.5% used; PBS-ANA ~6.6%. Resource checks are not job-success +monitoring. Local changes still uncommitted, awaiting main-branch approval. diff --git a/playbooks/beszel-guest-install.yaml b/playbooks/beszel-guest-install.yaml new file mode 100644 index 0000000..275bee8 --- /dev/null +++ b/playbooks/beszel-guest-install.yaml @@ -0,0 +1,6 @@ +steps: + - name: Install only Beszel through existing Proxmox guest agent + sudo: true + shell: | + set -euo pipefail + qm guest exec {{ vmid }} -- /bin/bash /tmp/beszel-priority2-install.sh | python3 -c 'import json,sys; r=json.load(sys.stdin); print(r.get("out-data","")); print(r.get("err-data","")); sys.exit(0 if r.get("exited") and r.get("exitcode")==0 else 1)' diff --git a/playbooks/beszel-guest-stage.yaml b/playbooks/beszel-guest-stage.yaml new file mode 100644 index 0000000..84d2624 --- /dev/null +++ b/playbooks/beszel-guest-stage.yaml @@ -0,0 +1,21 @@ +steps: + - name: Stage verified binary as existing SSH user + upload: + src: /tmp/beszel-priority1/beszel-agent + dest: /tmp/beszel-priority2-agent + mode: '0755' + - name: Stage service unit + upload: + src: configs/beszel-agent/beszel-agent.service + dest: /tmp/beszel-priority2.service + mode: '0644' + - name: Stage host mount configuration + upload: + src: '{{ env_file }}' + dest: /tmp/beszel-priority2.env + mode: '0644' + - name: Stage bounded agent installer + upload: + src: configs/beszel-agent/guest-install.sh + dest: /tmp/beszel-priority2-install.sh + mode: '0700' diff --git a/playbooks/beszel-native.yaml b/playbooks/beszel-native.yaml new file mode 100644 index 0000000..463074b --- /dev/null +++ b/playbooks/beszel-native.yaml @@ -0,0 +1,37 @@ +vars: + env_file: configs/beszel-agent/hosts/ana-nas.env + binary_file: /tmp/beszel-priority1/beszel-agent +steps: + - name: Create unprivileged agent account and config directory + sudo: true + shell: | + set -eu + getent passwd beszel >/dev/null || useradd --system --user-group --home-dir /var/lib/beszel-agent --shell /usr/sbin/nologin beszel + install -d -o root -g root -m 0755 /etc/beszel-agent + - name: Install verified version-matched binary + sudo: true + upload: + src: '{{ binary_file }}' + dest: /usr/local/bin/beszel-agent + mode: '0755' + - name: Install host-specific public key and mount configuration + sudo: true + upload: + src: '{{ env_file }}' + dest: /etc/beszel-agent/environment + mode: '0600' + - name: Install unprivileged service + sudo: true + upload: + src: configs/beszel-agent/beszel-agent.service + dest: /etc/systemd/system/beszel-agent.service + mode: '0644' + - name: Enable monitoring agent + sudo: true + shell: | + set -eu + systemd-analyze verify /etc/systemd/system/beszel-agent.service + systemctl daemon-reload + systemctl enable beszel-agent + systemctl restart beszel-agent + systemctl is-active beszel-agent diff --git a/playbooks/beszel-synology.yaml b/playbooks/beszel-synology.yaml new file mode 100644 index 0000000..ac0eef7 --- /dev/null +++ b/playbooks/beszel-synology.yaml @@ -0,0 +1,27 @@ +steps: + - name: Verify DSM paths and prepare agent-only directory + sudo: true + shell: | + set -eu + test -d /usr/share/zoneinfo + test -S /var/run/docker.sock + test ! -e /volume1/compose/beszel/compose.yaml + mkdir -p /volume1/compose/beszel/agent-data + - name: Upload version-matched Synology agent configuration + sudo: true + upload: + src: stacks/beszel/synology/compose.yaml + dest: /volume1/compose/beszel/compose.yaml + mode: '0644' + - name: Start only the monitoring agent + sudo: true + shell: | + set -eu + cd /volume1/compose/beszel + /usr/local/bin/docker compose config --quiet + /usr/local/bin/docker compose pull beszel-agent + /usr/local/bin/docker compose up -d beszel-agent +verify: + - name: Check agent and existing backup service + sudo: true + shell: /usr/local/bin/docker ps --format '{{.Names}} {{.Status}}' && /usr/local/bin/docker logs --tail 15 beszel-agent diff --git a/stacks/beszel/README.md b/stacks/beszel/README.md index 3573b45..678473c 100644 --- a/stacks/beszel/README.md +++ b/stacks/beszel/README.md @@ -1,8 +1,19 @@ # Beszel fleet monitoring +Priority 2 completed 2026-09-11: pfi-postgres, esh-vm-db, pbs-ana and pbs-nh3 +added with 16 alerts. Live samples verified, including both PBS datastores. +Fleet 17/18 up (known ana-ml2 outage). See `configs/beszel-agent/PRIORITY2.md`. + Hub: http://10.250.50.70:8090 (ana-docker), version 0.18.7 at the 2026-09-10 wiring. The hub also retains corviduo-dev's existing registration. +2026-09-11: five native agents added (ana-nas plus pfi-pve, nh3-pve, +esh-pve and esh-pve-nas); 13 systems total and 20 additional alert rules. +See `configs/beszel-agent/README.md`. Subsequently nh3-nas was installed via +DSM Docker and registered with four alerts (14 registrations total). Recovery +verification at 2026-09-12 01:51Z: all six priority-1 hosts up, Synology live +filesystem samples verified; fleet 13/14 up with known ana-ml2 outage. + | Host | Compose directory under `/opt/docker/compose/` | Additional filesystems | |---|---|---| | ana-docker | beszel | /mnt/backup | diff --git a/stacks/beszel/synology/compose.yaml b/stacks/beszel/synology/compose.yaml new file mode 100644 index 0000000..56c8257 --- /dev/null +++ b/stacks/beszel/synology/compose.yaml @@ -0,0 +1,16 @@ +services: + beszel-agent: + image: henrygd/beszel-agent:0.18.7 + container_name: beszel-agent + restart: unless-stopped + network_mode: host + volumes: + - ./agent-data:/var/lib/beszel-agent + - /var/run/docker.sock:/var/run/docker.sock:ro + - /usr/share/zoneinfo:/extra-filesystems/system:ro + - ./:/extra-filesystems/volume1:ro + environment: + PORT: '45876' + KEY: 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt' + FILESYSTEM: /extra-filesystems/system + EXTRA_FILESYSTEMS: /extra-filesystems/volume1