47 lines
2.7 KiB
Markdown
47 lines
2.7 KiB
Markdown
# Beszel fleet wiring — 2026-09-10
|
|
|
|
Operator requested `/tmp/beszel.md` handoff execution, selected **infra-ops inbox**
|
|
for alerts (Miranda later), approved creation of a dedicated monitoring superuser,
|
|
and asked for GPU usage/power telemetry and the card's health detail.
|
|
|
|
Completed: all seven requested hosts up, alongside previously registered
|
|
corviduo-dev (8/8). nh3-docker revived; nh3-dev added. vm-esh-nas was already up,
|
|
contrary to the handoff; access is lkraven, not infra-ops. Irvine's agent was
|
|
healthy but its hub record still pointed at retired 10.100.79.3; fixed to
|
|
100.64.0.6. althing-post-office container remained up throughout.
|
|
|
|
Docker agents need bind mounts, not merely EXTRA_FILESYSTEMS=/tank. Host
|
|
overrides under stacks/beszel/hosts provide read-only mounts. Existing project
|
|
directories/volumes preserved using new deploy-stack options DEPLOY_DEST_STACK
|
|
and DEPLOY_SUDO=1. nh3-dev uses legacy docker-compose and needed the external
|
|
traefik-net network even with agent-only profile. No host Docker upgrade.
|
|
|
|
ana-ml2 tank: 4548.68 / 8791.46 GiB (~51.7%). ana-docker root: ~83.1%, close
|
|
to 85% disk warning. irv-ml1 storetank ~77.4%. NVIDIA agent 0.18.7 on both
|
|
GPU hosts reports all four cards' utilization, VRAM and watts. No GPU power
|
|
limits or serving workloads changed. GPU watts do not size a whole-host PSU.
|
|
|
|
Homepage uses existing discovery labels and version-2 widget; verified one
|
|
card and live authenticated data. This overview shows systems/up only;
|
|
reachability is not a degraded-health score. Per-system widget can expose
|
|
CPU/memory/root disk/network; hub charts contain the additional disks and GPUs.
|
|
|
|
Approved dedicated PocketBase superuser beszel-monitoring@phasefinal.com,
|
|
Vaultwarden ana-docker/beszel-monitoring; Homepage live .env contains its
|
|
credential, labels only placeholders. Existing operator login unchanged.
|
|
|
|
Thirty rules: disk >85% for 5m, CPU >95% for 15m, memory >90% for 10m,
|
|
offline 2m on all seven, temperature >85C for 5m on GPU hosts. Existing unused
|
|
email route replaced with verified webhook. nh3-dev system service
|
|
beszel-althing forwards JSON via supported postbox CLI, sender/recipient
|
|
infra-ops; configurable recipient for later Miranda move. See service README.
|
|
|
|
Real alert test: ana-ml2 Disk 1%/1m fired at 15:29:45Z into althing thread
|
|
01M25Z0WFDJM92GPTJQF769HJ7, receipt confirmed infra-ops reachable. Restored
|
|
85%/5m afterward. Fixed hub appURL from localhost to 10.250.50.70:8090 for
|
|
clickable alert deep links. Inbox verification did not mark mail read.
|
|
|
|
Still separate: ZFS degradation/SMART/scrubs and independent hub/bridge/post
|
|
office outage detection. Bridge deliberately has no hidden delivery queue;
|
|
downstream failure is logged and HTTP 502, not a claimed delivery.
|