From 6d901c0e8d207f864c1be13f5b2362496b3225fb Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 25 Sep 2026 21:13:08 -0700 Subject: [PATCH] fix(beszel): GPU-LXC temperature alerts watch the GPU, not the hypervisor CPU nh3-ml1's first Temperature alert (2026-09-25 2105, 88.8 C) was nh3-pve's CPU package during the nightly vzdump; the GPU sat at 47 C. hwmon is not namespaced, so an LXC's agent reports its host's coretemp sensors under the LXC's name. - hosts/{nh3,esh}-ml1.yaml: SENSORS=-coretemp_*,acpitz (blacklist); GPU and NVMe temperatures verified still reported. - Hub: Temperature >95 C / 5 m on nh3-pve and esh-pve for the CPUs (i9-13900H TjMax 100 C; nh3-pve 2-hour averages reach 88 C on busy days). - nh3-pve README: runs warmer than esh-pve; workload confounds it, check airflow on the next visit. --- servers/esh-ml1/README.md | 2 +- servers/nh3-ml1/README.md | 2 +- servers/nh3-pve/README.md | 10 ++++++++++ stacks/beszel/README.md | 12 +++++++++++- stacks/beszel/hosts/esh-ml1.yaml | 7 +++++++ stacks/beszel/hosts/nh3-ml1.yaml | 7 +++++++ 6 files changed, 37 insertions(+), 3 deletions(-) diff --git a/servers/esh-ml1/README.md b/servers/esh-ml1/README.md index 5872687..e905c90 100644 --- a/servers/esh-ml1/README.md +++ b/servers/esh-ml1/README.md @@ -145,7 +145,7 @@ build and this CT will fail to start at the next boot. | layer | what | where | |---|---|---| -| **Beszel** (host + GPU telemetry) | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/esh-ml1.yaml`. Hub system `ridkfdpwfq3f730`. GPU util, VRAM, power and temperature are sampled. | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m (GPU included) | +| **Beszel** (host + GPU telemetry) | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/esh-ml1.yaml`. Hub system `ridkfdpwfq3f730`. GPU util, VRAM, power and temperature are sampled. | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m, **GPU + NVMe only** since 2026-09-25 (`SENSORS=-coretemp_*,acpitz`; an LXC otherwise reads esh-pve's CPU sensors). esh-pve's own system carries the CPU alert at >95 °C | | **Uptime Kuma** (service) | `Embed — Qwen3 0.6B (TEI, esh-ml1)` → `:8001/health` (#27); `Rerank — bge-v2-m3 (TEI, esh-ml1)` → `:8013/health` (#28). TEI's health runs the backend. | alerts → infra-ops via althing-alert-bridge; `stacks/uptimekuma/monitors.yaml` | | **Homepage** | three cards under *AI - Eval & Retrieval*. dockerd exposes tcp/2375 on `10.0.50.80` only (fleet norm, `playbooks/gpu-lxc.yaml`). | `stacks/homepage/conf/docker.yaml` → `esh-ml1-docker` | | **Dozzle** (logs) | agent `v10.4.1` on `10.0.50.80:7007`, compose dir `dozzle-agent` | hub on ana-docker :8088 | diff --git a/servers/nh3-ml1/README.md b/servers/nh3-ml1/README.md index 7c9eeef..e2f0af3 100644 --- a/servers/nh3-ml1/README.md +++ b/servers/nh3-ml1/README.md @@ -99,7 +99,7 @@ nh3-pve runs kernel 6.8.12-43, which the 2026-09-25 headers install pulled in | layer | what | where | |---|---|---| -| **Beszel** | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/nh3-ml1.yaml`, hub system `1feeeq61g4mkqre`; GPU util, VRAM and power are sampled | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m | +| **Beszel** | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/nh3-ml1.yaml`, hub system `1feeeq61g4mkqre`; GPU util, VRAM and power are sampled | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m (**GPU + NVMe only**: the agent drops the host's `coretemp_*`/`acpitz` with `SENSORS`, because an LXC reads nh3-pve's CPU sensors and the first alert, 2026-09-25 2105, was nh3-pve's CPU during vzdump). nh3-pve's own system carries the CPU alert at >95 °C | | **Uptime Kuma** | `Embed — Qwen3 0.6B (TEI, nh3-ml1)` → `:8001/health` (#29); `Rerank — bge-v2-m3 (TEI, nh3-ml1)` → `:8013/health` (#30) | `stacks/uptimekuma/monitors.yaml` | | **Homepage** | two cards under *AI - Eval & Retrieval*; dockerd on tcp/2375 bound to `10.100.50.80` | `stacks/homepage/conf/docker.yaml` → `nh3-ml1-docker` | | **Dozzle** | agent `v10.4.1` on `10.100.50.80:7007`, compose dir `dozzle-agent`; added to the hub's `DOZZLE_REMOTE_AGENT` | hub on ana-docker :8088 | diff --git a/servers/nh3-pve/README.md b/servers/nh3-pve/README.md index 0190d6c..d3fec4d 100644 --- a/servers/nh3-pve/README.md +++ b/servers/nh3-pve/README.md @@ -117,6 +117,16 @@ not power back on by itself. oops and has run fine, so it is benign so far: only the BT worker dies. The fix is to blacklist `btusb` on both hypervisors. That is not done, because it only takes effect at the next boot. +- **Runs warmer than esh-pve, its twin (Beszel, 2026-09-15 → 09-25).** + - nh3-pve's CPU package: median of 2-hour averages 76 °C, peak 2-hour average + 88 °C (2026-09-23, CPU ~15%). + - esh-pve: medians 52–55 °C, peak 20-min average 78 °C. + - During the 2026-09-25 2100 vzdump, the 1-minute samples reached 90 °C at + 13–21% CPU. They were back to 61 °C once the job finished. + - Workload is a confound: nh3-pve usually carries more load, so this does not + prove a cooling fault. Worth checking airflow and dust on the next visit. + - TjMax is 100 °C. The Beszel CPU alert on this host is set at >95 °C for + 5 minutes. - **lxc-pve 6.0.0-1 → 6.0.0-2** (2026-09-25 1533, that one package only). This is Proxmox fix #7006. Without it, runc 1.5 inside a nesting CT fails every `docker run`. `playbooks/gpu-lxc.yaml` now upgrades it first. diff --git a/stacks/beszel/README.md b/stacks/beszel/README.md index 11a65dd..879f43e 100644 --- a/stacks/beszel/README.md +++ b/stacks/beszel/README.md @@ -87,6 +87,15 @@ This collects per-card utilization, VRAM, temperature, and power draw without changing the serving containers or GPU power limits. Verified hub samples include both RTX PRO 6000 Blackwell cards, the RTX 3090, and the RTX A6000. +⚠ **A GPU LXC reads its hypervisor's CPU sensors.** hwmon is not namespaced, so +esh-ml1 and nh3-ml1 would report esh-pve's / nh3-pve's `coretemp_*`, and their +Temperature alert fired on the HOST CPU under the GPU box's name (nh3-ml1, +2026-09-25 2105: 88.8 °C = nh3-pve's package during the 2100 vzdump, GPU at 47 °C). +Their overrides set `SENSORS=-coretemp_*,acpitz` (blacklist mode), which keeps +the GPU and NVMe temperatures. The CPU alert lives on the hypervisor's own system +at >95 °C: the i9-13900H's TjMax is 100 °C, and nh3-pve's 2-hour averages have +reached 88 °C on busy days, so 85 °C would fire on normal work. + Power charts are actual GPU watts, not total wall power or a PSU/circuit sizing recommendation. Other system components and workload peaks still matter. @@ -121,7 +130,8 @@ Thirty rules cover the seven hosts above under the existing operator user: | CPU | >95% | 15 minutes | | Memory | >90% | 10 minutes | | Offline | down | 2 minutes | -| Temperature (fv-ml1 and irv-ml1) | >85 C | 5 minutes | +| Temperature (fv-ml1, irv-ml1, esh-ml1, nh3-ml1) | >85 C | 5 minutes | +| Temperature, hypervisor CPU (nh3-pve, esh-pve; added 2026-09-25) | >95 C | 5 minutes | CPU thresholds are sustained-load warnings; expected long-running compute may need tuning. GPU utilization alone is not an alarm because busy GPUs are normal. diff --git a/stacks/beszel/hosts/esh-ml1.yaml b/stacks/beszel/hosts/esh-ml1.yaml index 3b8579e..50f8534 100644 --- a/stacks/beszel/hosts/esh-ml1.yaml +++ b/stacks/beszel/hosts/esh-ml1.yaml @@ -5,6 +5,13 @@ services: beszel-agent: image: henrygd/beszel-agent-nvidia:0.18.7 + environment: + # Drop the HYPERVISOR's CPU sensors. An LXC reads the host's hwmon, so + # without this the Temperature alert here fires on the host CPU and is + # labelled as this GPU box (nh3-ml1, 2026-09-25 2100: "88.8 °C" = nh3-pve's + # package during vzdump, GPU at 47 °C). The GPU and NVMe temps stay; the + # host's own Beszel system alerts on its CPU. + - SENSORS=-coretemp_*,acpitz deploy: resources: reservations: diff --git a/stacks/beszel/hosts/nh3-ml1.yaml b/stacks/beszel/hosts/nh3-ml1.yaml index 6c27a50..d2313bb 100644 --- a/stacks/beszel/hosts/nh3-ml1.yaml +++ b/stacks/beszel/hosts/nh3-ml1.yaml @@ -6,6 +6,13 @@ services: beszel-agent: image: henrygd/beszel-agent-nvidia:0.18.7 + environment: + # Drop the HYPERVISOR's CPU sensors. An LXC reads the host's hwmon, so + # without this the Temperature alert here fires on the host CPU and is + # labelled as this GPU box (nh3-ml1, 2026-09-25 2100: "88.8 °C" = nh3-pve's + # package during vzdump, GPU at 47 °C). The GPU and NVMe temps stay; the + # host's own Beszel system alerts on its CPU. + - SENSORS=-coretemp_*,acpitz deploy: resources: reservations: