fix(beszel): GPU-LXC temperature alerts watch the GPU, not the hypervisor CPU

nh3-ml1's first Temperature alert (2026-09-25 2105, 88.8 C) was nh3-pve's CPU
package during the nightly vzdump; the GPU sat at 47 C. hwmon is not namespaced,
so an LXC's agent reports its host's coretemp sensors under the LXC's name.

- hosts/{nh3,esh}-ml1.yaml: SENSORS=-coretemp_*,acpitz (blacklist); GPU and
  NVMe temperatures verified still reported.
- Hub: Temperature >95 C / 5 m on nh3-pve and esh-pve for the CPUs (i9-13900H
  TjMax 100 C; nh3-pve 2-hour averages reach 88 C on busy days).
- nh3-pve README: runs warmer than esh-pve; workload confounds it, check
  airflow on the next visit.
This commit is contained in:
vh
2026-09-25 21:13:08 -07:00
parent 5960526c3f
commit 6d901c0e8d
6 changed files with 37 additions and 3 deletions
+11 -1
View File
@@ -87,6 +87,15 @@ This collects per-card utilization, VRAM, temperature, and power draw without
changing the serving containers or GPU power limits. Verified hub samples
include both RTX PRO 6000 Blackwell cards, the RTX 3090, and the RTX A6000.
⚠ **A GPU LXC reads its hypervisor's CPU sensors.** hwmon is not namespaced, so
esh-ml1 and nh3-ml1 would report esh-pve's / nh3-pve's `coretemp_*`, and their
Temperature alert fired on the HOST CPU under the GPU box's name (nh3-ml1,
2026-09-25 2105: 88.8 °C = nh3-pve's package during the 2100 vzdump, GPU at 47 °C).
Their overrides set `SENSORS=-coretemp_*,acpitz` (blacklist mode), which keeps
the GPU and NVMe temperatures. The CPU alert lives on the hypervisor's own system
at >95 °C: the i9-13900H's TjMax is 100 °C, and nh3-pve's 2-hour averages have
reached 88 °C on busy days, so 85 °C would fire on normal work.
Power charts are actual GPU watts, not total wall power or a PSU/circuit sizing
recommendation. Other system components and workload peaks still matter.
@@ -121,7 +130,8 @@ Thirty rules cover the seven hosts above under the existing operator user:
| CPU | >95% | 15 minutes |
| Memory | >90% | 10 minutes |
| Offline | down | 2 minutes |
| Temperature (fv-ml1 and irv-ml1) | >85 C | 5 minutes |
| Temperature (fv-ml1, irv-ml1, esh-ml1, nh3-ml1) | >85 C | 5 minutes |
| Temperature, hypervisor CPU (nh3-pve, esh-pve; added 2026-09-25) | >95 C | 5 minutes |
CPU thresholds are sustained-load warnings; expected long-running compute may
need tuning. GPU utilization alone is not an alarm because busy GPUs are normal.
+7
View File
@@ -5,6 +5,13 @@
services:
beszel-agent:
image: henrygd/beszel-agent-nvidia:0.18.7
environment:
# Drop the HYPERVISOR's CPU sensors. An LXC reads the host's hwmon, so
# without this the Temperature alert here fires on the host CPU and is
# labelled as this GPU box (nh3-ml1, 2026-09-25 2100: "88.8 °C" = nh3-pve's
# package during vzdump, GPU at 47 °C). The GPU and NVMe temps stay; the
# host's own Beszel system alerts on its CPU.
- SENSORS=-coretemp_*,acpitz
deploy:
resources:
reservations:
+7
View File
@@ -6,6 +6,13 @@
services:
beszel-agent:
image: henrygd/beszel-agent-nvidia:0.18.7
environment:
# Drop the HYPERVISOR's CPU sensors. An LXC reads the host's hwmon, so
# without this the Temperature alert here fires on the host CPU and is
# labelled as this GPU box (nh3-ml1, 2026-09-25 2100: "88.8 °C" = nh3-pve's
# package during vzdump, GPU at 47 °C). The GPU and NVMe temps stay; the
# host's own Beszel system alerts on its CPU.
- SENSORS=-coretemp_*,acpitz
deploy:
resources:
reservations: