fix(beszel): GPU-LXC temperature alerts watch the GPU, not the hypervisor CPU
nh3-ml1's first Temperature alert (2026-09-25 2105, 88.8 C) was nh3-pve's CPU
package during the nightly vzdump; the GPU sat at 47 C. hwmon is not namespaced,
so an LXC's agent reports its host's coretemp sensors under the LXC's name.
- hosts/{nh3,esh}-ml1.yaml: SENSORS=-coretemp_*,acpitz (blacklist); GPU and
NVMe temperatures verified still reported.
- Hub: Temperature >95 C / 5 m on nh3-pve and esh-pve for the CPUs (i9-13900H
TjMax 100 C; nh3-pve 2-hour averages reach 88 C on busy days).
- nh3-pve README: runs warmer than esh-pve; workload confounds it, check
airflow on the next visit.
This commit is contained in:
+11
-1
@@ -87,6 +87,15 @@ This collects per-card utilization, VRAM, temperature, and power draw without
|
||||
changing the serving containers or GPU power limits. Verified hub samples
|
||||
include both RTX PRO 6000 Blackwell cards, the RTX 3090, and the RTX A6000.
|
||||
|
||||
⚠ **A GPU LXC reads its hypervisor's CPU sensors.** hwmon is not namespaced, so
|
||||
esh-ml1 and nh3-ml1 would report esh-pve's / nh3-pve's `coretemp_*`, and their
|
||||
Temperature alert fired on the HOST CPU under the GPU box's name (nh3-ml1,
|
||||
2026-09-25 2105: 88.8 °C = nh3-pve's package during the 2100 vzdump, GPU at 47 °C).
|
||||
Their overrides set `SENSORS=-coretemp_*,acpitz` (blacklist mode), which keeps
|
||||
the GPU and NVMe temperatures. The CPU alert lives on the hypervisor's own system
|
||||
at >95 °C: the i9-13900H's TjMax is 100 °C, and nh3-pve's 2-hour averages have
|
||||
reached 88 °C on busy days, so 85 °C would fire on normal work.
|
||||
|
||||
Power charts are actual GPU watts, not total wall power or a PSU/circuit sizing
|
||||
recommendation. Other system components and workload peaks still matter.
|
||||
|
||||
@@ -121,7 +130,8 @@ Thirty rules cover the seven hosts above under the existing operator user:
|
||||
| CPU | >95% | 15 minutes |
|
||||
| Memory | >90% | 10 minutes |
|
||||
| Offline | down | 2 minutes |
|
||||
| Temperature (fv-ml1 and irv-ml1) | >85 C | 5 minutes |
|
||||
| Temperature (fv-ml1, irv-ml1, esh-ml1, nh3-ml1) | >85 C | 5 minutes |
|
||||
| Temperature, hypervisor CPU (nh3-pve, esh-pve; added 2026-09-25) | >95 C | 5 minutes |
|
||||
|
||||
CPU thresholds are sustained-load warnings; expected long-running compute may
|
||||
need tuning. GPU utilization alone is not an alarm because busy GPUs are normal.
|
||||
|
||||
@@ -5,6 +5,13 @@
|
||||
services:
|
||||
beszel-agent:
|
||||
image: henrygd/beszel-agent-nvidia:0.18.7
|
||||
environment:
|
||||
# Drop the HYPERVISOR's CPU sensors. An LXC reads the host's hwmon, so
|
||||
# without this the Temperature alert here fires on the host CPU and is
|
||||
# labelled as this GPU box (nh3-ml1, 2026-09-25 2100: "88.8 °C" = nh3-pve's
|
||||
# package during vzdump, GPU at 47 °C). The GPU and NVMe temps stay; the
|
||||
# host's own Beszel system alerts on its CPU.
|
||||
- SENSORS=-coretemp_*,acpitz
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
|
||||
@@ -6,6 +6,13 @@
|
||||
services:
|
||||
beszel-agent:
|
||||
image: henrygd/beszel-agent-nvidia:0.18.7
|
||||
environment:
|
||||
# Drop the HYPERVISOR's CPU sensors. An LXC reads the host's hwmon, so
|
||||
# without this the Temperature alert here fires on the host CPU and is
|
||||
# labelled as this GPU box (nh3-ml1, 2026-09-25 2100: "88.8 °C" = nh3-pve's
|
||||
# package during vzdump, GPU at 47 °C). The GPU and NVMe temps stay; the
|
||||
# host's own Beszel system alerts on its CPU.
|
||||
- SENSORS=-coretemp_*,acpitz
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
|
||||
Reference in New Issue
Block a user