fix(beszel): GPU-LXC temperature alerts watch the GPU, not the hypervisor CPU
nh3-ml1's first Temperature alert (2026-09-25 2105, 88.8 C) was nh3-pve's CPU
package during the nightly vzdump; the GPU sat at 47 C. hwmon is not namespaced,
so an LXC's agent reports its host's coretemp sensors under the LXC's name.
- hosts/{nh3,esh}-ml1.yaml: SENSORS=-coretemp_*,acpitz (blacklist); GPU and
NVMe temperatures verified still reported.
- Hub: Temperature >95 C / 5 m on nh3-pve and esh-pve for the CPUs (i9-13900H
TjMax 100 C; nh3-pve 2-hour averages reach 88 C on busy days).
- nh3-pve README: runs warmer than esh-pve; workload confounds it, check
airflow on the next visit.
This commit is contained in:
@@ -145,7 +145,7 @@ build and this CT will fail to start at the next boot.
|
|||||||
|
|
||||||
| layer | what | where |
|
| layer | what | where |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| **Beszel** (host + GPU telemetry) | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/esh-ml1.yaml`. Hub system `ridkfdpwfq3f730`. GPU util, VRAM, power and temperature are sampled. | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m (GPU included) |
|
| **Beszel** (host + GPU telemetry) | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/esh-ml1.yaml`. Hub system `ridkfdpwfq3f730`. GPU util, VRAM, power and temperature are sampled. | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m, **GPU + NVMe only** since 2026-09-25 (`SENSORS=-coretemp_*,acpitz`; an LXC otherwise reads esh-pve's CPU sensors). esh-pve's own system carries the CPU alert at >95 °C |
|
||||||
| **Uptime Kuma** (service) | `Embed — Qwen3 0.6B (TEI, esh-ml1)` → `:8001/health` (#27); `Rerank — bge-v2-m3 (TEI, esh-ml1)` → `:8013/health` (#28). TEI's health runs the backend. | alerts → infra-ops via althing-alert-bridge; `stacks/uptimekuma/monitors.yaml` |
|
| **Uptime Kuma** (service) | `Embed — Qwen3 0.6B (TEI, esh-ml1)` → `:8001/health` (#27); `Rerank — bge-v2-m3 (TEI, esh-ml1)` → `:8013/health` (#28). TEI's health runs the backend. | alerts → infra-ops via althing-alert-bridge; `stacks/uptimekuma/monitors.yaml` |
|
||||||
| **Homepage** | three cards under *AI - Eval & Retrieval*. dockerd exposes tcp/2375 on `10.0.50.80` only (fleet norm, `playbooks/gpu-lxc.yaml`). | `stacks/homepage/conf/docker.yaml` → `esh-ml1-docker` |
|
| **Homepage** | three cards under *AI - Eval & Retrieval*. dockerd exposes tcp/2375 on `10.0.50.80` only (fleet norm, `playbooks/gpu-lxc.yaml`). | `stacks/homepage/conf/docker.yaml` → `esh-ml1-docker` |
|
||||||
| **Dozzle** (logs) | agent `v10.4.1` on `10.0.50.80:7007`, compose dir `dozzle-agent` | hub on ana-docker :8088 |
|
| **Dozzle** (logs) | agent `v10.4.1` on `10.0.50.80:7007`, compose dir `dozzle-agent` | hub on ana-docker :8088 |
|
||||||
|
|||||||
@@ -99,7 +99,7 @@ nh3-pve runs kernel 6.8.12-43, which the 2026-09-25 headers install pulled in
|
|||||||
|
|
||||||
| layer | what | where |
|
| layer | what | where |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| **Beszel** | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/nh3-ml1.yaml`, hub system `1feeeq61g4mkqre`; GPU util, VRAM and power are sampled | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m |
|
| **Beszel** | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/nh3-ml1.yaml`, hub system `1feeeq61g4mkqre`; GPU util, VRAM and power are sampled | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m (**GPU + NVMe only**: the agent drops the host's `coretemp_*`/`acpitz` with `SENSORS`, because an LXC reads nh3-pve's CPU sensors and the first alert, 2026-09-25 2105, was nh3-pve's CPU during vzdump). nh3-pve's own system carries the CPU alert at >95 °C |
|
||||||
| **Uptime Kuma** | `Embed — Qwen3 0.6B (TEI, nh3-ml1)` → `:8001/health` (#29); `Rerank — bge-v2-m3 (TEI, nh3-ml1)` → `:8013/health` (#30) | `stacks/uptimekuma/monitors.yaml` |
|
| **Uptime Kuma** | `Embed — Qwen3 0.6B (TEI, nh3-ml1)` → `:8001/health` (#29); `Rerank — bge-v2-m3 (TEI, nh3-ml1)` → `:8013/health` (#30) | `stacks/uptimekuma/monitors.yaml` |
|
||||||
| **Homepage** | two cards under *AI - Eval & Retrieval*; dockerd on tcp/2375 bound to `10.100.50.80` | `stacks/homepage/conf/docker.yaml` → `nh3-ml1-docker` |
|
| **Homepage** | two cards under *AI - Eval & Retrieval*; dockerd on tcp/2375 bound to `10.100.50.80` | `stacks/homepage/conf/docker.yaml` → `nh3-ml1-docker` |
|
||||||
| **Dozzle** | agent `v10.4.1` on `10.100.50.80:7007`, compose dir `dozzle-agent`; added to the hub's `DOZZLE_REMOTE_AGENT` | hub on ana-docker :8088 |
|
| **Dozzle** | agent `v10.4.1` on `10.100.50.80:7007`, compose dir `dozzle-agent`; added to the hub's `DOZZLE_REMOTE_AGENT` | hub on ana-docker :8088 |
|
||||||
|
|||||||
@@ -117,6 +117,16 @@ not power back on by itself.
|
|||||||
oops and has run fine, so it is benign so far: only the BT worker dies. The fix
|
oops and has run fine, so it is benign so far: only the BT worker dies. The fix
|
||||||
is to blacklist `btusb` on both hypervisors. That is not done, because it only
|
is to blacklist `btusb` on both hypervisors. That is not done, because it only
|
||||||
takes effect at the next boot.
|
takes effect at the next boot.
|
||||||
|
- **Runs warmer than esh-pve, its twin (Beszel, 2026-09-15 → 09-25).**
|
||||||
|
- nh3-pve's CPU package: median of 2-hour averages 76 °C, peak 2-hour average
|
||||||
|
88 °C (2026-09-23, CPU ~15%).
|
||||||
|
- esh-pve: medians 52–55 °C, peak 20-min average 78 °C.
|
||||||
|
- During the 2026-09-25 2100 vzdump, the 1-minute samples reached 90 °C at
|
||||||
|
13–21% CPU. They were back to 61 °C once the job finished.
|
||||||
|
- Workload is a confound: nh3-pve usually carries more load, so this does not
|
||||||
|
prove a cooling fault. Worth checking airflow and dust on the next visit.
|
||||||
|
- TjMax is 100 °C. The Beszel CPU alert on this host is set at >95 °C for
|
||||||
|
5 minutes.
|
||||||
- **lxc-pve 6.0.0-1 → 6.0.0-2** (2026-09-25 1533, that one package only). This is
|
- **lxc-pve 6.0.0-1 → 6.0.0-2** (2026-09-25 1533, that one package only). This is
|
||||||
Proxmox fix #7006. Without it, runc 1.5 inside a nesting CT fails every
|
Proxmox fix #7006. Without it, runc 1.5 inside a nesting CT fails every
|
||||||
`docker run`. `playbooks/gpu-lxc.yaml` now upgrades it first.
|
`docker run`. `playbooks/gpu-lxc.yaml` now upgrades it first.
|
||||||
|
|||||||
+11
-1
@@ -87,6 +87,15 @@ This collects per-card utilization, VRAM, temperature, and power draw without
|
|||||||
changing the serving containers or GPU power limits. Verified hub samples
|
changing the serving containers or GPU power limits. Verified hub samples
|
||||||
include both RTX PRO 6000 Blackwell cards, the RTX 3090, and the RTX A6000.
|
include both RTX PRO 6000 Blackwell cards, the RTX 3090, and the RTX A6000.
|
||||||
|
|
||||||
|
⚠ **A GPU LXC reads its hypervisor's CPU sensors.** hwmon is not namespaced, so
|
||||||
|
esh-ml1 and nh3-ml1 would report esh-pve's / nh3-pve's `coretemp_*`, and their
|
||||||
|
Temperature alert fired on the HOST CPU under the GPU box's name (nh3-ml1,
|
||||||
|
2026-09-25 2105: 88.8 °C = nh3-pve's package during the 2100 vzdump, GPU at 47 °C).
|
||||||
|
Their overrides set `SENSORS=-coretemp_*,acpitz` (blacklist mode), which keeps
|
||||||
|
the GPU and NVMe temperatures. The CPU alert lives on the hypervisor's own system
|
||||||
|
at >95 °C: the i9-13900H's TjMax is 100 °C, and nh3-pve's 2-hour averages have
|
||||||
|
reached 88 °C on busy days, so 85 °C would fire on normal work.
|
||||||
|
|
||||||
Power charts are actual GPU watts, not total wall power or a PSU/circuit sizing
|
Power charts are actual GPU watts, not total wall power or a PSU/circuit sizing
|
||||||
recommendation. Other system components and workload peaks still matter.
|
recommendation. Other system components and workload peaks still matter.
|
||||||
|
|
||||||
@@ -121,7 +130,8 @@ Thirty rules cover the seven hosts above under the existing operator user:
|
|||||||
| CPU | >95% | 15 minutes |
|
| CPU | >95% | 15 minutes |
|
||||||
| Memory | >90% | 10 minutes |
|
| Memory | >90% | 10 minutes |
|
||||||
| Offline | down | 2 minutes |
|
| Offline | down | 2 minutes |
|
||||||
| Temperature (fv-ml1 and irv-ml1) | >85 C | 5 minutes |
|
| Temperature (fv-ml1, irv-ml1, esh-ml1, nh3-ml1) | >85 C | 5 minutes |
|
||||||
|
| Temperature, hypervisor CPU (nh3-pve, esh-pve; added 2026-09-25) | >95 C | 5 minutes |
|
||||||
|
|
||||||
CPU thresholds are sustained-load warnings; expected long-running compute may
|
CPU thresholds are sustained-load warnings; expected long-running compute may
|
||||||
need tuning. GPU utilization alone is not an alarm because busy GPUs are normal.
|
need tuning. GPU utilization alone is not an alarm because busy GPUs are normal.
|
||||||
|
|||||||
@@ -5,6 +5,13 @@
|
|||||||
services:
|
services:
|
||||||
beszel-agent:
|
beszel-agent:
|
||||||
image: henrygd/beszel-agent-nvidia:0.18.7
|
image: henrygd/beszel-agent-nvidia:0.18.7
|
||||||
|
environment:
|
||||||
|
# Drop the HYPERVISOR's CPU sensors. An LXC reads the host's hwmon, so
|
||||||
|
# without this the Temperature alert here fires on the host CPU and is
|
||||||
|
# labelled as this GPU box (nh3-ml1, 2026-09-25 2100: "88.8 °C" = nh3-pve's
|
||||||
|
# package during vzdump, GPU at 47 °C). The GPU and NVMe temps stay; the
|
||||||
|
# host's own Beszel system alerts on its CPU.
|
||||||
|
- SENSORS=-coretemp_*,acpitz
|
||||||
deploy:
|
deploy:
|
||||||
resources:
|
resources:
|
||||||
reservations:
|
reservations:
|
||||||
|
|||||||
@@ -6,6 +6,13 @@
|
|||||||
services:
|
services:
|
||||||
beszel-agent:
|
beszel-agent:
|
||||||
image: henrygd/beszel-agent-nvidia:0.18.7
|
image: henrygd/beszel-agent-nvidia:0.18.7
|
||||||
|
environment:
|
||||||
|
# Drop the HYPERVISOR's CPU sensors. An LXC reads the host's hwmon, so
|
||||||
|
# without this the Temperature alert here fires on the host CPU and is
|
||||||
|
# labelled as this GPU box (nh3-ml1, 2026-09-25 2100: "88.8 °C" = nh3-pve's
|
||||||
|
# package during vzdump, GPU at 47 °C). The GPU and NVMe temps stay; the
|
||||||
|
# host's own Beszel system alerts on its CPU.
|
||||||
|
- SENSORS=-coretemp_*,acpitz
|
||||||
deploy:
|
deploy:
|
||||||
resources:
|
resources:
|
||||||
reservations:
|
reservations:
|
||||||
|
|||||||
Reference in New Issue
Block a user