vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(beszel): GPU-LXC temperature alerts watch the GPU, not the hypervisor CPU

Browse Source 
nh3-ml1's first Temperature alert (2026-09-25 2105, 88.8 C) was nh3-pve's CPU
package during the nightly vzdump; the GPU sat at 47 C. hwmon is not namespaced,
so an LXC's agent reports its host's coretemp sensors under the LXC's name.

- hosts/{nh3,esh}-ml1.yaml: SENSORS=-coretemp_*,acpitz (blacklist); GPU and
  NVMe temperatures verified still reported.
- Hub: Temperature >95 C / 5 m on nh3-pve and esh-pve for the CPUs (i9-13900H
  TjMax 100 C; nh3-pve 2-hour averages reach 88 C on busy days).
- nh3-pve README: runs warmer than esh-pve; workload confounds it, check
  airflow on the next visit.
This commit is contained in:
vh
2026-09-25 21:13:08 -07:00
parent 5960526c3f
commit 6d901c0e8d
6 changed files with 37 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
servers/nh3-ml1/README.md
+1 -1
View File
@@ -99,7 +99,7 @@ nh3-pve runs kernel 6.8.12-43, which the 2026-09-25 headers install pulled in
| layer | what | where |
|---|---|---|
| **Beszel** | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/nh3-ml1.yaml`, hub system `1feeeq61g4mkqre`; GPU util, VRAM and power are sampled | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m |
| **Beszel** | NVIDIA agent `henrygd/beszel-agent-nvidia:0.18.7`, `stacks/beszel` + `hosts/nh3-ml1.yaml`, hub system `1feeeq61g4mkqre`; GPU util, VRAM and power are sampled | alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m (**GPU + NVMe only**: the agent drops the host's `coretemp_*`/`acpitz` with `SENSORS`, because an LXC reads nh3-pve's CPU sensors and the first alert, 2026-09-25 2105, was nh3-pve's CPU during vzdump). nh3-pve's own system carries the CPU alert at >95 °C |
| **Uptime Kuma** | `Embed — Qwen3 0.6B (TEI, nh3-ml1)` → `:8001/health` (#29); `Rerank — bge-v2-m3 (TEI, nh3-ml1)` → `:8013/health` (#30) | `stacks/uptimekuma/monitors.yaml` |
| **Homepage** | two cards under *AI - Eval & Retrieval*; dockerd on tcp/2375 bound to `10.100.50.80` | `stacks/homepage/conf/docker.yaml` → `nh3-ml1-docker` |
| **Dozzle** | agent `v10.4.1` on `10.100.50.80:7007`, compose dir `dozzle-agent`; added to the hub's `DOZZLE_REMOTE_AGENT` | hub on ana-docker :8088 |