vh
6d901c0e8d
nh3-ml1's first Temperature alert (2026-09-25 2105, 88.8 C) was nh3-pve's CPU
package during the nightly vzdump; the GPU sat at 47 C. hwmon is not namespaced,
so an LXC's agent reports its host's coretemp sensors under the LXC's name.
- hosts/{nh3,esh}-ml1.yaml: SENSORS=-coretemp_*,acpitz (blacklist); GPU and
NVMe temperatures verified still reported.
- Hub: Temperature >95 C / 5 m on nh3-pve and esh-pve for the CPUs (i9-13900H
TjMax 100 C; nh3-pve 2-hour averages reach 88 C on busy days).
- nh3-pve README: runs warmer than esh-pve; workload confounds it, check
airflow on the next visit.
6.1 KiB
nh3-ml1
GPU LXC for the NH3 site: CT 109 on nh3-pve, holding the NVIDIA RTX 2000E
Ada (16 GB, 50 W, 01:00.0, PCIe gen4 x8). It is the second embedding and
reranking backend, the twin of esh-ml1: same card, same
driver, same TEI image and models. Built 2026-09-25, after the NH3 site visit
turned Secure Boot off on nh3-pve.
⚠ Not behind the gateway yet. It serves on its own ports and is monitored, but
LiteLLM still routes qwen3-embedding and reranker to esh-ml1 alone. Routing is
Prime's call (recommendation: load-share; see below).
| IP | 10.100.50.80/24, VLAN 50 (nh3-servers), gateway 10.100.50.1 (static, outside the UDM's .150–.249 DHCP pool) |
| DNS | nh3-ml1.nh3.internal |
| SSH | ssh nh3-ml1 → infra-ops@10.100.50.80 (NOPASSWD sudo) · from the host: pct enter 109 |
| OS | Debian 12, unprivileged, nesting=1,keyctl=1 |
| Size | 6 cores, 16 GB RAM + 2 GB swap, 80 GB rootfs on local-zfs |
| Boot | onboot: 1, startup: order=30, after the site's core guests, so a GPU fault never delays NH3's DNS or mesh route |
| Backups | None, on purpose. nh3-pve's vzdump job is all 1; the playbook added 109 to its exclude list. Everything is rebuilt from the playbooks and the stack; models re-download. |
What it serves
stacks/embed-rerank (/opt/docker/compose/embed-rerank), TEI 1.9.4
(89-1.9.4). The live .env differs from .env.example only in
HOST_NAME=nh3-ml1 and HOST_IP=10.100.50.80, which feed the Homepage labels.
| container | model | port |
|---|---|---|
tei-embed |
Qwen/Qwen3-Embedding-0.6B |
8001 (/v1/embeddings, /embed) |
tei-rerank |
BAAI/bge-reranker-v2-m3 |
8013 (/rerank, body query + texts) |
VRAM ~2.7 GB for both, so ~13 GB is free.
Parity and speed vs esh-ml1 (2026-09-25 ~1540 PT)
Client on nh3-dev, calling both seats directly. Corpus: 1,120 paragraphs from this repo's docs plus 6 fixed texts (CJK, code, emoji, a 1-char input, a 6k-char passage); 50 instruction-format queries; each host embedded everything twice.
| embedding check | result | noise floor / control |
|---|---|---|
| per-text cosine, esh vs nh3 | median 0.999998, min 0.999993 | esh vs esh 0.999998 / 0.999995; nh3 vs nh3 0.999998 / 0.999993 |
| overlap@10, nh3 queries on the esh index | 1.000 | esh rerun 1.000; positive control MRL-256 truncation 0.684 |
| overlap@10, nh3 index + nh3 queries | 1.000 | — |
| hit@1 own paragraph | 0.76 (both hosts) | — |
| negative control, different texts | cosine median 0.50 | — |
| rerank check (100 queries × 20 docs) | esh vs nh3 | esh vs esh | positive control (query cut to 4 words) |
|---|---|---|---|
| top-1 agreement | 1.00 | 1.00 | 0.94 |
| top-5 exact order | 0.98 | 0.97 | 0.09 |
| max score difference | 0.0014 | 0.0020 | 0.97 |
The two hosts cannot be told apart. Every esh-vs-nh3 figure sits inside the esh-vs-esh noise. Sensitivity floor: this method cannot resolve an embedding cosine gap below ~5×10⁻⁶ or a rerank score difference below ~0.002. An index built on either host serves queries from the other.
Speed, on-box, 3 interleaved reps per host (range across reps):
| workload | nh3-ml1 | esh-ml1 |
|---|---|---|
| embed 1 short query, p50 | 6.60–6.74 ms | 6.97–7.02 ms |
| bulk embed, 64 per request, passages/s | 103.7–105.0 | 105.5–108.1 |
| rerank 20 docs, p50 | 155.8–159.2 ms | 151.8–160.5 ms |
The two hosts run at the same speed. esh-ml1 was also holding the idle reward seat (~8 GB VRAM, 0% util) during these runs.
Gateway routing (Prime's call). The recommendation is load-share. The 2026-09-25 rule against load-sharing came from pairing esh-ml1 with the much faster fv-ml1. These two cards are identical, and a second site removes the single-host outage the esh-ml1 README warns about.
How it is built
playbooks/pve-nvidia-host.yamlon nh3-pve: driver 580.178.04 (open modules, DKMS) plus thenvidia-persistencedunit. It needs Secure Boot off, which was turned off in the BIOS on the 2026-09-25 visit; the pre-flight refuses otherwise.playbooks/gpu-lxc.yamlwith the "Run (nh3-ml1)"--varline from its header. ⚠ nh3-pve was on lxc-pve 6.0.0-1. With it, everydocker runin the CT failed with "open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission denied" (runc 1.5 against the old AppArmor profile). Upgrading lxc-pve alone to 6.0.0-2 (Proxmox fix #7006) and then runningpct reboot 109fixed it. The playbook now does the upgrade as its first step.scripts/deploy-stack.sh nh3-ml1 embed-rerank, then setHOST_NAME/HOST_IPin.env, thendocker compose up -d.
The driver version lock and the DKMS/kernel notes in
esh-ml1's README apply here unchanged.
nh3-pve runs kernel 6.8.12-43, which the 2026-09-25 headers install pulled in
(see servers/nh3-pve/README.md).
Monitoring and telemetry (wired 2026-09-25)
| layer | what | where |
|---|---|---|
| Beszel | NVIDIA agent henrygd/beszel-agent-nvidia:0.18.7, stacks/beszel + hosts/nh3-ml1.yaml, hub system 1feeeq61g4mkqre; GPU util, VRAM and power are sampled |
alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m (GPU + NVMe only: the agent drops the host's coretemp_*/acpitz with SENSORS, because an LXC reads nh3-pve's CPU sensors and the first alert, 2026-09-25 2105, was nh3-pve's CPU during vzdump). nh3-pve's own system carries the CPU alert at >95 °C |
| Uptime Kuma | Embed — Qwen3 0.6B (TEI, nh3-ml1) → :8001/health (#29); Rerank — bge-v2-m3 (TEI, nh3-ml1) → :8013/health (#30) |
stacks/uptimekuma/monitors.yaml |
| Homepage | two cards under AI - Eval & Retrieval; dockerd on tcp/2375 bound to 10.100.50.80 |
stacks/homepage/conf/docker.yaml → nh3-ml1-docker |
| Dozzle | agent v10.4.1 on 10.100.50.80:7007, compose dir dozzle-agent; added to the hub's DOZZLE_REMOTE_AGENT |
hub on ana-docker :8088 |