esh-pfi-infrastructure/servers/nh3-ml1/README.md

nh3-ml1

GPU LXC for the NH3 site: CT 109 on nh3-pve, holding the NVIDIA RTX 2000E Ada (16 GB, 50 W, 01:00.0, PCIe gen4 x8). It is the second embedding and reranking backend, the twin of esh-ml1: same card, same driver, same TEI image and models. Built 2026-09-25, after the NH3 site visit turned Secure Boot off on nh3-pve.

⚠ Not behind the gateway yet. It serves on its own ports and is monitored, but LiteLLM still routes qwen3-embedding and reranker to esh-ml1 alone. Routing is Prime's call (recommendation: load-share; see below).

IP	10.100.50.80/24, VLAN 50 (nh3-servers), gateway 10.100.50.1 (static, outside the UDM's .150–.249 DHCP pool)
DNS	nh3-ml1.nh3.internal
SSH	ssh nh3-ml1 → infra-ops@10.100.50.80 (NOPASSWD sudo) · from the host: pct enter 109
OS	Debian 12, unprivileged, nesting=1,keyctl=1
Size	6 cores, 16 GB RAM + 2 GB swap, 80 GB rootfs on local-zfs
Boot	onboot: 1, startup: order=30, after the site's core guests, so a GPU fault never delays NH3's DNS or mesh route
Backups	None, on purpose. nh3-pve's vzdump job is all 1; the playbook added 109 to its exclude list. Everything is rebuilt from the playbooks and the stack; models re-download.

What it serves

stacks/embed-rerank (/opt/docker/compose/embed-rerank), TEI 1.9.4 (89-1.9.4). The live .env differs from .env.example only in HOST_NAME=nh3-ml1 and HOST_IP=10.100.50.80, which feed the Homepage labels.

container	model	port
tei-embed	Qwen/Qwen3-Embedding-0.6B	8001 (/v1/embeddings, /embed)
tei-rerank	BAAI/bge-reranker-v2-m3	8013 (/rerank, body query + texts)

VRAM ~2.7 GB for both, so ~13 GB is free.

Parity and speed vs esh-ml1 (2026-09-25 ~1540 PT)

Client on nh3-dev, calling both seats directly. Corpus: 1,120 paragraphs from this repo's docs plus 6 fixed texts (CJK, code, emoji, a 1-char input, a 6k-char passage); 50 instruction-format queries; each host embedded everything twice.

embedding check	result	noise floor / control
per-text cosine, esh vs nh3	median 0.999998, min 0.999993	esh vs esh 0.999998 / 0.999995; nh3 vs nh3 0.999998 / 0.999993
overlap@10, nh3 queries on the esh index	1.000	esh rerun 1.000; positive control MRL-256 truncation 0.684
overlap@10, nh3 index + nh3 queries	1.000	—
hit@1 own paragraph	0.76 (both hosts)	—
negative control, different texts	cosine median 0.50	—

rerank check (100 queries × 20 docs)	esh vs nh3	esh vs esh	positive control (query cut to 4 words)
top-1 agreement	1.00	1.00	0.94
top-5 exact order	0.98	0.97	0.09
max score difference	0.0014	0.0020	0.97

The two hosts cannot be told apart. Every esh-vs-nh3 figure sits inside the esh-vs-esh noise. Sensitivity floor: this method cannot resolve an embedding cosine gap below ~5×10⁻⁶ or a rerank score difference below ~0.002. An index built on either host serves queries from the other.

Speed, on-box, 3 interleaved reps per host (range across reps):

workload	nh3-ml1	esh-ml1
embed 1 short query, p50	6.60–6.74 ms	6.97–7.02 ms
bulk embed, 64 per request, passages/s	103.7–105.0	105.5–108.1
rerank 20 docs, p50	155.8–159.2 ms	151.8–160.5 ms

The two hosts run at the same speed. esh-ml1 was also holding the idle reward seat (~8 GB VRAM, 0% util) during these runs.

Gateway routing (Prime's call). The recommendation is load-share. The 2026-09-25 rule against load-sharing came from pairing esh-ml1 with the much faster fv-ml1. These two cards are identical, and a second site removes the single-host outage the esh-ml1 README warns about.

How it is built

1. playbooks/pve-nvidia-host.yaml on nh3-pve: driver 580.178.04 (open modules, DKMS) plus the nvidia-persistenced unit. It needs Secure Boot off, which was turned off in the BIOS on the 2026-09-25 visit; the pre-flight refuses otherwise.
2. playbooks/gpu-lxc.yaml with the "Run (nh3-ml1)" --var line from its header. ⚠ nh3-pve was on lxc-pve 6.0.0-1. With it, every docker run in the CT failed with "open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission denied" (runc 1.5 against the old AppArmor profile). Upgrading lxc-pve alone to 6.0.0-2 (Proxmox fix #7006) and then running pct reboot 109 fixed it. The playbook now does the upgrade as its first step.
3. scripts/deploy-stack.sh nh3-ml1 embed-rerank, then set HOST_NAME / HOST_IP in .env, then docker compose up -d.

The driver version lock and the DKMS/kernel notes in esh-ml1's README apply here unchanged. nh3-pve runs kernel 6.8.12-43, which the 2026-09-25 headers install pulled in (see servers/nh3-pve/README.md).

Monitoring and telemetry (wired 2026-09-25)

layer	what	where
Beszel	NVIDIA agent henrygd/beszel-agent-nvidia:0.18.7, stacks/beszel + hosts/nh3-ml1.yaml, hub system 1feeeq61g4mkqre; GPU util, VRAM and power are sampled	alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m
Uptime Kuma	Embed — Qwen3 0.6B (TEI, nh3-ml1) → :8001/health (#29); Rerank — bge-v2-m3 (TEI, nh3-ml1) → :8013/health (#30)	stacks/uptimekuma/monitors.yaml
Homepage	two cards under AI - Eval & Retrieval; dockerd on tcp/2375 bound to 10.100.50.80	stacks/homepage/conf/docker.yaml → nh3-ml1-docker
Dozzle	agent v10.4.1 on 10.100.50.80:7007, compose dir dozzle-agent; added to the hub's DOZZLE_REMOTE_AGENT	hub on ana-docker :8088