esh-pfi-infrastructure/servers/nh3-ml1/README.md

nh3-ml1

GPU LXC for the NH3 site: CT 109 on nh3-pve, holding the NVIDIA RTX 2000E Ada (16 GB, 50 W, 01:00.0, PCIe gen4 x8). It is the second embedding and reranking backend, the twin of esh-ml1: same card, same driver, same TEI image and models. Built 2026-09-25, after the NH3 site visit turned Secure Boot off on nh3-pve.

Embedding routing since 2026-10-05 0030 (Prime): primary + failover, no longer shuffled. qwen3-embedding (fleet default) = esh-ml1 order 1, nh3-ml1 order 2; qwen3-embedding-bulk (albok-service's write path) = nh3-ml1 order 1, esh-ml1 order 2. Measured 0730: 6/6 of each alias on its primary; with nh3-ml1's tei-embed stopped, bulk went 6/6 to esh-ml1 with 200s and fleet stayed on esh-ml1, then bulk returned 6/6 to nh3-ml1 once it was back. (esh-ml1-down → fleet failover is the same code path, not separately tested.) The reranker groups are still load-shared as below.

Load-shared with esh-ml1 since 2026-09-26 (Prime); embeddings moved to primary/failover 2026-10-05, above. The gateway split qwen3-embedding, reranker and reranker-a3-bge-v2-m3 across both hosts at random, one simple-shuffle deployment each. Measured with 40 + 40 requests: embed 12 esh / 28 nh3, rerank 22 / 18.

Failover, measured by stopping this host's TEI (2026-09-26 0747–0753):

• Without failover settings: 7/40 embeds returned 500. The router's default 2 retries re-picked at random and hit the dead backend again.
• With router_settings.enable_weighted_failover: true, which only affects multi-deployment groups:
  • rerank 80/80 OK.
  • embeds 38/40 in the first test (2 failures at outage onset) and 60/60 across a 34 s sustained outage.
  • Once TEI restarted, nh3 was back in rotation (15/30).
• Residual: in LiteLLM 1.97 the embedding path does not stamp the failed deployment, so embeds rely on the router's cooldown rather than per-request re-pick. Expect a couple of failed embeds when a backend first dies. n = 2 outages.

IP	10.100.50.80/24, VLAN 50 (nh3-servers), gateway 10.100.50.1 (static, outside the UDM's .150–.249 DHCP pool)
DNS	nh3-ml1.nh3.internal
SSH	ssh nh3-ml1 → infra-ops@10.100.50.80 (NOPASSWD sudo) · from the host: pct enter 109
OS	Debian 12, unprivileged, nesting=1,keyctl=1
Size	6 cores, 16 GB RAM + 2 GB swap, 80 GB rootfs on local-zfs
Boot	onboot: 1, startup: order=30, after the site's core guests, so a GPU fault never delays NH3's DNS or mesh route
Backups	None, on purpose. nh3-pve's vzdump job is all 1; the playbook added 109 to its exclude list. Everything is rebuilt from the playbooks and the stack; models re-download.

What it serves

stacks/embed-rerank (/opt/docker/compose/embed-rerank), TEI 1.9.4 (89-1.9.4). The live .env differs from .env.example only in HOST_NAME=nh3-ml1 and HOST_IP=10.100.50.80, which feed the Homepage labels.

container	model	port
tei-embed	Qwen/Qwen3-Embedding-0.6B	8001 (/v1/embeddings, /embed)
tei-rerank	BAAI/bge-reranker-v2-m3	8013 (/rerank, body query + texts)

VRAM ~2.7 GB for both, so ~13 GB is free.

Dataset-foundry utility seats (2026-09-26, brokkr; operator-approved):

• stacks/lfm-vl-seat: LFM2.5-VL-3B on llama.cpp, :8030, gateway lfm25-vl-3b.

• stacks/vibevoice-asr-seat: VibeVoice-ASR-Streaming-1.5B on audio.cpp, :8031, direct only.

• stacks/lfm-vl-uncensored-seat: an abliterated LFM2.5-VL-3B running in parallel for brokkr's NSFW-caption A/B, :8032, direct only.

GPU total with TEI is ~11.3 of 16 GB (the ASR seat moved to Q8_0 on 2026-09-26). All of these are batch workloads, which suits this card.

Tried and removed (2026-09-25): a copy of the code-completion seat (Qwen2.5-Coder-1.5B). It gave the same quality but ran about 5× slower than on fv-ml1 (a 64-token FIM completion took ~1.0 s vs ~0.2 s), so it stays on fv-ml1 (Prime). The recipe and measurements are in stacks/coder-seat/README.md. Lesson for the next candidate: this card has about a fifth of a Blackwell's decode speed on small generative models, so it suits embedding, reranking and classification, not latency-sensitive generation.

Parity and speed vs esh-ml1 (2026-09-25 ~1540 PT)

Client on nh3-dev, calling both seats directly. Corpus: 1,120 paragraphs from this repo's docs plus 6 fixed texts (CJK, code, emoji, a 1-char input, a 6k-char passage); 50 instruction-format queries; each host embedded everything twice.

embedding check	result	noise floor / control
per-text cosine, esh vs nh3	median 0.999998, min 0.999993	esh vs esh 0.999998 / 0.999995; nh3 vs nh3 0.999998 / 0.999993
overlap@10, nh3 queries on the esh index	1.000	esh rerun 1.000; positive control MRL-256 truncation 0.684
overlap@10, nh3 index + nh3 queries	1.000	—
hit@1 own paragraph	0.76 (both hosts)	—
negative control, different texts	cosine median 0.50	—

rerank check (100 queries × 20 docs)	esh vs nh3	esh vs esh	positive control (query cut to 4 words)
top-1 agreement	1.00	1.00	0.94
top-5 exact order	0.98	0.97	0.09
max score difference	0.0014	0.0020	0.97

The two hosts cannot be told apart. Every esh-vs-nh3 figure sits inside the esh-vs-esh noise. Sensitivity floor: this method cannot resolve an embedding cosine gap below ~5×10⁻⁶ or a rerank score difference below ~0.002. An index built on either host serves queries from the other.

Speed, on-box, 3 interleaved reps per host (range across reps):

workload	nh3-ml1	esh-ml1
embed 1 short query, p50	6.60–6.74 ms	6.97–7.02 ms
bulk embed, 64 per request, passages/s	103.7–105.0	105.5–108.1
rerank 20 docs, p50	155.8–159.2 ms	151.8–160.5 ms

The two hosts run at the same speed. esh-ml1 was also holding the idle reward seat (~8 GB VRAM, 0% util) during these runs.

Gateway routing (Prime's call). The recommendation is load-share. The 2026-09-25 rule against load-sharing came from pairing esh-ml1 with the much faster fv-ml1. These two cards are identical, and a second site removes the single-host outage the esh-ml1 README warns about.

How it is built

1. playbooks/pve-nvidia-host.yaml on nh3-pve: driver 580.178.04 (open modules, DKMS) plus the nvidia-persistenced unit. It needs Secure Boot off, which was turned off in the BIOS on the 2026-09-25 visit; the pre-flight refuses otherwise.
2. playbooks/gpu-lxc.yaml with the "Run (nh3-ml1)" --var line from its header. ⚠ nh3-pve was on lxc-pve 6.0.0-1. With it, every docker run in the CT failed with "open sysctl net.ipv4.ip_unprivileged_port_start file: reopen fd 8: permission denied" (runc 1.5 against the old AppArmor profile). Upgrading lxc-pve alone to 6.0.0-2 (Proxmox fix #7006) and then running pct reboot 109 fixed it. The playbook now does the upgrade as its first step.
3. scripts/deploy-stack.sh nh3-ml1 embed-rerank, then set HOST_NAME / HOST_IP in .env, then docker compose up -d.

The driver version lock and the DKMS/kernel notes in esh-ml1's README apply here unchanged. nh3-pve runs kernel 6.8.12-43, which the 2026-09-25 headers install pulled in (see servers/nh3-pve/README.md).

Monitoring and telemetry (wired 2026-09-25)

layer	what	where
Beszel	NVIDIA agent henrygd/beszel-agent-nvidia:0.18.7, stacks/beszel + hosts/nh3-ml1.yaml, hub system 1feeeq61g4mkqre; GPU util, VRAM and power are sampled	alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m (GPU + NVMe only: the agent drops the host's coretemp_*/acpitz with SENSORS, because an LXC reads nh3-pve's CPU sensors and the first alert, 2026-09-25 2105, was nh3-pve's CPU during vzdump). nh3-pve's own system carries the CPU alert at >95 °C
Uptime Kuma	Embed — Qwen3 0.6B (TEI, nh3-ml1) → :8001/health (#29); Rerank — bge-v2-m3 (TEI, nh3-ml1) → :8013/health (#30)	stacks/uptimekuma/monitors.yaml
Homepage	two cards under AI - Eval & Retrieval; dockerd on tcp/2375 bound to 10.100.50.80	stacks/homepage/conf/docker.yaml → nh3-ml1-docker
Dozzle	agent v10.4.1 on 10.100.50.80:7007, compose dir dozzle-agent; added to the hub's DOZZLE_REMOTE_AGENT	hub on ana-docker :8088