esh-pfi-infrastructure/servers/esh-ml1/README.md

esh-ml1

GPU LXC for the ESH home lab: CT 110 on esh-pve, holding the NVIDIA RTX 2000E Ada (16 GB, 50 W, 01:00.0). It is the fleet's embedding and reranking service — the only backend behind the gateway's qwen3-embedding, reranker and reranker-a3-bge-v2-m3 since 2026-09-25. Built 2026-09-24.

⚠ Still the single gateway backend. The second RTX 2000 is live as nh3-ml1 (2026-09-25, parity-verified: the two hosts cannot be told apart), but LiteLLM does not route to it yet; that is Prime's call. Until it does, if esh-ml1, esh-pve, or ESH's mesh route (esh-scale, CT 108) is down, fleet embeddings and reranking are down: Worldtree recall, nevermore clustering, Open WebUI RAG.

IP	10.0.50.80/24, VLAN 50, gateway 10.0.50.1 (static, outside the UDM's .150–.250 DHCP pool)
DNS	esh-ml1.esh.internal
SSH	ssh esh-ml1 → infra-ops@10.0.50.80 (NOPASSWD sudo) · from the host: pct enter 110
OS	Debian 12, unprivileged, nesting=1,keyctl=1
Size	6 cores, 16 GB RAM + 2 GB swap, 80 GB rootfs on local-lvm
Boot	onboot: 1, startup: order=30 — after esh-scale (1), esh-vm-db (10) and esh-vm-docker (20), so a GPU fault never delays ESH's DNS or mesh route
Backups	None, on purpose. esh-pve's vzdump job lists vmids explicitly and 110 is not one. Everything is rebuilt from the playbooks and the stack; models re-download.

What it serves

stacks/embed-rerank (/opt/docker/compose/embed-rerank), Hugging Face TEI 1.9.4 — the fleet's embed/rerank engine from 2026-09-25 (Prime). It replaced vLLM here after a bake-off; see docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md.

container	model	port	gateway name (LiteLLM provider)
tei-embed	Qwen/Qwen3-Embedding-0.6B	8001	qwen3-embedding (hosted_vllm/, /v1)
tei-rerank	BAAI/bge-reranker-v2-m3	8013	reranker, reranker-a3-bge-v2-m3 (huggingface/, no /v1)

VRAM ~2.6 GB for both.

Also here since 2026-09-25: the reward seat — stacks/reward-seat, vllm-reward (Skywork-Reward-V2-Llama-3.1-8B AWQ, vLLM v0.24.0) on :8003, behind the gateway passthrough /scalar-judge. It is the one vLLM seat on the box, because TEI cannot serve a Llama classifier. Audit and parity: stacks/reward-seat/README.md. GPU total with all three: ~10.4 of 16.4 GB.

Cut-over verified 2026-09-25 through the gateway against fv-ml1's vLLM seats just before they were retired: embed cosine median 0.999927 / min 0.999881 (n=203); rerank top-1 and top-3 agreement 29/30. Two of the 30 lists contained the source paragraph twice, which makes #1 a tie either engine may break either way. The disagreeing query wasn't logged, so that is the likely cause, not a proven one.

History — the vLLM era (2026-09-24 → 25)

These measurements are of vLLM on this card vs vLLM on fv-ml1, and are kept for reference.

Parity, measured 2026-09-24 (11 texts incl. CJK, code, a 6k-char passage; 2 runs per site):

	median	min
embed cosine FV vs ESH, same text	0.999908	0.999772
noise floor FV vs FV	0.999927	0.999791
noise floor ESH vs ESH	0.999911	0.999809
negative control, different texts	0.232	0.071

The cross-site difference is inside each site's own run-to-run noise; this method cannot resolve a cosine gap below ~2×10⁻⁴. Reranker scores differed by at most 0.000145 (FV-vs-FV floor 0.000181), with identical ranking.

Speed vs fv-ml1, measured 2026-09-25 0628–0640 PT (same vLLM v0.24.0 and flags; fv-ml1 GPU 1 is shared with other seats but read 0% util before and after; synthetic text; each cell = median of 6 reps = 2 interleaved runs × 3):

workload	on-box FV	on-box ESH	from ana-docker FV	from ana-docker ESH
embed 1 query, p50	~12 ms	~9 ms	~29 ms	~23 ms
embed 1 × ~512 tok, p50	~15 ms	~23 ms	~31 ms	~31 ms
embed 1 × ~2k tok, p50	~25 ms	~86 ms	~48 ms	~90 ms
bulk embed, passages/s (64×512 tok, conc 4)	~403	~50	~374	~51
rerank 20 docs, p50	~51 ms	~170 ms	~82 ms	~175 ms
rerank 20 docs, req/s at conc 8	~55	~5.7	~52	~5.9

Read it as: single queries are a wash (ESH is ~6 ms faster from the gateway because it is closer — 6 ms RTT vs 17.6 ms), anything bulk is 3–10× slower on the 50 W Ada. Noise floor (run vs run, same site): ~3–5% on throughput, ±3 ms on small-query latency on fv-ml1, so the small-query rows cannot rank the sites. Positive control: 2k-token inputs were slower than 512-token ones on both sites. One fv-ml1 rep of the gateway-vantage rerank burst stalled to 1.0 req/s (64 requests in ~64 s); it did not recur in 11 other reps — unexplained, n=1. Consequence at the time: keep esh-ml1 as failover, don't load-share (superseded 2026-09-25 by the move) — sharing would roughly double rerank latency for half the fleet's calls.

Whole-novel embedding, measured 2026-09-25 0653–0717 PT — The Stand (uncut, ~1,150 pages): 470,783 words, 12,814 <p> paragraphs (median 22 words), 617,833 tokens. Client on nh3-dev (where worldtree-gateway runs); 3 reps each:

how	fv-ml1	esh-ml1
via gateway, 1 paragraph per request (1,000-para sample, scaled)	59.8 ms/para → ~12.8 min	55.3 ms/para → ~11.8 min
via gateway, 64 per request, 1 at a time	100–123 s	91–100 s
via gateway, 64 per request, 4 in flight	46–54 s	45–50 s
direct to the seat, 64 per request, 4 in flight	13.4–14.1 s	28.8–29.6 s

Through the gateway the GPU is not the bottleneck: one-per-request is network + LiteLLM overhead, and batched runs pinned LiteLLM at ~76% CPU (64×1024-float JSON per response). Only the direct path shows the card: esh-ml1 is ~2.2× slower, ~21k vs ~45k tokens/s.

VRAM: 0.20 × 16,380 MiB each; 4,823 MiB in use with both loaded, ~11 GB free.

How it is built

1. playbooks/pve-nvidia-host.yaml — driver 580.178.04 (open modules, DKMS) on the hypervisor, the nvidia-persistenced unit that creates the device nodes before pve-guests, and removal of the old VFIO/blacklist config.
2. playbooks/gpu-lxc.yaml — the CT, dev0–3 GPU nodes, the NVIDIA userspace from the same .run with --no-kernel-modules, fleet ids (infra-ops 850, docker 851, vh 1000), docker-ce and nvidia-container-toolkit with no-cgroups = true. Host-generic since 2026-09-25 (it also builds nh3-ml1). Its host vars have no defaults, so pass esh-ml1's set: the exact --var line is in the playbook header under "Run (esh-ml1)".
3. scripts/deploy-stack.sh esh-ml1 embed-rerank, then docker compose up -d.

Both playbooks are idempotent; re-run them to repair.

⚠ Driver version lock

The kernel module lives on esh-pve; the libraries live in this container. They must be the same version, or every CUDA call fails with "driver/library version mismatch". To upgrade: bump driver_version + driver_sha256 in the host playbook and driver_version in the LXC playbook, run the host one, then the LXC one, then restart the stack.

A PVE kernel update is handled by DKMS (proxmox-headers-6.8 pulls headers for each new kernel). Moving esh-pve to a different kernel series (6.14 opt-in) needs that series' headers meta-package installed first, or the module will not build and this CT will fail to start at the next boot.

Monitoring and telemetry (wired 2026-09-25)

layer	what	where
Beszel (host + GPU telemetry)	NVIDIA agent henrygd/beszel-agent-nvidia:0.18.7, stacks/beszel + hosts/esh-ml1.yaml. Hub system ridkfdpwfq3f730. GPU util, VRAM, power and temperature are sampled.	alerts → infra-ops: Status down 2 m, Disk >85% 5 m, CPU >95% 15 m, Memory >90% 10 m, Temperature >85 °C 5 m (GPU included)
Uptime Kuma (service)	Embed — Qwen3 0.6B (TEI, esh-ml1) → :8001/health (#27); Rerank — bge-v2-m3 (TEI, esh-ml1) → :8013/health (#28). TEI's health runs the backend.	alerts → infra-ops via althing-alert-bridge; stacks/uptimekuma/monitors.yaml
Homepage	three cards under AI - Eval & Retrieval. dockerd exposes tcp/2375 on 10.0.50.80 only (fleet norm, playbooks/gpu-lxc.yaml).	stacks/homepage/conf/docker.yaml → esh-ml1-docker
Dozzle (logs)	agent v10.4.1 on 10.0.50.80:7007, compose dir dozzle-agent	hub on ana-docker :8088

The reward seat has no Kuma check on purpose: seats are outside Kuma's lane, and this one has no working consumer. Beszel and Homepage cover it.

⚠ Disk: rootfs is ~66% used (vLLM image ~30 GB, TEI ~8 GB). The alert fires at 85%.

Path: every consumer reaches esh-ml1 through the gateway at ana-docker, so ESH callers hairpin ESH → Anaheim → ESH over the mesh. A mesh outage cuts every consumer off, the ones at ESH included.