vh
7bdac80878
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet from now on; fv-ml1 retires both once esh-ml1 is up. - stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports 8001/8013, fail-closed truncation (--auto-truncate false; embed --max-batch-tokens 32768). - litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address); reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target. - Verified via the gateway against the retiring fv-ml1 seats: embed cosine median 0.999927 (n=203); rerank top-1/top-3 29/30. - stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged. - Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md; CLAUDE.md gains the TEI convention.
148 lines
7.6 KiB
Markdown
148 lines
7.6 KiB
Markdown
# esh-ml1
|
||
|
||
GPU LXC for the ESH home lab: **CT 110 on esh-pve**, holding the **NVIDIA RTX
|
||
2000E Ada** (16 GB, 50 W, `01:00.0`). **It is the fleet's embedding and
|
||
reranking service** — the only backend behind the gateway's `qwen3-embedding`,
|
||
`reranker` and `reranker-a3-bge-v2-m3` since 2026-09-25. Built 2026-09-24.
|
||
|
||
⚠ **Single backend until the second RTX 2000 arrives** (Prime, 2026-09-25). If
|
||
esh-ml1, esh-pve, or ESH's mesh route (esh-scale, CT 108) is down, fleet
|
||
embeddings and reranking are down: Worldtree recall, nevermore clustering, Open
|
||
WebUI RAG.
|
||
|
||
| | |
|
||
|---|---|
|
||
| **IP** | `10.0.50.80/24`, VLAN 50, gateway `10.0.50.1` (static, outside the UDM's `.150–.250` DHCP pool) |
|
||
| **DNS** | `esh-ml1.esh.internal` |
|
||
| **SSH** | `ssh esh-ml1` → `infra-ops@10.0.50.80` (NOPASSWD sudo) · from the host: `pct enter 110` |
|
||
| **OS** | Debian 12, unprivileged, `nesting=1,keyctl=1` |
|
||
| **Size** | 6 cores, 16 GB RAM + 2 GB swap, 80 GB rootfs on `local-lvm` |
|
||
| **Boot** | `onboot: 1`, `startup: order=30` — after esh-scale (1), esh-vm-db (10) and esh-vm-docker (20), so a GPU fault never delays ESH's DNS or mesh route |
|
||
| **Backups** | **None, on purpose.** esh-pve's vzdump job lists vmids explicitly and 110 is not one. Everything is rebuilt from the playbooks and the stack; models re-download. |
|
||
|
||
## What it serves
|
||
|
||
`stacks/embed-rerank` (`/opt/docker/compose/embed-rerank`), **Hugging Face TEI
|
||
1.9.4** — the fleet's embed/rerank engine from 2026-09-25 (Prime). It replaced
|
||
vLLM here after a bake-off; see
|
||
[`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`](../../docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md).
|
||
|
||
| container | model | port | gateway name (LiteLLM provider) |
|
||
|---|---|---|---|
|
||
| `tei-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 | `qwen3-embedding` (`hosted_vllm/`, `/v1`) |
|
||
| `tei-rerank` | `BAAI/bge-reranker-v2-m3` | 8013 | `reranker`, `reranker-a3-bge-v2-m3` (`huggingface/`, no `/v1`) |
|
||
|
||
VRAM ~2.6 GB for both; the rest of the 16 GB is free for further utility seats.
|
||
The vLLM image (`vllm/vllm-openai:v0.24.0`, ~30 GB) is still on disk: kept in
|
||
case a vLLM-only seat (the reward classifier) moves here.
|
||
|
||
**Cut-over verified 2026-09-25** through the gateway against fv-ml1's vLLM seats
|
||
just before they were retired: embed cosine median 0.999927 / min 0.999881
|
||
(n=203); rerank top-1 and top-3 agreement 29/30. Two of the 30 lists contained
|
||
the source paragraph twice, which makes #1 a tie either engine may break either
|
||
way. The disagreeing query wasn't logged, so that is the likely cause, not a
|
||
proven one.
|
||
|
||
### History — the vLLM era (2026-09-24 → 25)
|
||
|
||
These measurements are of vLLM on this card vs vLLM on fv-ml1, and are kept for
|
||
reference.
|
||
|
||
**Parity, measured 2026-09-24** (11 texts incl. CJK, code, a 6k-char passage;
|
||
2 runs per site):
|
||
|
||
| | median | min |
|
||
|---|---|---|
|
||
| embed cosine FV vs ESH, same text | 0.999908 | 0.999772 |
|
||
| noise floor FV vs FV | 0.999927 | 0.999791 |
|
||
| noise floor ESH vs ESH | 0.999911 | 0.999809 |
|
||
| negative control, different texts | 0.232 | 0.071 |
|
||
|
||
The cross-site difference is inside each site's own run-to-run noise; this
|
||
method cannot resolve a cosine gap below ~2×10⁻⁴. Reranker scores differed by
|
||
at most 0.000145 (FV-vs-FV floor 0.000181), with identical ranking.
|
||
|
||
**Speed vs fv-ml1, measured 2026-09-25 0628–0640 PT** (same vLLM v0.24.0 and
|
||
flags; fv-ml1 GPU 1 is shared with other seats but read 0% util before and
|
||
after; synthetic text; each cell = median of 6 reps = 2 interleaved runs × 3):
|
||
|
||
| workload | on-box FV | on-box ESH | from ana-docker FV | from ana-docker ESH |
|
||
|---|---|---|---|---|
|
||
| embed 1 query, p50 | ~12 ms | ~9 ms | ~29 ms | ~23 ms |
|
||
| embed 1 × ~512 tok, p50 | ~15 ms | ~23 ms | ~31 ms | ~31 ms |
|
||
| embed 1 × ~2k tok, p50 | ~25 ms | ~86 ms | ~48 ms | ~90 ms |
|
||
| bulk embed, passages/s (64×512 tok, conc 4) | ~403 | ~50 | ~374 | ~51 |
|
||
| rerank 20 docs, p50 | ~51 ms | ~170 ms | ~82 ms | ~175 ms |
|
||
| rerank 20 docs, req/s at conc 8 | ~55 | ~5.7 | ~52 | ~5.9 |
|
||
|
||
Read it as: **single queries are a wash** (ESH is ~6 ms faster from the gateway
|
||
because it is closer — 6 ms RTT vs 17.6 ms), **anything bulk is 3–10× slower**
|
||
on the 50 W Ada. Noise floor (run vs run, same site): ~3–5% on throughput, ±3 ms
|
||
on small-query latency on fv-ml1, so the small-query rows cannot rank the sites.
|
||
Positive control: 2k-token inputs were slower than 512-token ones on both
|
||
sites. One fv-ml1 rep of the gateway-vantage rerank burst stalled to 1.0 req/s
|
||
(64 requests in ~64 s); it did not recur in 11 other reps — unexplained, n=1.
|
||
**Consequence at the time: keep esh-ml1 as failover, don't load-share** (superseded 2026-09-25 by the move) — sharing would
|
||
roughly double rerank latency for half the fleet's calls.
|
||
|
||
**Whole-novel embedding, measured 2026-09-25 0653–0717 PT** — *The Stand*
|
||
(uncut, ~1,150 pages): 470,783 words, 12,814 `<p>` paragraphs (median 22 words),
|
||
617,833 tokens. Client on nh3-dev (where worldtree-gateway runs); 3 reps each:
|
||
|
||
| how | fv-ml1 | esh-ml1 |
|
||
|---|---|---|
|
||
| via gateway, 1 paragraph per request (1,000-para sample, scaled) | 59.8 ms/para → ~12.8 min | 55.3 ms/para → ~11.8 min |
|
||
| via gateway, 64 per request, 1 at a time | 100–123 s | 91–100 s |
|
||
| via gateway, 64 per request, 4 in flight | 46–54 s | 45–50 s |
|
||
| direct to the seat, 64 per request, 4 in flight | 13.4–14.1 s | 28.8–29.6 s |
|
||
|
||
Through the gateway the GPU is not the bottleneck: one-per-request is network +
|
||
LiteLLM overhead, and batched runs pinned LiteLLM at ~76% CPU (64×1024-float
|
||
JSON per response). Only the direct path shows the card: esh-ml1 is ~2.2×
|
||
slower, ~21k vs ~45k tokens/s.
|
||
|
||
**VRAM:** 0.20 × 16,380 MiB each; 4,823 MiB in use with both loaded, ~11 GB
|
||
free.
|
||
|
||
## How it is built
|
||
|
||
1. [`playbooks/esh-pve-nvidia-host.yaml`](../../playbooks/esh-pve-nvidia-host.yaml)
|
||
— driver **580.178.04** (open modules, DKMS) on the **hypervisor**, the
|
||
`nvidia-persistenced` unit that creates the device nodes before
|
||
`pve-guests`, and removal of the old VFIO/blacklist config.
|
||
2. [`playbooks/esh-ml1-lxc.yaml`](../../playbooks/esh-ml1-lxc.yaml) — the CT,
|
||
`dev0–3` GPU nodes, the NVIDIA userspace from the **same `.run`** with
|
||
`--no-kernel-modules`, fleet ids (infra-ops 850, docker 851, vh 1000),
|
||
docker-ce and nvidia-container-toolkit with `no-cgroups = true`.
|
||
3. `scripts/deploy-stack.sh esh-ml1 embed-rerank`, then `docker compose up -d`.
|
||
|
||
Both playbooks are idempotent; re-run them to repair.
|
||
|
||
## ⚠ Driver version lock
|
||
|
||
The kernel module lives on esh-pve; the libraries live in this container. They
|
||
**must be the same version**, or every CUDA call fails with *"driver/library
|
||
version mismatch"*. To upgrade: bump `driver_version` + `driver_sha256` in the
|
||
host playbook and `driver_version` in the LXC playbook, run the host one, then
|
||
the LXC one, then restart the stack.
|
||
|
||
A PVE kernel update is handled by DKMS (`proxmox-headers-6.8` pulls headers
|
||
for each new kernel). Moving esh-pve to a different kernel series (6.14 opt-in)
|
||
needs that series' headers meta-package installed first, or the module will not
|
||
build and this CT will fail to start at the next boot.
|
||
|
||
## Not yet wired
|
||
|
||
⚠ **esh-ml1 became load-bearing on 2026-09-25, so these are no longer
|
||
optional.** Nothing alerts if it dies today. The gateway will just start
|
||
returning errors for `qwen3-embedding` and `reranker`.
|
||
|
||
- **Beszel**: no agent yet.
|
||
- **Uptime Kuma**: no check on `:8001/health` / `:8013/health` yet.
|
||
- **Homepage**: the compose carries labels, but esh-ml1 is not in
|
||
`stacks/homepage/conf/docker.yaml` (it would need dockerd on tcp/2375 like
|
||
the other hosts).
|
||
- **Path:** every consumer reaches esh-ml1 through the gateway at ana-docker, so
|
||
ESH callers hairpin ESH → Anaheim → ESH over the mesh. A mesh outage cuts
|
||
every consumer off, the ones at ESH included.
|