vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(embed-rerank): TEI is the fleet embed/rerank engine; esh-ml1 sole backend; retire fv-ml1 seats

Browse Source 
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet
from now on; fv-ml1 retires both once esh-ml1 is up.

- stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports
  8001/8013, fail-closed truncation (--auto-truncate false; embed
  --max-batch-tokens 32768).
- litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address);
  reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's
  `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target.
- Verified via the gateway against the retiring fv-ml1 seats: embed cosine
  median 0.999927 (n=203); rerank top-1/top-3 29/30.
- stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on
  fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged.
- Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md;
  CLAUDE.md gains the TEI convention.
This commit is contained in:
vh
2026-09-25 08:30:53 -07:00
parent 49b4bf0177
commit 7bdac80878
17 changed files with 271 additions and 418 deletions
Download Patch File Download Diff File Expand all files Collapse all files
servers/esh-ml1/README.md
+40 -15
View File
@@ -1,8 +1,14 @@
# esh-ml1
GPU LXC for the ESH home lab: **CT 110 on esh-pve**, holding the **NVIDIA RTX
2000E Ada** (16 GB, 50 W, `01:00.0`). It serves the fleet's embedding and
reranking models locally at ESH. Built 2026-09-24.
2000E Ada** (16 GB, 50 W, `01:00.0`). **It is the fleet's embedding and
reranking service** — the only backend behind the gateway's `qwen3-embedding`,
`reranker` and `reranker-a3-bge-v2-m3` since 2026-09-25. Built 2026-09-24.
⚠ **Single backend until the second RTX 2000 arrives** (Prime, 2026-09-25). If
esh-ml1, esh-pve, or ESH's mesh route (esh-scale, CT 108) is down, fleet
embeddings and reranking are down: Worldtree recall, nevermore clustering, Open
WebUI RAG.
| | |
|---|---|
@@ -16,17 +22,31 @@ reranking models locally at ESH. Built 2026-09-24.
## What it serves
`stacks/embed-rerank` (`/opt/docker/compose/embed-rerank`):
`stacks/embed-rerank` (`/opt/docker/compose/embed-rerank`), **Hugging Face TEI
1.9.4** — the fleet's embed/rerank engine from 2026-09-25 (Prime). It replaced
vLLM here after a bake-off; see
[`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`](../../docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md).
| container | model | port | gateway name |
| container | model | port | gateway name (LiteLLM provider) |
|---|---|---|---|
| `vllm-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 | `qwen3-embedding` (order 2) |
| `vllm-rerank-bge` | `BAAI/bge-reranker-v2-m3` | 8013 | `reranker` (order 2) |
| `tei-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 | `qwen3-embedding` (`hosted_vllm/`, `/v1`) |
| `tei-rerank` | `BAAI/bge-reranker-v2-m3` | 8013 | `reranker`, `reranker-a3-bge-v2-m3` (`huggingface/`, no `/v1`) |
The same models, vLLM version (`v0.24.0`, digest `251eba5cc7c1`) and flags as
fv-ml1's `vllm` stack, so the two sites are interchangeable. In LiteLLM they
are the **order-2 failover** behind fv-ml1: fv-ml1 serves every request while
it is up.
VRAM ~2.6 GB for both; the rest of the 16 GB is free for further utility seats.
The vLLM image (`vllm/vllm-openai:v0.24.0`, ~30 GB) is still on disk: kept in
case a vLLM-only seat (the reward classifier) moves here.
**Cut-over verified 2026-09-25** through the gateway against fv-ml1's vLLM seats
just before they were retired: embed cosine median 0.999927 / min 0.999881
(n=203); rerank top-1 and top-3 agreement 29/30. Two of the 30 lists contained
the source paragraph twice, which makes #1 a tie either engine may break either
way. The disagreeing query wasn't logged, so that is the likely cause, not a
proven one.
### History — the vLLM era (2026-09-24 → 25)
These measurements are of vLLM on this card vs vLLM on fv-ml1, and are kept for
reference.
**Parity, measured 2026-09-24** (11 texts incl. CJK, code, a 6k-char passage;
2 runs per site):
@@ -62,7 +82,7 @@ on small-query latency on fv-ml1, so the small-query rows cannot rank the sites.
Positive control: 2k-token inputs were slower than 512-token ones on both
sites. One fv-ml1 rep of the gateway-vantage rerank burst stalled to 1.0 req/s
(64 requests in ~64 s); it did not recur in 11 other reps — unexplained, n=1.
**Consequence: keep esh-ml1 as failover, don't load-share** — sharing would
**Consequence at the time: keep esh-ml1 as failover, don't load-share** (superseded 2026-09-25 by the move) — sharing would
roughly double rerank latency for half the fleet's calls.
**Whole-novel embedding, measured 2026-09-25 0653–0717 PT** — *The Stand*
@@ -113,10 +133,15 @@ build and this CT will fail to start at the next boot.
## Not yet wired
⚠ **esh-ml1 became load-bearing on 2026-09-25, so these are no longer
optional.** Nothing alerts if it dies today. The gateway will just start
returning errors for `qwen3-embedding` and `reranker`.
- **Beszel**: no agent yet.
- **Uptime Kuma**: no check on `:8001/health` / `:8013/health` yet.
- **Homepage**: the compose carries labels, but esh-ml1 is not in
`stacks/homepage/conf/docker.yaml` (it would need dockerd on tcp/2375 like
the other hosts).
- **Beszel**: no agent yet.
- **ESH consumers** (Open WebUI RAG, Paperless) still go through the gateway
at ana-docker, so they do not survive a mesh outage. See persistent-memory
for the open decision.
- **Path:** every consumer reaches esh-ml1 through the gateway at ana-docker, so
ESH callers hairpin ESH → Anaheim → ESH over the mesh. A mesh outage cuts
every consumer off, the ones at ESH included.