feat(tei-bakeoff): TEI 1.9.4 vs vLLM on esh-ml1 — parity holds, not faster, much lighter

2026-09-25 08:05:17 -07:00
parent a1885718e9
commit 582b150131
4 changed files with 178 additions and 0 deletions
@@ -67,3 +67,14 @@ faster because it is network-closer). Bulk: ESH 3–10× slower (bulk embed ~50
 ~403 passages/s; rerank-20 ~170 vs ~51 ms p50, ~5.7 vs ~55 req/s at conc 8).
 Noise floor 3–5% on throughput. ⇒ failover role confirmed; load-sharing would
 roughly double rerank latency for half the calls.
+
+**TEI bake-off (2026-09-25, `stacks/tei-bakeoff/README.md`).** Prime asked to
+evaluate a purpose-built engine instead of vLLM → HF Text Embeddings Inference
+1.9.4 (Infinity ruled out: last release 2025-08-22, dropped by us in May for no
+Qwen3). Verdict: TEI **matches** (embed cosine vs vLLM-FV median 0.999925, inside
+noise; TEI queries against the OLD vLLM index overlap@10 0.988 vs 0.986 self-noise
+→ no re-embed needed; rerank decisions identical, tail scores move ≤0.019) but is
+**not faster** on the Ada (novel 40 s vs 31 s; tiny queries ~25% faster). It is
+**much lighter**: ~2.6 GB VRAM for both vs vLLM's 4.8 GB budget, 8 GB image vs 30,
+~4 s restart vs ~24 s. Gateway: embed drop-in via `hosted_vllm/`; rerank needs the
+`huggingface/` provider. Both engines left running side by side pending Prime's call.