feat(embed-rerank): TEI is the fleet embed/rerank engine; esh-ml1 sole backend; retire fv-ml1 seats
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet from now on; fv-ml1 retires both once esh-ml1 is up. - stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports 8001/8013, fail-closed truncation (--auto-truncate false; embed --max-batch-tokens 32768). - litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address); reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target. - Verified via the gateway against the retiring fv-ml1 seats: embed cosine median 0.999927 (n=203); rerank top-1/top-3 29/30. - stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged. - Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md; CLAUDE.md gains the TEI convention.
This commit is contained in:
@@ -0,0 +1,123 @@
|
||||
# Embed/rerank engine bake-off — TEI vs vLLM on esh-ml1 (2026-09-25)
|
||||
|
||||
Prime, 2026-09-25: *"run the TEI bake-off."* Hugging Face **Text Embeddings
|
||||
Inference 1.9.4** (`89-1.9.4`, the Ada build) served the same two models as the
|
||||
then-vLLM `embed-rerank` stack (vLLM v0.24.0), side by side on the same RTX 2000E
|
||||
Ada, as a temporary `tei-bakeoff` stack.
|
||||
|
||||
**Outcome (Prime, 2026-09-25): "TEI is embed/reranker server for esh-ml1 and the
|
||||
FLEET in general, in future."** The same day TEI replaced vLLM in
|
||||
[`stacks/embed-rerank`](../../stacks/embed-rerank/) on ports 8001/8013. The
|
||||
gateway was repointed with esh-ml1 as the sole backend, and fv-ml1's
|
||||
`vllm-embed` + `vllm-rerank-a3` seats were retired, freeing ~6.1 GB on fv-ml1
|
||||
GPU 1. The bake-off stack was removed; this file is its record.
|
||||
|
||||
## Results (2026-09-25 0744–0805 PT)
|
||||
|
||||
### Parity — TEI matches the existing vLLM vectors
|
||||
|
||||
Reference = vLLM on fv-ml1, the engine every existing index was built with.
|
||||
306 texts (6 fixed incl. CJK/code/6k-char + 300 *The Stand* paragraphs);
|
||||
retrieval = 2,000-paragraph corpus, 50 instruction-format queries.
|
||||
|
||||
| embedding check | result | noise floor / control |
|
||||
|---|---|---|
|
||||
| cosine TEI vs vLLM-FV | median 0.999925, min 0.999861 | vLLM-FV vs itself: 0.999916 / 0.999796 |
|
||||
| overlap@10, TEI index + TEI queries | 0.976 | vLLM-FV rerun 0.986; vLLM-ESH 0.972 |
|
||||
| overlap@10, **TEI queries vs the OLD vLLM index** (migration case) | **0.988** | positive control (MRL-256 dims) 0.648 |
|
||||
| hit@1 own paragraph | 0.80 | vLLM-FV 0.80 |
|
||||
| different-text negative control | cosine median 0.31 | — |
|
||||
|
||||
TEI is deterministic (TEI vs itself: min 0.999995). **Switching engines does not
|
||||
require re-embedding existing indexes.**
|
||||
|
||||
| rerank check (100 queries × 20 docs, 9 same-chapter distractors) | TEI vs vLLM-FV | vLLM-FV vs itself |
|
||||
|---|---|---|
|
||||
| source paragraph ranked #1 | 0.95 (same as vLLM) | 0.95 |
|
||||
| top-1 agreement | 1.00 | 1.00 |
|
||||
| top-3 set agreement | 1.00 | 1.00 |
|
||||
| top-5 exact order | 0.98 | 1.00 |
|
||||
| order among docs scoring > 0.05 | 1.00 (n=37) | 1.00 |
|
||||
| full 20-doc order | **0.55** | 0.99 |
|
||||
| max score difference | **0.019** (p99 0.0034) | 0.0016 |
|
||||
|
||||
Every decision that matters agrees; the differences are shuffles among
|
||||
near-zero-scoring tail documents. ⚠ A consumer that **thresholds** on the rerank
|
||||
score could see a borderline document flip (scores move up to ~0.02). An
|
||||
earlier 30-query run with random distractors had one top-1 disagreement (29/30);
|
||||
across both runs, 129/130.
|
||||
|
||||
### Speed — TEI is NOT faster on this card
|
||||
|
||||
On-box, 2 interleaved runs × 3 reps each, medians:
|
||||
|
||||
| workload | vLLM | TEI |
|
||||
|---|---|---|
|
||||
| embed 1 short query, p50 | ~9.1 ms | **~6.9 ms** |
|
||||
| embed 1 × ~512 tok, p50 | ~23 ms (bimodal 10–25) | ~28 ms |
|
||||
| embed 1 × ~2k tok, p50 | **~86 ms** | ~108 ms |
|
||||
| bulk embed, passages/s | **~50** | ~37 |
|
||||
| whole novel (*The Stand*, 12,814 paragraphs), 64/request, 4 in flight | **31 s** | 40 s |
|
||||
| rerank 20 docs, p50 | **~169 ms** | ~180 ms |
|
||||
| rerank 20 docs, req/s at conc 8 | ~5.7 | ~5.6 |
|
||||
|
||||
TEI ran its fused `FlashQwen3` path. `--max-batch-tokens 32768` did not change
|
||||
the novel time (39–40 s), so it stays at the default. **Through the gateway these
|
||||
gaps mostly vanish**: LiteLLM is the bottleneck there (see
|
||||
[`servers/esh-ml1/README.md`](../../servers/esh-ml1/README.md)).
|
||||
|
||||
### Footprint — TEI is much lighter
|
||||
|
||||
| | vLLM (both) | TEI (both) |
|
||||
|---|---|---|
|
||||
| VRAM (host `nvidia-smi`, per process) | 3,298 + 1,512 MiB | 1,352 + 1,256 MiB |
|
||||
| image | 29.9 GB | 8.16 GB |
|
||||
| warm restart to healthy (n=3) | ~24 s | ~4 s |
|
||||
| container RAM just after start | 2.5 + 5.1 GiB | 0.7 + 1.7 GiB |
|
||||
|
||||
⚠ vLLM's VRAM figure is mostly a **setting** (`--gpu-memory-utilization 0.20`
|
||||
each). A lower setting would shrink it; that has not been tested.
|
||||
|
||||
### External review — Dvalin (Grok research peer), 2026-09-25
|
||||
|
||||
**Agree with caveats: adopt TEI for these two seats; footprint is the right
|
||||
reason.** His external evidence (not re-verified here): a Runpod 2026-09-14
|
||||
engine comparison shows vLLM ahead of TEI on Qwen3-Embedding bulk throughput
|
||||
(median ~2.6×, single unreplicated runs, near parity on smaller cards) and TEI
|
||||
ahead on BERT-family models — consistent with our 1.3× on a 50 W card and the
|
||||
rerank near-tie. Open TEI 1.9 issue #857 (tokio panic "No backend receiver"
|
||||
under load, n=1) is a watch item, not a gate. A lower vLLM memory setting
|
||||
would shrink only the embedder (the reranker at 1,512 MiB is already under
|
||||
its cap); 0.12 is the only setting he'd expect to both boot and help — untested.
|
||||
His caveats, and status: fail-closed truncation (**done**, above); recheck score-
|
||||
threshold consumers (**done**, nevermore, above); two gateway providers must be
|
||||
written down at cut-over (open).
|
||||
|
||||
## Gateway compatibility (LiteLLM v1.97, temp aliases, deleted after)
|
||||
|
||||
- Embeddings: **drop-in**. TEI's `/v1/embeddings` works with the existing
|
||||
`hosted_vllm/` provider.
|
||||
- Rerank: needs the **`huggingface/` provider** (scores identical to direct
|
||||
TEI). `hosted_vllm/` fails with 422 (`missing field texts`); TEI's `/rerank`
|
||||
body differs from vLLM's.
|
||||
|
||||
## Behaviour differences to carry into any cut-over
|
||||
|
||||
- **Truncation — now fail-closed (`--auto-truncate false`).** Measured with
|
||||
TEI's default: a ~40k-token input returned **200 with a vector of its first
|
||||
16,384 tokens**, silently, where vLLM returns 400. With truncation off, TEI
|
||||
embed rejects > 32,768 tokens and TEI rerank rejects > 8,192 (both 422,
|
||||
verified). Truncation off requires `--max-batch-tokens` at or above the model's max
|
||||
input (32,768 for Qwen3-Embedding) or TEI refuses to start; VRAM unchanged.
|
||||
Remaining difference: TEI embed **accepts** 8,193–32,767 tokens that vLLM
|
||||
(`--max-model-len 8192`) rejects. It embeds the whole input, which is wider,
|
||||
not lossy.
|
||||
- **nevermore thresholds rerank scores** (`NEVERMORE_RERANK_THRESHOLD`, default
|
||||
0.3, cluster-member filter in `digest.py`). Across the 2,000 bake-off scores,
|
||||
**0 flips** at 0.3 or 0.5 for TEI vs vLLM — but only 1 score landed within
|
||||
±0.02 of 0.3 (bge scores sit near 0 or 1), so this cannot exclude rare flips
|
||||
on borderline headlines. Worst case: a same-story headline clusters
|
||||
differently. Low stakes.
|
||||
- TEI caps inputs per request at `--max-client-batch-size` (default 32; set to 128 here).
|
||||
- TEI on CUDA runs float16 only (vLLM served Qwen3-Embedding in bf16); parity
|
||||
above says it does not matter here.
|
||||
@@ -124,8 +124,6 @@ context.
|
||||
| `gen-small-reasoning` | 8026 |
|
||||
| `image-judge` | 8022 |
|
||||
| `qwen-image-bench` | 8022 |
|
||||
| `qwen3-embedding` | 8001 |
|
||||
| `reranker` | 8013 |
|
||||
| `sec` | 8025 |
|
||||
| `sec-reasoning` | 8025 |
|
||||
| `summarizer` | 8026 |
|
||||
@@ -138,3 +136,7 @@ context.
|
||||
*Regenerate with `scripts/seat-inventory.py` after ANY seat change —
|
||||
model swap, quant change, context or utilization edit, or speculative-decoding
|
||||
change. Run `--check` in CI to catch a stale document.*
|
||||
|
||||
> **2026-09-25:** `qwen3-embedding` (:8001) and `reranker` (:8013) LEFT fv-ml1 —
|
||||
> their vLLM seats were retired and the names now route to TEI on esh-ml1
|
||||
> (`stacks/embed-rerank`). ~6.1 GB freed on GPU 1.
|
||||
|
||||
@@ -243,3 +243,12 @@ on fv-ml1; `/model/delete` the two aliases (DB-persisted); weights retained in H
|
||||
form, not load-balancing: fv-ml1 serves every request while it is up. Parity: |score|
|
||||
FV-vs-ESH max 0.000145 vs FV-vs-FV floor 0.000181, identical ranking over 11 documents.
|
||||
See `servers/esh-ml1/README.md`.
|
||||
|
||||
## 2026-09-25 — the reranker moved to TEI on esh-ml1 (Prime)
|
||||
|
||||
- TEI adopted as the fleet embed/rerank engine after a bake-off
|
||||
(`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`). `vllm-rerank-a3` on fv-ml1 was
|
||||
**retired**; `reranker` and the DB alias `reranker-a3-bge-v2-m3` now both route to
|
||||
`tei-rerank` on esh-ml1 :8013 via LiteLLM's `huggingface/` provider. Same model
|
||||
(bge-reranker-v2-m3): top-1/top-3 identical to the retired seat, and scores move
|
||||
up to 0.019 in the near-zero tail.
|
||||
|
||||
Reference in New Issue
Block a user