90 lines
4.0 KiB
Markdown
90 lines
4.0 KiB
Markdown
# tei-bakeoff — TEI vs vLLM for embedding + reranking on esh-ml1
|
||
|
||
Prime, 2026-09-25: *"run the TEI bake-off."* Hugging Face **Text Embeddings
|
||
Inference 1.9.4** (`89-1.9.4`, the Ada build) serving the same two models as
|
||
[`embed-rerank`](../embed-rerank/) (vLLM v0.24.0), side by side on the same RTX
|
||
2000E Ada. Not wired into the gateway. This stack either becomes the
|
||
embed-rerank stack or is torn down.
|
||
|
||
## Results (2026-09-25 0744–0805 PT)
|
||
|
||
### Parity — TEI matches the existing vLLM vectors
|
||
|
||
Reference = vLLM on fv-ml1, the engine every existing index was built with.
|
||
306 texts (6 fixed incl. CJK/code/6k-char + 300 *The Stand* paragraphs);
|
||
retrieval = 2,000-paragraph corpus, 50 instruction-format queries.
|
||
|
||
| embedding check | result | noise floor / control |
|
||
|---|---|---|
|
||
| cosine TEI vs vLLM-FV | median 0.999925, min 0.999861 | vLLM-FV vs itself: 0.999916 / 0.999796 |
|
||
| overlap@10, TEI index + TEI queries | 0.976 | vLLM-FV rerun 0.986; vLLM-ESH 0.972 |
|
||
| overlap@10, **TEI queries vs the OLD vLLM index** (migration case) | **0.988** | positive control (MRL-256 dims) 0.648 |
|
||
| hit@1 own paragraph | 0.80 | vLLM-FV 0.80 |
|
||
| different-text negative control | cosine median 0.31 | — |
|
||
|
||
TEI is deterministic (TEI vs itself: min 0.999995). **Switching engines does not
|
||
require re-embedding existing indexes.**
|
||
|
||
| rerank check (100 queries × 20 docs, 9 same-chapter distractors) | TEI vs vLLM-FV | vLLM-FV vs itself |
|
||
|---|---|---|
|
||
| source paragraph ranked #1 | 0.95 (same as vLLM) | 0.95 |
|
||
| top-1 agreement | 1.00 | 1.00 |
|
||
| top-3 set agreement | 1.00 | 1.00 |
|
||
| top-5 exact order | 0.98 | 1.00 |
|
||
| order among docs scoring > 0.05 | 1.00 (n=37) | 1.00 |
|
||
| full 20-doc order | **0.55** | 0.99 |
|
||
| max score difference | **0.019** (p99 0.0034) | 0.0016 |
|
||
|
||
Every decision that matters agrees; the differences are shuffles among
|
||
near-zero-scoring tail documents. ⚠ A consumer that **thresholds** on the rerank
|
||
score could see a borderline document flip (scores move up to ~0.02). An
|
||
earlier 30-query run with random distractors had one top-1 disagreement (29/30);
|
||
across both runs, 129/130.
|
||
|
||
### Speed — TEI is NOT faster on this card
|
||
|
||
On-box, 2 interleaved runs × 3 reps each, medians:
|
||
|
||
| workload | vLLM | TEI |
|
||
|---|---|---|
|
||
| embed 1 short query, p50 | ~9.1 ms | **~6.9 ms** |
|
||
| embed 1 × ~512 tok, p50 | ~23 ms (bimodal 10–25) | ~28 ms |
|
||
| embed 1 × ~2k tok, p50 | **~86 ms** | ~108 ms |
|
||
| bulk embed, passages/s | **~50** | ~37 |
|
||
| whole novel (*The Stand*, 12,814 paragraphs), 64/request, 4 in flight | **31 s** | 40 s |
|
||
| rerank 20 docs, p50 | **~169 ms** | ~180 ms |
|
||
| rerank 20 docs, req/s at conc 8 | ~5.7 | ~5.6 |
|
||
|
||
TEI ran its fused `FlashQwen3` path. `--max-batch-tokens 32768` did not change
|
||
the novel time (39–40 s), so it stays at the default. **Through the gateway these
|
||
gaps mostly vanish**: LiteLLM is the bottleneck there (see
|
||
`servers/esh-ml1/README.md`).
|
||
|
||
### Footprint — TEI is much lighter
|
||
|
||
| | vLLM (both) | TEI (both) |
|
||
|---|---|---|
|
||
| VRAM (host `nvidia-smi`, per process) | 3,298 + 1,512 MiB | 1,352 + 1,256 MiB |
|
||
| image | 29.9 GB | 8.16 GB |
|
||
| warm restart to healthy (n=3) | ~24 s | ~4 s |
|
||
| container RAM just after start | 2.5 + 5.1 GiB | 0.7 + 1.7 GiB |
|
||
|
||
⚠ vLLM's VRAM figure is mostly a **setting** (`--gpu-memory-utilization 0.20`
|
||
each). A lower setting would shrink it; that has not been tested.
|
||
|
||
### Gateway compatibility (LiteLLM v1.97, temp aliases, deleted after)
|
||
|
||
- Embeddings: **drop-in**. TEI's `/v1/embeddings` works with the existing
|
||
`hosted_vllm/` provider.
|
||
- Rerank: needs the **`huggingface/` provider** (scores identical to direct
|
||
TEI). `hosted_vllm/` fails with 422 (`missing field texts`); TEI's `/rerank`
|
||
body differs from vLLM's.
|
||
|
||
## Behaviour differences to carry into any cut-over
|
||
|
||
- TEI **auto-truncates** over-length input by default; vLLM (at
|
||
`--max-model-len 8192`) rejects it with an error.
|
||
- TEI caps inputs per request at `--max-client-batch-size` (default 32; set to 128 here).
|
||
- TEI on CUDA runs float16 only (vLLM served Qwen3-Embedding in bf16); parity
|
||
above says it does not matter here.
|