4.0 KiB
tei-bakeoff — TEI vs vLLM for embedding + reranking on esh-ml1
Prime, 2026-09-25: "run the TEI bake-off." Hugging Face Text Embeddings
Inference 1.9.4 (89-1.9.4, the Ada build) serving the same two models as
embed-rerank (vLLM v0.24.0), side by side on the same RTX
2000E Ada. Not wired into the gateway. This stack either becomes the
embed-rerank stack or is torn down.
Results (2026-09-25 0744–0805 PT)
Parity — TEI matches the existing vLLM vectors
Reference = vLLM on fv-ml1, the engine every existing index was built with. 306 texts (6 fixed incl. CJK/code/6k-char + 300 The Stand paragraphs); retrieval = 2,000-paragraph corpus, 50 instruction-format queries.
| embedding check | result | noise floor / control |
|---|---|---|
| cosine TEI vs vLLM-FV | median 0.999925, min 0.999861 | vLLM-FV vs itself: 0.999916 / 0.999796 |
| overlap@10, TEI index + TEI queries | 0.976 | vLLM-FV rerun 0.986; vLLM-ESH 0.972 |
| overlap@10, TEI queries vs the OLD vLLM index (migration case) | 0.988 | positive control (MRL-256 dims) 0.648 |
| hit@1 own paragraph | 0.80 | vLLM-FV 0.80 |
| different-text negative control | cosine median 0.31 | — |
TEI is deterministic (TEI vs itself: min 0.999995). Switching engines does not require re-embedding existing indexes.
| rerank check (100 queries × 20 docs, 9 same-chapter distractors) | TEI vs vLLM-FV | vLLM-FV vs itself |
|---|---|---|
| source paragraph ranked #1 | 0.95 (same as vLLM) | 0.95 |
| top-1 agreement | 1.00 | 1.00 |
| top-3 set agreement | 1.00 | 1.00 |
| top-5 exact order | 0.98 | 1.00 |
| order among docs scoring > 0.05 | 1.00 (n=37) | 1.00 |
| full 20-doc order | 0.55 | 0.99 |
| max score difference | 0.019 (p99 0.0034) | 0.0016 |
Every decision that matters agrees; the differences are shuffles among near-zero-scoring tail documents. ⚠ A consumer that thresholds on the rerank score could see a borderline document flip (scores move up to ~0.02). An earlier 30-query run with random distractors had one top-1 disagreement (29/30); across both runs, 129/130.
Speed — TEI is NOT faster on this card
On-box, 2 interleaved runs × 3 reps each, medians:
| workload | vLLM | TEI |
|---|---|---|
| embed 1 short query, p50 | ~9.1 ms | ~6.9 ms |
| embed 1 × ~512 tok, p50 | ~23 ms (bimodal 10–25) | ~28 ms |
| embed 1 × ~2k tok, p50 | ~86 ms | ~108 ms |
| bulk embed, passages/s | ~50 | ~37 |
| whole novel (The Stand, 12,814 paragraphs), 64/request, 4 in flight | 31 s | 40 s |
| rerank 20 docs, p50 | ~169 ms | ~180 ms |
| rerank 20 docs, req/s at conc 8 | ~5.7 | ~5.6 |
TEI ran its fused FlashQwen3 path. --max-batch-tokens 32768 did not change
the novel time (39–40 s), so it stays at the default. Through the gateway these
gaps mostly vanish: LiteLLM is the bottleneck there (see
servers/esh-ml1/README.md).
Footprint — TEI is much lighter
| vLLM (both) | TEI (both) | |
|---|---|---|
VRAM (host nvidia-smi, per process) |
3,298 + 1,512 MiB | 1,352 + 1,256 MiB |
| image | 29.9 GB | 8.16 GB |
| warm restart to healthy (n=3) | ~24 s | ~4 s |
| container RAM just after start | 2.5 + 5.1 GiB | 0.7 + 1.7 GiB |
⚠ vLLM's VRAM figure is mostly a setting (--gpu-memory-utilization 0.20
each). A lower setting would shrink it; that has not been tested.
Gateway compatibility (LiteLLM v1.97, temp aliases, deleted after)
- Embeddings: drop-in. TEI's
/v1/embeddingsworks with the existinghosted_vllm/provider. - Rerank: needs the
huggingface/provider (scores identical to direct TEI).hosted_vllm/fails with 422 (missing field texts); TEI's/rerankbody differs from vLLM's.
Behaviour differences to carry into any cut-over
- TEI auto-truncates over-length input by default; vLLM (at
--max-model-len 8192) rejects it with an error. - TEI caps inputs per request at
--max-client-batch-size(default 32; set to 128 here). - TEI on CUDA runs float16 only (vLLM served Qwen3-Embedding in bf16); parity above says it does not matter here.