esh-pfi-infrastructure/docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md

Embed/rerank engine bake-off — TEI vs vLLM on esh-ml1 (2026-09-25)

Prime, 2026-09-25: "run the TEI bake-off." Hugging Face Text Embeddings Inference 1.9.4 (89-1.9.4, the Ada build) served the same two models as the then-vLLM embed-rerank stack (vLLM v0.24.0), side by side on the same RTX 2000E Ada, as a temporary tei-bakeoff stack.

Outcome (Prime, 2026-09-25): "TEI is embed/reranker server for esh-ml1 and the FLEET in general, in future." The same day TEI replaced vLLM in stacks/embed-rerank on ports 8001/8013. The gateway was repointed with esh-ml1 as the sole backend, and fv-ml1's vllm-embed + vllm-rerank-a3 seats were retired, freeing ~6.1 GB on fv-ml1 GPU 1. The bake-off stack was removed; this file is its record.

Results (2026-09-25 0744–0805 PT)

Parity — TEI matches the existing vLLM vectors

Reference = vLLM on fv-ml1, the engine every existing index was built with. 306 texts (6 fixed incl. CJK/code/6k-char + 300 The Stand paragraphs); retrieval = 2,000-paragraph corpus, 50 instruction-format queries.

embedding check	result	noise floor / control
cosine TEI vs vLLM-FV	median 0.999925, min 0.999861	vLLM-FV vs itself: 0.999916 / 0.999796
overlap@10, TEI index + TEI queries	0.976	vLLM-FV rerun 0.986; vLLM-ESH 0.972
overlap@10, TEI queries vs the OLD vLLM index (migration case)	0.988	positive control (MRL-256 dims) 0.648
hit@1 own paragraph	0.80	vLLM-FV 0.80
different-text negative control	cosine median 0.31	—

TEI is deterministic (TEI vs itself: min 0.999995). Switching engines does not require re-embedding existing indexes.

rerank check (100 queries × 20 docs, 9 same-chapter distractors)	TEI vs vLLM-FV	vLLM-FV vs itself
source paragraph ranked #1	0.95 (same as vLLM)	0.95
top-1 agreement	1.00	1.00
top-3 set agreement	1.00	1.00
top-5 exact order	0.98	1.00
order among docs scoring > 0.05	1.00 (n=37)	1.00
full 20-doc order	0.55	0.99
max score difference	0.019 (p99 0.0034)	0.0016

Every decision that matters agrees; the differences are shuffles among near-zero-scoring tail documents. ⚠ A consumer that thresholds on the rerank score could see a borderline document flip (scores move up to ~0.02). An earlier 30-query run with random distractors had one top-1 disagreement (29/30); across both runs, 129/130.

Speed — TEI is NOT faster on this card

On-box, 2 interleaved runs × 3 reps each, medians:

workload	vLLM	TEI
embed 1 short query, p50	~9.1 ms	~6.9 ms
embed 1 × ~512 tok, p50	~23 ms (bimodal 10–25)	~28 ms
embed 1 × ~2k tok, p50	~86 ms	~108 ms
bulk embed, passages/s	~50	~37
whole novel (The Stand, 12,814 paragraphs), 64/request, 4 in flight	31 s	40 s
rerank 20 docs, p50	~169 ms	~180 ms
rerank 20 docs, req/s at conc 8	~5.7	~5.6

TEI ran its fused FlashQwen3 path. --max-batch-tokens 32768 did not change the novel time (39–40 s), so it stays at the default. Through the gateway these gaps mostly vanish: LiteLLM is the bottleneck there (see servers/esh-ml1/README.md).

Footprint — TEI is much lighter

	vLLM (both)	TEI (both)
VRAM (host nvidia-smi, per process)	3,298 + 1,512 MiB	1,352 + 1,256 MiB
image	29.9 GB	8.16 GB
warm restart to healthy (n=3)	~24 s	~4 s
container RAM just after start	2.5 + 5.1 GiB	0.7 + 1.7 GiB

⚠ vLLM's VRAM figure is mostly a setting (--gpu-memory-utilization 0.20 each). A lower setting would shrink it; that has not been tested.

External review — Dvalin (Grok research peer), 2026-09-25

Agree with caveats: adopt TEI for these two seats; footprint is the right reason. His external evidence (not re-verified here): a Runpod 2026-09-14 engine comparison shows vLLM ahead of TEI on Qwen3-Embedding bulk throughput (median ~2.6×, single unreplicated runs, near parity on smaller cards) and TEI ahead on BERT-family models — consistent with our 1.3× on a 50 W card and the rerank near-tie. Open TEI 1.9 issue #857 (tokio panic "No backend receiver" under load, n=1) is a watch item, not a gate. A lower vLLM memory setting would shrink only the embedder (the reranker at 1,512 MiB is already under its cap); 0.12 is the only setting he'd expect to both boot and help — untested. His caveats, and status: fail-closed truncation (done, above); recheck score- threshold consumers (done, nevermore, above); two gateway providers must be written down at cut-over (open).

Gateway compatibility (LiteLLM v1.97, temp aliases, deleted after)

• Embeddings: drop-in. TEI's /v1/embeddings works with the existing hosted_vllm/ provider.
• Rerank: needs the huggingface/ provider (scores identical to direct TEI). hosted_vllm/ fails with 422 (missing field texts); TEI's /rerank body differs from vLLM's.

Behaviour differences to carry into any cut-over

• Truncation — now fail-closed (--auto-truncate false). Measured with TEI's default: a ~40k-token input returned 200 with a vector of its first 16,384 tokens, silently, where vLLM returns 400. With truncation off, TEI embed rejects > 32,768 tokens and TEI rerank rejects > 8,192 (both 422, verified). Truncation off requires --max-batch-tokens at or above the model's max input (32,768 for Qwen3-Embedding) or TEI refuses to start; VRAM unchanged. Remaining difference: TEI embed accepts 8,193–32,767 tokens that vLLM (--max-model-len 8192) rejects. It embeds the whole input, which is wider, not lossy.
• nevermore thresholds rerank scores (NEVERMORE_RERANK_THRESHOLD, default 0.3, cluster-member filter in digest.py). Across the 2,000 bake-off scores, 0 flips at 0.3 or 0.5 for TEI vs vLLM — but only 1 score landed within ±0.02 of 0.3 (bge scores sit near 0 or 1), so this cannot exclude rare flips on borderline headlines. Worst case: a same-story headline clusters differently. Low stakes.
• TEI caps inputs per request at --max-client-batch-size (default 32; set to 128 here).
• TEI on CUDA runs float16 only (vLLM served Qwen3-Embedding in bf16); parity above says it does not matter here.