# Embed/rerank engine bake-off — TEI vs vLLM on esh-ml1 (2026-09-25)

Prime, 2026-09-25: *"run the TEI bake-off."* Hugging Face **Text Embeddings
Inference 1.9.4** (`89-1.9.4`, the Ada build) served the same two models as the
then-vLLM `embed-rerank` stack (vLLM v0.24.0), side by side on the same RTX 2000E
Ada, as a temporary `tei-bakeoff` stack.

**Outcome (Prime, 2026-09-25): "TEI is embed/reranker server for esh-ml1 and the
FLEET in general, in future."** The same day TEI replaced vLLM in
[`stacks/embed-rerank`](../../stacks/embed-rerank/) on ports 8001/8013. The
gateway was repointed with esh-ml1 as the sole backend, and fv-ml1's
`vllm-embed` + `vllm-rerank-a3` seats were retired, freeing ~6.1 GB on fv-ml1
GPU 1. The bake-off stack was removed; this file is its record.

## Results (2026-09-25 0744–0805 PT)

### Parity — TEI matches the existing vLLM vectors

Reference = vLLM on fv-ml1, the engine every existing index was built with.
306 texts (6 fixed incl. CJK/code/6k-char + 300 *The Stand* paragraphs);
retrieval = 2,000-paragraph corpus, 50 instruction-format queries.

| embedding check | result | noise floor / control |
|---|---|---|
| cosine TEI vs vLLM-FV | median 0.999925, min 0.999861 | vLLM-FV vs itself: 0.999916 / 0.999796 |
| overlap@10, TEI index + TEI queries | 0.976 | vLLM-FV rerun 0.986; vLLM-ESH 0.972 |
| overlap@10, **TEI queries vs the OLD vLLM index** (migration case) | **0.988** | positive control (MRL-256 dims) 0.648 |
| hit@1 own paragraph | 0.80 | vLLM-FV 0.80 |
| different-text negative control | cosine median 0.31 | — |

TEI is deterministic (TEI vs itself: min 0.999995). **Switching engines does not
require re-embedding existing indexes.**

| rerank check (100 queries × 20 docs, 9 same-chapter distractors) | TEI vs vLLM-FV | vLLM-FV vs itself |
|---|---|---|
| source paragraph ranked #1 | 0.95 (same as vLLM) | 0.95 |
| top-1 agreement | 1.00 | 1.00 |
| top-3 set agreement | 1.00 | 1.00 |
| top-5 exact order | 0.98 | 1.00 |
| order among docs scoring > 0.05 | 1.00 (n=37) | 1.00 |
| full 20-doc order | **0.55** | 0.99 |
| max score difference | **0.019** (p99 0.0034) | 0.0016 |

Every decision that matters agrees; the differences are shuffles among
near-zero-scoring tail documents. ⚠ A consumer that **thresholds** on the rerank
score could see a borderline document flip (scores move up to ~0.02). An
earlier 30-query run with random distractors had one top-1 disagreement (29/30);
across both runs, 129/130.

### Speed — TEI is NOT faster on this card

On-box, 2 interleaved runs × 3 reps each, medians:

| workload | vLLM | TEI |
|---|---|---|
| embed 1 short query, p50 | ~9.1 ms | **~6.9 ms** |
| embed 1 × ~512 tok, p50 | ~23 ms (bimodal 10–25) | ~28 ms |
| embed 1 × ~2k tok, p50 | **~86 ms** | ~108 ms |
| bulk embed, passages/s | **~50** | ~37 |
| whole novel (*The Stand*, 12,814 paragraphs), 64/request, 4 in flight | **31 s** | 40 s |
| rerank 20 docs, p50 | **~169 ms** | ~180 ms |
| rerank 20 docs, req/s at conc 8 | ~5.7 | ~5.6 |

TEI ran its fused `FlashQwen3` path. `--max-batch-tokens 32768` did not change
the novel time (39–40 s), so it stays at the default. **Through the gateway these
gaps mostly vanish**: LiteLLM is the bottleneck there (see
[`servers/esh-ml1/README.md`](../../servers/esh-ml1/README.md)).

### Footprint — TEI is much lighter

| | vLLM (both) | TEI (both) |
|---|---|---|
| VRAM (host `nvidia-smi`, per process) | 3,298 + 1,512 MiB | 1,352 + 1,256 MiB |
| image | 29.9 GB | 8.16 GB |
| warm restart to healthy (n=3) | ~24 s | ~4 s |
| container RAM just after start | 2.5 + 5.1 GiB | 0.7 + 1.7 GiB |

⚠ vLLM's VRAM figure is mostly a **setting** (`--gpu-memory-utilization 0.20`
each). A lower setting would shrink it; that has not been tested.

### External review — Dvalin (Grok research peer), 2026-09-25

**Agree with caveats: adopt TEI for these two seats; footprint is the right
reason.** His external evidence (not re-verified here): a Runpod 2026-09-14
engine comparison shows vLLM ahead of TEI on Qwen3-Embedding bulk throughput
(median ~2.6×, single unreplicated runs, near parity on smaller cards) and TEI
ahead on BERT-family models — consistent with our 1.3× on a 50 W card and the
rerank near-tie. Open TEI 1.9 issue #857 (tokio panic "No backend receiver"
under load, n=1) is a watch item, not a gate. A lower vLLM memory setting
would shrink only the embedder (the reranker at 1,512 MiB is already under
its cap); 0.12 is the only setting he'd expect to both boot and help — untested.
His caveats, and status: fail-closed truncation (**done**, above); recheck score-
threshold consumers (**done**, nevermore, above); two gateway providers must be
written down at cut-over (open).

## Gateway compatibility (LiteLLM v1.97, temp aliases, deleted after)

- Embeddings: **drop-in**. TEI's `/v1/embeddings` works with the existing
  `hosted_vllm/` provider.
- Rerank: needs the **`huggingface/` provider** (scores identical to direct
  TEI). `hosted_vllm/` fails with 422 (`missing field texts`); TEI's `/rerank`
  body differs from vLLM's.

## Behaviour differences to carry into any cut-over

- **Truncation — now fail-closed (`--auto-truncate false`).** Measured with
  TEI's default: a ~40k-token input returned **200 with a vector of its first
  16,384 tokens**, silently, where vLLM returns 400. With truncation off, TEI
  embed rejects > 32,768 tokens and TEI rerank rejects > 8,192 (both 422,
  verified). Truncation off requires `--max-batch-tokens` at or above the model's max
  input (32,768 for Qwen3-Embedding) or TEI refuses to start; VRAM unchanged.
  Remaining difference: TEI embed **accepts** 8,193–32,767 tokens that vLLM
  (`--max-model-len 8192`) rejects. It embeds the whole input, which is wider,
  not lossy.
- **nevermore thresholds rerank scores** (`NEVERMORE_RERANK_THRESHOLD`, default
  0.3, cluster-member filter in `digest.py`). Across the 2,000 bake-off scores,
  **0 flips** at 0.3 or 0.5 for TEI vs vLLM — but only 1 score landed within
  ±0.02 of 0.3 (bge scores sit near 0 or 1), so this cannot exclude rare flips
  on borderline headlines. Worst case: a same-story headline clusters
  differently. Low stakes.
- TEI caps inputs per request at `--max-client-batch-size` (default 32; set to 128 here).
- TEI on CUDA runs float16 only (vLLM served Qwen3-Embedding in bf16); parity
  above says it does not matter here.