vh
7bdac80878
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet from now on; fv-ml1 retires both once esh-ml1 is up. - stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports 8001/8013, fail-closed truncation (--auto-truncate false; embed --max-batch-tokens 32768). - litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address); reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target. - Verified via the gateway against the retiring fv-ml1 seats: embed cosine median 0.999927 (n=203); rerank top-1/top-3 29/30. - stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged. - Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md; CLAUDE.md gains the TEI convention.
124 lines
6.3 KiB
Markdown
124 lines
6.3 KiB
Markdown
# Embed/rerank engine bake-off — TEI vs vLLM on esh-ml1 (2026-09-25)
|
||
|
||
Prime, 2026-09-25: *"run the TEI bake-off."* Hugging Face **Text Embeddings
|
||
Inference 1.9.4** (`89-1.9.4`, the Ada build) served the same two models as the
|
||
then-vLLM `embed-rerank` stack (vLLM v0.24.0), side by side on the same RTX 2000E
|
||
Ada, as a temporary `tei-bakeoff` stack.
|
||
|
||
**Outcome (Prime, 2026-09-25): "TEI is embed/reranker server for esh-ml1 and the
|
||
FLEET in general, in future."** The same day TEI replaced vLLM in
|
||
[`stacks/embed-rerank`](../../stacks/embed-rerank/) on ports 8001/8013. The
|
||
gateway was repointed with esh-ml1 as the sole backend, and fv-ml1's
|
||
`vllm-embed` + `vllm-rerank-a3` seats were retired, freeing ~6.1 GB on fv-ml1
|
||
GPU 1. The bake-off stack was removed; this file is its record.
|
||
|
||
## Results (2026-09-25 0744–0805 PT)
|
||
|
||
### Parity — TEI matches the existing vLLM vectors
|
||
|
||
Reference = vLLM on fv-ml1, the engine every existing index was built with.
|
||
306 texts (6 fixed incl. CJK/code/6k-char + 300 *The Stand* paragraphs);
|
||
retrieval = 2,000-paragraph corpus, 50 instruction-format queries.
|
||
|
||
| embedding check | result | noise floor / control |
|
||
|---|---|---|
|
||
| cosine TEI vs vLLM-FV | median 0.999925, min 0.999861 | vLLM-FV vs itself: 0.999916 / 0.999796 |
|
||
| overlap@10, TEI index + TEI queries | 0.976 | vLLM-FV rerun 0.986; vLLM-ESH 0.972 |
|
||
| overlap@10, **TEI queries vs the OLD vLLM index** (migration case) | **0.988** | positive control (MRL-256 dims) 0.648 |
|
||
| hit@1 own paragraph | 0.80 | vLLM-FV 0.80 |
|
||
| different-text negative control | cosine median 0.31 | — |
|
||
|
||
TEI is deterministic (TEI vs itself: min 0.999995). **Switching engines does not
|
||
require re-embedding existing indexes.**
|
||
|
||
| rerank check (100 queries × 20 docs, 9 same-chapter distractors) | TEI vs vLLM-FV | vLLM-FV vs itself |
|
||
|---|---|---|
|
||
| source paragraph ranked #1 | 0.95 (same as vLLM) | 0.95 |
|
||
| top-1 agreement | 1.00 | 1.00 |
|
||
| top-3 set agreement | 1.00 | 1.00 |
|
||
| top-5 exact order | 0.98 | 1.00 |
|
||
| order among docs scoring > 0.05 | 1.00 (n=37) | 1.00 |
|
||
| full 20-doc order | **0.55** | 0.99 |
|
||
| max score difference | **0.019** (p99 0.0034) | 0.0016 |
|
||
|
||
Every decision that matters agrees; the differences are shuffles among
|
||
near-zero-scoring tail documents. ⚠ A consumer that **thresholds** on the rerank
|
||
score could see a borderline document flip (scores move up to ~0.02). An
|
||
earlier 30-query run with random distractors had one top-1 disagreement (29/30);
|
||
across both runs, 129/130.
|
||
|
||
### Speed — TEI is NOT faster on this card
|
||
|
||
On-box, 2 interleaved runs × 3 reps each, medians:
|
||
|
||
| workload | vLLM | TEI |
|
||
|---|---|---|
|
||
| embed 1 short query, p50 | ~9.1 ms | **~6.9 ms** |
|
||
| embed 1 × ~512 tok, p50 | ~23 ms (bimodal 10–25) | ~28 ms |
|
||
| embed 1 × ~2k tok, p50 | **~86 ms** | ~108 ms |
|
||
| bulk embed, passages/s | **~50** | ~37 |
|
||
| whole novel (*The Stand*, 12,814 paragraphs), 64/request, 4 in flight | **31 s** | 40 s |
|
||
| rerank 20 docs, p50 | **~169 ms** | ~180 ms |
|
||
| rerank 20 docs, req/s at conc 8 | ~5.7 | ~5.6 |
|
||
|
||
TEI ran its fused `FlashQwen3` path. `--max-batch-tokens 32768` did not change
|
||
the novel time (39–40 s), so it stays at the default. **Through the gateway these
|
||
gaps mostly vanish**: LiteLLM is the bottleneck there (see
|
||
[`servers/esh-ml1/README.md`](../../servers/esh-ml1/README.md)).
|
||
|
||
### Footprint — TEI is much lighter
|
||
|
||
| | vLLM (both) | TEI (both) |
|
||
|---|---|---|
|
||
| VRAM (host `nvidia-smi`, per process) | 3,298 + 1,512 MiB | 1,352 + 1,256 MiB |
|
||
| image | 29.9 GB | 8.16 GB |
|
||
| warm restart to healthy (n=3) | ~24 s | ~4 s |
|
||
| container RAM just after start | 2.5 + 5.1 GiB | 0.7 + 1.7 GiB |
|
||
|
||
⚠ vLLM's VRAM figure is mostly a **setting** (`--gpu-memory-utilization 0.20`
|
||
each). A lower setting would shrink it; that has not been tested.
|
||
|
||
### External review — Dvalin (Grok research peer), 2026-09-25
|
||
|
||
**Agree with caveats: adopt TEI for these two seats; footprint is the right
|
||
reason.** His external evidence (not re-verified here): a Runpod 2026-09-14
|
||
engine comparison shows vLLM ahead of TEI on Qwen3-Embedding bulk throughput
|
||
(median ~2.6×, single unreplicated runs, near parity on smaller cards) and TEI
|
||
ahead on BERT-family models — consistent with our 1.3× on a 50 W card and the
|
||
rerank near-tie. Open TEI 1.9 issue #857 (tokio panic "No backend receiver"
|
||
under load, n=1) is a watch item, not a gate. A lower vLLM memory setting
|
||
would shrink only the embedder (the reranker at 1,512 MiB is already under
|
||
its cap); 0.12 is the only setting he'd expect to both boot and help — untested.
|
||
His caveats, and status: fail-closed truncation (**done**, above); recheck score-
|
||
threshold consumers (**done**, nevermore, above); two gateway providers must be
|
||
written down at cut-over (open).
|
||
|
||
## Gateway compatibility (LiteLLM v1.97, temp aliases, deleted after)
|
||
|
||
- Embeddings: **drop-in**. TEI's `/v1/embeddings` works with the existing
|
||
`hosted_vllm/` provider.
|
||
- Rerank: needs the **`huggingface/` provider** (scores identical to direct
|
||
TEI). `hosted_vllm/` fails with 422 (`missing field texts`); TEI's `/rerank`
|
||
body differs from vLLM's.
|
||
|
||
## Behaviour differences to carry into any cut-over
|
||
|
||
- **Truncation — now fail-closed (`--auto-truncate false`).** Measured with
|
||
TEI's default: a ~40k-token input returned **200 with a vector of its first
|
||
16,384 tokens**, silently, where vLLM returns 400. With truncation off, TEI
|
||
embed rejects > 32,768 tokens and TEI rerank rejects > 8,192 (both 422,
|
||
verified). Truncation off requires `--max-batch-tokens` at or above the model's max
|
||
input (32,768 for Qwen3-Embedding) or TEI refuses to start; VRAM unchanged.
|
||
Remaining difference: TEI embed **accepts** 8,193–32,767 tokens that vLLM
|
||
(`--max-model-len 8192`) rejects. It embeds the whole input, which is wider,
|
||
not lossy.
|
||
- **nevermore thresholds rerank scores** (`NEVERMORE_RERANK_THRESHOLD`, default
|
||
0.3, cluster-member filter in `digest.py`). Across the 2,000 bake-off scores,
|
||
**0 flips** at 0.3 or 0.5 for TEI vs vLLM — but only 1 score landed within
|
||
±0.02 of 0.3 (bge scores sit near 0 or 1), so this cannot exclude rare flips
|
||
on borderline headlines. Worst case: a same-story headline clusters
|
||
differently. Low stakes.
|
||
- TEI caps inputs per request at `--max-client-batch-size` (default 32; set to 128 here).
|
||
- TEI on CUDA runs float16 only (vLLM served Qwen3-Embedding in bf16); parity
|
||
above says it does not matter here.
|