feat(embed-rerank): TEI is the fleet embed/rerank engine; esh-ml1 sole backend; retire fv-ml1 seats

Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet from now on; fv-ml1 retires both once esh-ml1 is up. - stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports 8001/8013, fail-closed truncation (--auto-truncate false; embed --max-batch-tokens 32768). - litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address); reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target. - Verified via the gateway against the retiring fv-ml1 seats: embed cosine median 0.999927 (n=203); rerank top-1/top-3 29/30. - stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged. - Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md; CLAUDE.md gains the TEI convention.
2026-09-25 08:30:53 -07:00
parent 49b4bf0177
commit 7bdac80878
17 changed files with 271 additions and 418 deletions
@@ -33,8 +33,8 @@ on someone else's tab" matters, get a project-scoped key.
 `summarizer` / `summarizer-large` / `classifier` — the always-available local seat for
 summarization, naming/labeling, classification and triage. Essentially free compute;
 reach for it before spending premium API tokens on work a small local model handles.
-Embeddings and rerankers (`qwen3-embedding`, `qwen3-reranker`, `reranker`) are local
-and free too. Anything named `*-frontier`, `glm-*` or `kimi-*` is **PAID**.
+Embeddings and reranking (`qwen3-embedding`, `reranker`) are local and free too —
+served by TEI on esh-ml1 since 2026-09-25 (`qwen3-reranker` was retired 2026-08-20). Anything named `*-frontier`, `glm-*` or `kimi-*` is **PAID**.

 ## Traps

@@ -0,0 +1,123 @@
+# Embed/rerank engine bake-off — TEI vs vLLM on esh-ml1 (2026-09-25)
+
+Prime, 2026-09-25: *"run the TEI bake-off."* Hugging Face **Text Embeddings
+Inference 1.9.4** (`89-1.9.4`, the Ada build) served the same two models as the
+then-vLLM `embed-rerank` stack (vLLM v0.24.0), side by side on the same RTX 2000E
+Ada, as a temporary `tei-bakeoff` stack.
+
+**Outcome (Prime, 2026-09-25): "TEI is embed/reranker server for esh-ml1 and the
+FLEET in general, in future."** The same day TEI replaced vLLM in
+[`stacks/embed-rerank`](../../stacks/embed-rerank/) on ports 8001/8013. The
+gateway was repointed with esh-ml1 as the sole backend, and fv-ml1's
+`vllm-embed` + `vllm-rerank-a3` seats were retired, freeing ~6.1 GB on fv-ml1
+GPU 1. The bake-off stack was removed; this file is its record.
+
+## Results (2026-09-25 0744–0805 PT)
+
+### Parity — TEI matches the existing vLLM vectors
+
+Reference = vLLM on fv-ml1, the engine every existing index was built with.
+306 texts (6 fixed incl. CJK/code/6k-char + 300 *The Stand* paragraphs);
+retrieval = 2,000-paragraph corpus, 50 instruction-format queries.
+
+| embedding check | result | noise floor / control |
+|---|---|---|
+| cosine TEI vs vLLM-FV | median 0.999925, min 0.999861 | vLLM-FV vs itself: 0.999916 / 0.999796 |
+| overlap@10, TEI index + TEI queries | 0.976 | vLLM-FV rerun 0.986; vLLM-ESH 0.972 |
+| overlap@10, **TEI queries vs the OLD vLLM index** (migration case) | **0.988** | positive control (MRL-256 dims) 0.648 |
+| hit@1 own paragraph | 0.80 | vLLM-FV 0.80 |
+| different-text negative control | cosine median 0.31 | — |
+
+TEI is deterministic (TEI vs itself: min 0.999995). **Switching engines does not
+require re-embedding existing indexes.**
+
+| rerank check (100 queries × 20 docs, 9 same-chapter distractors) | TEI vs vLLM-FV | vLLM-FV vs itself |
+|---|---|---|
+| source paragraph ranked #1 | 0.95 (same as vLLM) | 0.95 |
+| top-1 agreement | 1.00 | 1.00 |
+| top-3 set agreement | 1.00 | 1.00 |
+| top-5 exact order | 0.98 | 1.00 |
+| order among docs scoring > 0.05 | 1.00 (n=37) | 1.00 |
+| full 20-doc order | **0.55** | 0.99 |
+| max score difference | **0.019** (p99 0.0034) | 0.0016 |
+
+Every decision that matters agrees; the differences are shuffles among
+near-zero-scoring tail documents. ⚠ A consumer that **thresholds** on the rerank
+score could see a borderline document flip (scores move up to ~0.02). An
+earlier 30-query run with random distractors had one top-1 disagreement (29/30);
+across both runs, 129/130.
+
+### Speed — TEI is NOT faster on this card
+
+On-box, 2 interleaved runs × 3 reps each, medians:
+
+| workload | vLLM | TEI |
+|---|---|---|
+| embed 1 short query, p50 | ~9.1 ms | **~6.9 ms** |
+| embed 1 × ~512 tok, p50 | ~23 ms (bimodal 10–25) | ~28 ms |
+| embed 1 × ~2k tok, p50 | **~86 ms** | ~108 ms |
+| bulk embed, passages/s | **~50** | ~37 |
+| whole novel (*The Stand*, 12,814 paragraphs), 64/request, 4 in flight | **31 s** | 40 s |
+| rerank 20 docs, p50 | **~169 ms** | ~180 ms |
+| rerank 20 docs, req/s at conc 8 | ~5.7 | ~5.6 |
+
+TEI ran its fused `FlashQwen3` path. `--max-batch-tokens 32768` did not change
+the novel time (39–40 s), so it stays at the default. **Through the gateway these
+gaps mostly vanish**: LiteLLM is the bottleneck there (see
+[`servers/esh-ml1/README.md`](../../servers/esh-ml1/README.md)).
+
+### Footprint — TEI is much lighter
+
+| | vLLM (both) | TEI (both) |
+|---|---|---|
+| VRAM (host `nvidia-smi`, per process) | 3,298 + 1,512 MiB | 1,352 + 1,256 MiB |
+| image | 29.9 GB | 8.16 GB |
+| warm restart to healthy (n=3) | ~24 s | ~4 s |
+| container RAM just after start | 2.5 + 5.1 GiB | 0.7 + 1.7 GiB |
+
+⚠ vLLM's VRAM figure is mostly a **setting** (`--gpu-memory-utilization 0.20`
+each). A lower setting would shrink it; that has not been tested.
+
+### External review — Dvalin (Grok research peer), 2026-09-25
+
+**Agree with caveats: adopt TEI for these two seats; footprint is the right
+reason.** His external evidence (not re-verified here): a Runpod 2026-09-14
+engine comparison shows vLLM ahead of TEI on Qwen3-Embedding bulk throughput
+(median ~2.6×, single unreplicated runs, near parity on smaller cards) and TEI
+ahead on BERT-family models — consistent with our 1.3× on a 50 W card and the
+rerank near-tie. Open TEI 1.9 issue #857 (tokio panic "No backend receiver"
+under load, n=1) is a watch item, not a gate. A lower vLLM memory setting
+would shrink only the embedder (the reranker at 1,512 MiB is already under
+its cap); 0.12 is the only setting he'd expect to both boot and help — untested.
+His caveats, and status: fail-closed truncation (**done**, above); recheck score-
+threshold consumers (**done**, nevermore, above); two gateway providers must be
+written down at cut-over (open).
+
+## Gateway compatibility (LiteLLM v1.97, temp aliases, deleted after)
+
+- Embeddings: **drop-in**. TEI's `/v1/embeddings` works with the existing
+  `hosted_vllm/` provider.
+- Rerank: needs the **`huggingface/` provider** (scores identical to direct
+  TEI). `hosted_vllm/` fails with 422 (`missing field texts`); TEI's `/rerank`
+  body differs from vLLM's.
+
+## Behaviour differences to carry into any cut-over
+
+- **Truncation — now fail-closed (`--auto-truncate false`).** Measured with
+  TEI's default: a ~40k-token input returned **200 with a vector of its first
+  16,384 tokens**, silently, where vLLM returns 400. With truncation off, TEI
+  embed rejects > 32,768 tokens and TEI rerank rejects > 8,192 (both 422,
+  verified). Truncation off requires `--max-batch-tokens` at or above the model's max
+  input (32,768 for Qwen3-Embedding) or TEI refuses to start; VRAM unchanged.
+  Remaining difference: TEI embed **accepts** 8,193–32,767 tokens that vLLM
+  (`--max-model-len 8192`) rejects. It embeds the whole input, which is wider,
+  not lossy.
+- **nevermore thresholds rerank scores** (`NEVERMORE_RERANK_THRESHOLD`, default
+  0.3, cluster-member filter in `digest.py`). Across the 2,000 bake-off scores,
+  **0 flips** at 0.3 or 0.5 for TEI vs vLLM — but only 1 score landed within
+  ±0.02 of 0.3 (bge scores sit near 0 or 1), so this cannot exclude rare flips
+  on borderline headlines. Worst case: a same-story headline clusters
+  differently. Low stakes.
+- TEI caps inputs per request at `--max-client-batch-size` (default 32; set to 128 here).
+- TEI on CUDA runs float16 only (vLLM served Qwen3-Embedding in bf16); parity
+  above says it does not matter here.
@@ -124,8 +124,6 @@ context.
 | `gen-small-reasoning` | 8026 |
 | `image-judge` | 8022 |
 | `qwen-image-bench` | 8022 |
-| `qwen3-embedding` | 8001 |
-| `reranker` | 8013 |
 | `sec` | 8025 |
 | `sec-reasoning` | 8025 |
 | `summarizer` | 8026 |
@@ -138,3 +136,7 @@ context.
 *Regenerate with `scripts/seat-inventory.py` after ANY seat change —
 model swap, quant change, context or utilization edit, or speculative-decoding
 change. Run `--check` in CI to catch a stale document.*
+
+> **2026-09-25:** `qwen3-embedding` (:8001) and `reranker` (:8013) LEFT fv-ml1 —
+> their vLLM seats were retired and the names now route to TEI on esh-ml1
+> (`stacks/embed-rerank`). ~6.1 GB freed on GPU 1.
@@ -243,3 +243,12 @@ on fv-ml1; `/model/delete` the two aliases (DB-persisted); weights retained in H
  form, not load-balancing: fv-ml1 serves every request while it is up. Parity: |score|
  FV-vs-ESH max 0.000145 vs FV-vs-FV floor 0.000181, identical ranking over 11 documents.
  See `servers/esh-ml1/README.md`.
+
+## 2026-09-25 — the reranker moved to TEI on esh-ml1 (Prime)
+
+- TEI adopted as the fleet embed/rerank engine after a bake-off
+  (`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`). `vllm-rerank-a3` on fv-ml1 was
+  **retired**; `reranker` and the DB alias `reranker-a3-bge-v2-m3` now both route to
+  `tei-rerank` on esh-ml1 :8013 via LiteLLM's `huggingface/` provider. Same model
+  (bge-reranker-v2-m3): top-1/top-3 identical to the retired seat, and scores move
+  up to 0.019 in the near-zero tail.