diff --git a/persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md b/persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md index 26ade76..3d75470 100644 --- a/persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md +++ b/persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md @@ -67,3 +67,14 @@ faster because it is network-closer). Bulk: ESH 3–10× slower (bulk embed ~50 ~403 passages/s; rerank-20 ~170 vs ~51 ms p50, ~5.7 vs ~55 req/s at conc 8). Noise floor 3–5% on throughput. ⇒ failover role confirmed; load-sharing would roughly double rerank latency for half the calls. + +**TEI bake-off (2026-09-25, `stacks/tei-bakeoff/README.md`).** Prime asked to +evaluate a purpose-built engine instead of vLLM → HF Text Embeddings Inference +1.9.4 (Infinity ruled out: last release 2025-08-22, dropped by us in May for no +Qwen3). Verdict: TEI **matches** (embed cosine vs vLLM-FV median 0.999925, inside +noise; TEI queries against the OLD vLLM index overlap@10 0.988 vs 0.986 self-noise +→ no re-embed needed; rerank decisions identical, tail scores move ≤0.019) but is +**not faster** on the Ada (novel 40 s vs 31 s; tiny queries ~25% faster). It is +**much lighter**: ~2.6 GB VRAM for both vs vLLM's 4.8 GB budget, 8 GB image vs 30, +~4 s restart vs ~24 s. Gateway: embed drop-in via `hosted_vllm/`; rerank needs the +`huggingface/` provider. Both engines left running side by side pending Prime's call. diff --git a/persistent-memory.md b/persistent-memory.md index 4ca4fa7..f5b4573 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -117,6 +117,15 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-09-24 ~2245 PT._ +### ⭐ esh-ml1: TEI vs vLLM decision pending (Prime), then "what else moves" + +Prime's direction 2026-09-25: move the utility seats off fv-ml1 to esh-ml1 +(accepting slower); **parakeet STAYS on fv-ml1** (real-time chat path). TEI +bake-off done — results `stacks/tei-bakeoff/README.md`; tei-embed :8081 + +tei-rerank :8083 running beside vLLM, NOT gateway-wired. Next: Prime picks the +engine, then the rest of the move (reward model is the open candidate; it needs +vLLM — TEI can't serve a Llama classifier). + ### ✅ esh-ml1 built (2026-09-24) — two follow-ups The RTX 2000E Ada now serves `qwen3-embedding` + `reranker` from **CT 110 diff --git a/stacks/tei-bakeoff/README.md b/stacks/tei-bakeoff/README.md new file mode 100644 index 0000000..8aba1db --- /dev/null +++ b/stacks/tei-bakeoff/README.md @@ -0,0 +1,89 @@ +# tei-bakeoff — TEI vs vLLM for embedding + reranking on esh-ml1 + +Prime, 2026-09-25: *"run the TEI bake-off."* Hugging Face **Text Embeddings +Inference 1.9.4** (`89-1.9.4`, the Ada build) serving the same two models as +[`embed-rerank`](../embed-rerank/) (vLLM v0.24.0), side by side on the same RTX +2000E Ada. Not wired into the gateway. This stack either becomes the +embed-rerank stack or is torn down. + +## Results (2026-09-25 0744–0805 PT) + +### Parity — TEI matches the existing vLLM vectors + +Reference = vLLM on fv-ml1, the engine every existing index was built with. +306 texts (6 fixed incl. CJK/code/6k-char + 300 *The Stand* paragraphs); +retrieval = 2,000-paragraph corpus, 50 instruction-format queries. + +| embedding check | result | noise floor / control | +|---|---|---| +| cosine TEI vs vLLM-FV | median 0.999925, min 0.999861 | vLLM-FV vs itself: 0.999916 / 0.999796 | +| overlap@10, TEI index + TEI queries | 0.976 | vLLM-FV rerun 0.986; vLLM-ESH 0.972 | +| overlap@10, **TEI queries vs the OLD vLLM index** (migration case) | **0.988** | positive control (MRL-256 dims) 0.648 | +| hit@1 own paragraph | 0.80 | vLLM-FV 0.80 | +| different-text negative control | cosine median 0.31 | — | + +TEI is deterministic (TEI vs itself: min 0.999995). **Switching engines does not +require re-embedding existing indexes.** + +| rerank check (100 queries × 20 docs, 9 same-chapter distractors) | TEI vs vLLM-FV | vLLM-FV vs itself | +|---|---|---| +| source paragraph ranked #1 | 0.95 (same as vLLM) | 0.95 | +| top-1 agreement | 1.00 | 1.00 | +| top-3 set agreement | 1.00 | 1.00 | +| top-5 exact order | 0.98 | 1.00 | +| order among docs scoring > 0.05 | 1.00 (n=37) | 1.00 | +| full 20-doc order | **0.55** | 0.99 | +| max score difference | **0.019** (p99 0.0034) | 0.0016 | + +Every decision that matters agrees; the differences are shuffles among +near-zero-scoring tail documents. ⚠ A consumer that **thresholds** on the rerank +score could see a borderline document flip (scores move up to ~0.02). An +earlier 30-query run with random distractors had one top-1 disagreement (29/30); +across both runs, 129/130. + +### Speed — TEI is NOT faster on this card + +On-box, 2 interleaved runs × 3 reps each, medians: + +| workload | vLLM | TEI | +|---|---|---| +| embed 1 short query, p50 | ~9.1 ms | **~6.9 ms** | +| embed 1 × ~512 tok, p50 | ~23 ms (bimodal 10–25) | ~28 ms | +| embed 1 × ~2k tok, p50 | **~86 ms** | ~108 ms | +| bulk embed, passages/s | **~50** | ~37 | +| whole novel (*The Stand*, 12,814 paragraphs), 64/request, 4 in flight | **31 s** | 40 s | +| rerank 20 docs, p50 | **~169 ms** | ~180 ms | +| rerank 20 docs, req/s at conc 8 | ~5.7 | ~5.6 | + +TEI ran its fused `FlashQwen3` path. `--max-batch-tokens 32768` did not change +the novel time (39–40 s), so it stays at the default. **Through the gateway these +gaps mostly vanish**: LiteLLM is the bottleneck there (see +`servers/esh-ml1/README.md`). + +### Footprint — TEI is much lighter + +| | vLLM (both) | TEI (both) | +|---|---|---| +| VRAM (host `nvidia-smi`, per process) | 3,298 + 1,512 MiB | 1,352 + 1,256 MiB | +| image | 29.9 GB | 8.16 GB | +| warm restart to healthy (n=3) | ~24 s | ~4 s | +| container RAM just after start | 2.5 + 5.1 GiB | 0.7 + 1.7 GiB | + +⚠ vLLM's VRAM figure is mostly a **setting** (`--gpu-memory-utilization 0.20` +each). A lower setting would shrink it; that has not been tested. + +### Gateway compatibility (LiteLLM v1.97, temp aliases, deleted after) + +- Embeddings: **drop-in**. TEI's `/v1/embeddings` works with the existing + `hosted_vllm/` provider. +- Rerank: needs the **`huggingface/` provider** (scores identical to direct + TEI). `hosted_vllm/` fails with 422 (`missing field texts`); TEI's `/rerank` + body differs from vLLM's. + +## Behaviour differences to carry into any cut-over + +- TEI **auto-truncates** over-length input by default; vLLM (at + `--max-model-len 8192`) rejects it with an error. +- TEI caps inputs per request at `--max-client-batch-size` (default 32; set to 128 here). +- TEI on CUDA runs float16 only (vLLM served Qwen3-Embedding in bf16); parity + above says it does not matter here. diff --git a/stacks/tei-bakeoff/compose.yaml b/stacks/tei-bakeoff/compose.yaml new file mode 100644 index 0000000..5a37a2a --- /dev/null +++ b/stacks/tei-bakeoff/compose.yaml @@ -0,0 +1,69 @@ +# tei-bakeoff — Hugging Face Text Embeddings Inference (TEI) serving the SAME two +# models as stacks/embed-rerank (vLLM), side by side on esh-ml1, so the two +# engines can be compared on identical hardware. Prime, 2026-09-25: "run the +# TEI bake-off". Not wired into the gateway. Temporary: either this becomes the +# embed-rerank stack or it is torn down. +# +# tei-embed Qwen/Qwen3-Embedding-0.6B → /v1/embeddings, /embed :8081 +# tei-rerank BAAI/bge-reranker-v2-m3 → /rerank :8083 +# +# 89-* = the Ada Lovelace (sm_89) build. Pinned to an exact release. +# TEI on CUDA offers float16/float32 only (no bfloat16), while vLLM serves +# Qwen3-Embedding in bf16 — the parity measurement is what decides whether +# that matters. +# (--max-batch-tokens 32768 was tried on tei-embed: whole-novel time unchanged, +# 39.2–40.4 s vs 39.9–40.3 s, so it stays at the default.) +# --max-client-batch-size 128: the default is 32; vLLM has no such cap and the +# bake-off sends 64 per request. +# Separate HF cache from vLLM's, so neither engine can disturb the other's files. + +name: tei-bakeoff + +services: + tei-embed: + image: ghcr.io/huggingface/text-embeddings-inference:89-1.9.4 + container_name: tei-embed + restart: unless-stopped + ports: + - "8081:80" + volumes: + - /opt/aimodels/tei-cache:/data + command: + - --model-id + - Qwen/Qwen3-Embedding-0.6B + - --served-model-name + - Qwen/Qwen3-Embedding-0.6B + - --dtype + - float16 + - --max-client-batch-size + - "128" + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["0"] + capabilities: [gpu] + + tei-rerank: + image: ghcr.io/huggingface/text-embeddings-inference:89-1.9.4 + container_name: tei-rerank + restart: unless-stopped + ports: + - "8083:80" + volumes: + - /opt/aimodels/tei-cache:/data + command: + - --model-id + - BAAI/bge-reranker-v2-m3 + - --dtype + - float16 + - --max-client-batch-size + - "128" + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["0"] + capabilities: [gpu]