feat(embed-rerank): TEI is the fleet embed/rerank engine; esh-ml1 sole backend; retire fv-ml1 seats
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet from now on; fv-ml1 retires both once esh-ml1 is up. - stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports 8001/8013, fail-closed truncation (--auto-truncate false; embed --max-batch-tokens 32768). - litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address); reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target. - Verified via the gateway against the retiring fv-ml1 seats: embed cosine median 0.999927 (n=203); rerank top-1/top-3 29/30. - stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged. - Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md; CLAUDE.md gains the TEI convention.
This commit is contained in:
@@ -1,18 +1,40 @@
|
||||
# embed-rerank
|
||||
|
||||
The fleet's embedding + reranking models on **esh-ml1** (CT 110 on esh-pve,
|
||||
RTX 2000E Ada). The second site for `qwen3-embedding` and `reranker`; fv-ml1's
|
||||
[`vllm`](../vllm/) stack is the first.
|
||||
**The fleet's embedding + reranking service**, on **esh-ml1** (CT 110 on esh-pve,
|
||||
RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**.
|
||||
Since 2026-09-25 it is the only backend behind the gateway names
|
||||
`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3`.
|
||||
|
||||
| container | model | port |
|
||||
|---|---|---|
|
||||
| `vllm-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 |
|
||||
| `vllm-rerank-bge` | `BAAI/bge-reranker-v2-m3` | 8013 |
|
||||
**TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or
|
||||
reranking seats go on TEI, not vLLM. Why, and the measurements behind it:
|
||||
[`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`](../../docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md).
|
||||
|
||||
**Keep it in lockstep with `stacks/vllm`:** same models, same `VLLM_VERSION`,
|
||||
same `--max-model-len`. The LiteLLM groups treat the two sites as one model,
|
||||
so any drift in the embedding model or its version silently mixes
|
||||
incompatible vectors into consumers' indexes.
|
||||
| container | model | port | endpoint |
|
||||
|---|---|---|---|
|
||||
| `tei-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 | `/v1/embeddings` (OpenAI), `/embed` |
|
||||
| `tei-rerank` | `BAAI/bge-reranker-v2-m3` | 8013 | `/rerank` — body `{"query", "texts"}` |
|
||||
|
||||
## Gateway wiring (stacks/litellm/conf/config.yaml)
|
||||
|
||||
- `qwen3-embedding` → `hosted_vllm/Qwen/Qwen3-Embedding-0.6B`, `api_base: http://10.0.50.80:8001/v1`.
|
||||
- `reranker` → **`huggingface/`**`BAAI/bge-reranker-v2-m3`, `api_base: http://10.0.50.80:8013` (**no `/v1`**).
|
||||
⚠ `hosted_vllm/` sends `documents` and TEI answers **422 "missing field texts"**.
|
||||
- `reranker-a3-bge-v2-m3` is a DB-only alias (not in config.yaml) with the same target.
|
||||
|
||||
## ⚠ Invariants
|
||||
|
||||
- **Changing `EMBED_MODEL` invalidates every index built on it** (Worldtree,
|
||||
nevermore, Open WebUI). An engine change is allowed only with a parity
|
||||
measurement against the current vectors. The TEI switch measured cosine median
|
||||
0.999925 and old-index retrieval overlap 0.988.
|
||||
- **Truncation is fail-closed** (`--auto-truncate false`). TEI's default silently
|
||||
embeds a prefix of an over-length input and returns 200. Now: embed rejects
|
||||
more than 32,768 tokens and rerank more than 8,192, both with 422.
|
||||
- **The image tag is GPU-generation specific**: `89-` is Ada. A different card
|
||||
needs a different prefix (TEI README image table).
|
||||
- nevermore thresholds rerank scores at 0.3. TEI scores differ from the old vLLM
|
||||
seat by up to 0.019, which produced 0 flips in 2,000 scores. Recheck
|
||||
thresholding consumers after any engine or version change.
|
||||
|
||||
## Deploy
|
||||
|
||||
@@ -30,8 +52,5 @@ Host prerequisites (driver, LXC, docker, toolkit) are in
|
||||
curl -s http://10.0.50.80:8001/v1/embeddings -H 'content-type: application/json' \
|
||||
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"hello"}' | jq '.data[0].embedding | length' # 1024
|
||||
curl -s http://10.0.50.80:8013/rerank -H 'content-type: application/json' \
|
||||
-d '{"model":"BAAI/bge-reranker-v2-m3","query":"cat","documents":["a cat","a car"]}' | jq '.results[0]'
|
||||
-d '{"query":"cat","texts":["a cat","a car"]}' # index 0 scores ~0.9986
|
||||
```
|
||||
|
||||
Parity against fv-ml1 is recorded in the host README; re-measure after any
|
||||
version bump on either side.
|
||||
|
||||
Reference in New Issue
Block a user