# embed-rerank

**The fleet's embedding + reranking service**, on **esh-ml1** (CT 110 on esh-pve,
RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**.
It runs on **two hosts**, esh-ml1 and **nh3-ml1** (CT 109 on nh3-pve, the same
card). The two are parity-verified as indistinguishable, and since 2026-09-26
(Prime) they are **load-shared** behind the gateway names `qwen3-embedding`,
`reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover
(`servers/nh3-ml1/README.md` has the measurements).

**TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or
reranking seats go on TEI, not vLLM. Why, and the measurements behind it:
[`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`](../../docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md).

| container | model | port | endpoint |
|---|---|---|---|
| `tei-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 | `/v1/embeddings` (OpenAI), `/embed` |
| `tei-rerank` | `BAAI/bge-reranker-v2-m3` | 8013 | `/rerank` — body `{"query", "texts"}` |

## Gateway wiring (stacks/litellm/conf/config.yaml)

- `qwen3-embedding` → `hosted_vllm/Qwen/Qwen3-Embedding-0.6B`, `api_base: http://10.0.50.80:8001/v1`.
- `reranker` → **`huggingface/`**`BAAI/bge-reranker-v2-m3`, `api_base: http://10.0.50.80:8013` (**no `/v1`**).
  ⚠ `hosted_vllm/` sends `documents` and TEI answers **422 "missing field texts"**.
- `reranker-a3-bge-v2-m3` is a DB-only alias (not in config.yaml) with the same target.

## ⚠ Invariants

- **Changing `EMBED_MODEL` invalidates every index built on it** (Worldtree,
  nevermore, Open WebUI). An engine change is allowed only with a parity
  measurement against the current vectors. The TEI switch measured cosine median
  0.999925 and old-index retrieval overlap 0.988.
- **Truncation is fail-closed** (`--auto-truncate false`). TEI's default silently
  embeds a prefix of an over-length input and returns 200. Now: embed rejects
  more than 32,768 tokens and rerank more than 8,192, both with 422.
- **The image tag is GPU-generation specific**: `89-` is Ada. A different card
  needs a different prefix (TEI README image table).
- nevermore thresholds rerank scores at 0.3. TEI scores differ from the old vLLM
  seat by up to 0.019, which produced 0 flips in 2,000 scores. Recheck
  thresholding consumers after any engine or version change.

## Deploy

```bash
scripts/deploy-stack.sh esh-ml1 embed-rerank
ssh esh-ml1 'cd /opt/docker/compose/embed-rerank && cp -n .env.example .env && docker compose config -q && docker compose up -d'
```

nh3-ml1 is the same, plus `HOST_NAME=nh3-ml1` and `HOST_IP=10.100.50.80` in its
`.env` (they only feed the Homepage labels).

Host prerequisites (driver, LXC, docker, toolkit) are in
[`servers/esh-ml1/README.md`](../../servers/esh-ml1/README.md).

## Smoke test

```bash
curl -s http://10.0.50.80:8001/v1/embeddings -H 'content-type: application/json' \
  -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"hello"}' | jq '.data[0].embedding | length'   # 1024
curl -s http://10.0.50.80:8013/rerank -H 'content-type: application/json' \
  -d '{"query":"cat","texts":["a cat","a car"]}'                      # index 0 scores ~0.9986
```