vh
5960526c3f
NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled. - nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml. - nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4) deployed with HOST_NAME/HOST_IP labels. - Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min 0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control 0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed identical within rep spread. - gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix #7006). With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl reopen; applied on nh3-pve (one package). - pve-nvidia-host.yaml: document that the headers meta drags in the newest kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot). - Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal. - nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels, AMT cabled but unreachable on the network. Gateway routing to nh3-ml1 is not changed.
63 lines
3.1 KiB
Markdown
63 lines
3.1 KiB
Markdown
# embed-rerank
|
|
|
|
**The fleet's embedding + reranking service**, on **esh-ml1** (CT 110 on esh-pve,
|
|
RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**.
|
|
Since 2026-09-25 it is the only backend behind the gateway names
|
|
`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3`. A second instance
|
|
runs on **nh3-ml1** (CT 109 on nh3-pve, the same card), parity-verified against
|
|
esh-ml1 on 2026-09-25 (`servers/nh3-ml1/README.md`). It is not in the gateway
|
|
yet; that is Prime's call.
|
|
|
|
**TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or
|
|
reranking seats go on TEI, not vLLM. Why, and the measurements behind it:
|
|
[`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`](../../docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md).
|
|
|
|
| container | model | port | endpoint |
|
|
|---|---|---|---|
|
|
| `tei-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 | `/v1/embeddings` (OpenAI), `/embed` |
|
|
| `tei-rerank` | `BAAI/bge-reranker-v2-m3` | 8013 | `/rerank` — body `{"query", "texts"}` |
|
|
|
|
## Gateway wiring (stacks/litellm/conf/config.yaml)
|
|
|
|
- `qwen3-embedding` → `hosted_vllm/Qwen/Qwen3-Embedding-0.6B`, `api_base: http://10.0.50.80:8001/v1`.
|
|
- `reranker` → **`huggingface/`**`BAAI/bge-reranker-v2-m3`, `api_base: http://10.0.50.80:8013` (**no `/v1`**).
|
|
⚠ `hosted_vllm/` sends `documents` and TEI answers **422 "missing field texts"**.
|
|
- `reranker-a3-bge-v2-m3` is a DB-only alias (not in config.yaml) with the same target.
|
|
|
|
## ⚠ Invariants
|
|
|
|
- **Changing `EMBED_MODEL` invalidates every index built on it** (Worldtree,
|
|
nevermore, Open WebUI). An engine change is allowed only with a parity
|
|
measurement against the current vectors. The TEI switch measured cosine median
|
|
0.999925 and old-index retrieval overlap 0.988.
|
|
- **Truncation is fail-closed** (`--auto-truncate false`). TEI's default silently
|
|
embeds a prefix of an over-length input and returns 200. Now: embed rejects
|
|
more than 32,768 tokens and rerank more than 8,192, both with 422.
|
|
- **The image tag is GPU-generation specific**: `89-` is Ada. A different card
|
|
needs a different prefix (TEI README image table).
|
|
- nevermore thresholds rerank scores at 0.3. TEI scores differ from the old vLLM
|
|
seat by up to 0.019, which produced 0 flips in 2,000 scores. Recheck
|
|
thresholding consumers after any engine or version change.
|
|
|
|
## Deploy
|
|
|
|
```bash
|
|
scripts/deploy-stack.sh esh-ml1 embed-rerank
|
|
ssh esh-ml1 'cd /opt/docker/compose/embed-rerank && cp -n .env.example .env && docker compose config -q && docker compose up -d'
|
|
```
|
|
|
|
nh3-ml1 is the same, plus `HOST_NAME=nh3-ml1` and `HOST_IP=10.100.50.80` in its
|
|
`.env` (they only feed the Homepage labels).
|
|
|
|
Host prerequisites (driver, LXC, docker, toolkit) are in
|
|
[`servers/esh-ml1/README.md`](../../servers/esh-ml1/README.md).
|
|
|
|
## Smoke test
|
|
|
|
```bash
|
|
curl -s http://10.0.50.80:8001/v1/embeddings -H 'content-type: application/json' \
|
|
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"hello"}' | jq '.data[0].embedding | length' # 1024
|
|
curl -s http://10.0.50.80:8013/rerank -H 'content-type: application/json' \
|
|
-d '{"query":"cat","texts":["a cat","a car"]}' # index 0 scores ~0.9986
|
|
```
|