vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(embed-rerank): TEI is the fleet embed/rerank engine; esh-ml1 sole backend; retire fv-ml1 seats

Browse Source 
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet
from now on; fv-ml1 retires both once esh-ml1 is up.

- stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports
  8001/8013, fail-closed truncation (--auto-truncate false; embed
  --max-batch-tokens 32768).
- litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address);
  reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's
  `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target.
- Verified via the gateway against the retiring fv-ml1 seats: embed cosine
  median 0.999927 (n=203); rerank top-1/top-3 29/30.
- stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on
  fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged.
- Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md;
  CLAUDE.md gains the TEI convention.
This commit is contained in:
vh
2026-09-25 08:30:53 -07:00
parent 49b4bf0177
commit 7bdac80878
17 changed files with 271 additions and 418 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/embed-rerank/.env.example
+8 -18
View File
@@ -1,29 +1,19 @@
# embed-rerank tunables (esh-ml1). Copy to `.env` on the server.
#
# Everything model-shaped here MUST match fv-ml1's stacks/vllm .env: the same
# models, the same vLLM version, the same max-model-len. A drift in the
# embedding model or its version makes this seat's vectors incompatible with
# every index built against fv-ml1's.
# embed-rerank tunables (esh-ml1, TEI). Copy to `.env` on the server.
# PINNED to fv-ml1's version. Bump both sites together.
VLLM_VERSION=v0.24.0
# TEI image tag. `89-` = the Ada Lovelace (sm_89) build; a different GPU
# generation needs a different prefix (see the TEI README's image table).
# PINNED to an exact release.
TEI_TAG=89-1.9.4
# Same host ports as fv-ml1 (container listens on 8000).
# Kept from the vLLM era so the embedding gateway entry did not have to change.
EMBED_PORT=8001
RERANK_PORT=8013
# ⚠ Changing EMBED_MODEL invalidates every index built on it.
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
RERANK_MODEL=BAAI/bge-reranker-v2-m3
# Fractions of the RTX 2000E Ada's 16,380 MiB. fv-ml1 runs 0.03 of a 96 GB
# card (~2.9 GB each); 0.20 here is ~3.2 GB each — the same budget plus a
# little, leaving ~9.5 GB free.
EMBED_GPU_MEM_UTIL=0.20
RERANK_GPU_MEM_UTIL=0.20
EMBED_MAX_MODEL_LEN=8192
RERANK_MAX_MODEL_LEN=8192
MAX_CLIENT_BATCH_SIZE=128
# fv-ml1's seats run with no API key (LiteLLM fronts them); match that.
API_KEY=
# Both models are public; no token needed.
HF_TOKEN=
stacks/embed-rerank/README.md
+34 -15
View File
@@ -1,18 +1,40 @@
# embed-rerank
The fleet's embedding + reranking models on **esh-ml1** (CT 110 on esh-pve,
RTX 2000E Ada). The second site for `qwen3-embedding` and `reranker`; fv-ml1's
[`vllm`](../vllm/) stack is the first.
**The fleet's embedding + reranking service**, on **esh-ml1** (CT 110 on esh-pve,
RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**.
Since 2026-09-25 it is the only backend behind the gateway names
`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3`.
| container | model | port |
|---|---|---|
| `vllm-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 |
| `vllm-rerank-bge` | `BAAI/bge-reranker-v2-m3` | 8013 |
**TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or
reranking seats go on TEI, not vLLM. Why, and the measurements behind it:
[`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`](../../docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md).
**Keep it in lockstep with `stacks/vllm`:** same models, same `VLLM_VERSION`,
same `--max-model-len`. The LiteLLM groups treat the two sites as one model,
so any drift in the embedding model or its version silently mixes
incompatible vectors into consumers' indexes.
| container | model | port | endpoint |
|---|---|---|---|
| `tei-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 | `/v1/embeddings` (OpenAI), `/embed` |
| `tei-rerank` | `BAAI/bge-reranker-v2-m3` | 8013 | `/rerank` — body `{"query", "texts"}` |
## Gateway wiring (stacks/litellm/conf/config.yaml)
- `qwen3-embedding` → `hosted_vllm/Qwen/Qwen3-Embedding-0.6B`, `api_base: http://10.0.50.80:8001/v1`.
- `reranker` → **`huggingface/`**`BAAI/bge-reranker-v2-m3`, `api_base: http://10.0.50.80:8013` (**no `/v1`**).
⚠ `hosted_vllm/` sends `documents` and TEI answers **422 "missing field texts"**.
- `reranker-a3-bge-v2-m3` is a DB-only alias (not in config.yaml) with the same target.
## ⚠ Invariants
- **Changing `EMBED_MODEL` invalidates every index built on it** (Worldtree,
nevermore, Open WebUI). An engine change is allowed only with a parity
measurement against the current vectors. The TEI switch measured cosine median
0.999925 and old-index retrieval overlap 0.988.
- **Truncation is fail-closed** (`--auto-truncate false`). TEI's default silently
embeds a prefix of an over-length input and returns 200. Now: embed rejects
more than 32,768 tokens and rerank more than 8,192, both with 422.
- **The image tag is GPU-generation specific**: `89-` is Ada. A different card
needs a different prefix (TEI README image table).
- nevermore thresholds rerank scores at 0.3. TEI scores differ from the old vLLM
seat by up to 0.019, which produced 0 flips in 2,000 scores. Recheck
thresholding consumers after any engine or version change.
## Deploy
@@ -30,8 +52,5 @@ Host prerequisites (driver, LXC, docker, toolkit) are in
curl -s http://10.0.50.80:8001/v1/embeddings -H 'content-type: application/json' \
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"hello"}' | jq '.data[0].embedding | length' # 1024
curl -s http://10.0.50.80:8013/rerank -H 'content-type: application/json' \
-d '{"model":"BAAI/bge-reranker-v2-m3","query":"cat","documents":["a cat","a car"]}' | jq '.results[0]'
-d '{"query":"cat","texts":["a cat","a car"]}' # index 0 scores ~0.9986
```
Parity against fv-ml1 is recorded in the host README; re-measure after any
version bump on either side.
stacks/embed-rerank/compose.yaml
+66 -73
View File
@@ -1,63 +1,66 @@
# embed-rerank — the fleet's embedding + reranking models, served locally at ESH
# on esh-ml1 (CT 110 on esh-pve, RTX 2000E Ada, 16 GB).
# embed-rerank — THE fleet's embedding + reranking service, on esh-ml1 (CT 110 on
# esh-pve, RTX 2000E Ada, 16 GB). Served by Hugging Face Text Embeddings
# Inference (TEI).
#
# WHY: Prime's decision 2026-09-24 — the Ada card offloads embedding and
# reranking so ESH consumers (Open WebUI RAG, Paperless) are not tied to one
# seat on fv-ml1. It serves the SAME models as fv-ml1's `vllm` stack, with the
# SAME vLLM version and flags, because embedding vectors are model-specific:
# a different embedding model here would silently poison every index built
# against fv-ml1's. Mirror stacks/vllm/compose.yaml when that changes.
# Prime, 2026-09-25: "TEI is embed/reranker server for esh-ml1 and the FLEET in
# general, in future." It replaced vLLM here the same day, after a side-by-side
# bake-off on this card (docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md). TEI
# gives the same vectors as the old vLLM seats (no re-embedding), but it is
# ~1.3x slower on bulk work on this card. It is much lighter (2.6 GB VRAM for
# both, 8 GB image, ~4 s restart). fv-ml1's vLLM embed/rerank seats were
# retired after this went live.
#
# vllm-embed Qwen/Qwen3-Embedding-0.6B → /v1/embeddings :8001
# vllm-rerank-bge BAAI/bge-reranker-v2-m3 → /rerank, /score :8013
# tei-embed Qwen/Qwen3-Embedding-0.6B → /v1/embeddings (OpenAI), /embed :8001
# tei-rerank BAAI/bge-reranker-v2-m3 → /rerank (body: query + texts) :8013
#
# Ports match fv-ml1 on purpose, so a LiteLLM deployment for either site
# differs only in the host part of api_base.
# Ports kept from the vLLM era (and fv-ml1), so the embedding gateway entry did
# not change address. ⚠ The RERANK gateway entry must use LiteLLM's
# `huggingface/` provider: `hosted_vllm/` sends `documents` and TEI answers 422
# "missing field texts".
#
# `vllm-rerank-bge`, not fv-ml1's `vllm-rerank-a3`: that name carries R43
# bake-off provenance for THAT container; and plain `vllm-rerank` was the
# retired Qwen3-Reranker that measured harmful. Name the model instead.
# ⚠ Embedding vectors are model-specific. Never change EMBED_MODEL without a
# re-embedding plan for every index built on it (Worldtree, nevermore, Open WebUI).
#
# NO `tnet`/traefik-net: esh-ml1 runs no traefik, and an external network
# that does not exist would stop the stack from starting. Consumers reach the
# published ports directly.
# FAIL-CLOSED truncation (--auto-truncate false). TEI's default silently
# embedded the first 16,384 tokens of a ~40k-token input and returned 200.
# Turning it off requires --max-batch-tokens >= the model's max input (32,768
# for Qwen3-Embedding), or TEI refuses to start.
#
# Host setup (driver, LXC, docker, toolkit): playbooks/esh-pve-nvidia-host.yaml
# then playbooks/esh-ml1-lxc.yaml. Tunables live in .env — edit that, not this.
# NO `tnet`/traefik-net: esh-ml1 runs no traefik; consumers reach the published
# ports, and in practice only the LiteLLM gateway does (verified from the seats'
# logs 2026-09-25: every request matched a gateway spend-log row).
#
# Host setup: playbooks/esh-pve-nvidia-host.yaml, then playbooks/esh-ml1-lxc.yaml.
# Tunables live in .env.
name: embed-rerank
services:
vllm-embed:
image: vllm/vllm-openai:${VLLM_VERSION}
container_name: vllm-embed
tei-embed:
image: ghcr.io/huggingface/text-embeddings-inference:${TEI_TAG}
container_name: tei-embed
restart: unless-stopped
ipc: host
ports:
- "${EMBED_PORT}:8000"
- "${EMBED_PORT}:80"
volumes:
- /opt/aimodels/huggingface:/hfcache
- /opt/aimodels/tei-cache:/data
environment:
- HF_HOME=/hfcache
- HF_HUB_CACHE=/hfcache/hub
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
- VLLM_API_KEY=${API_KEY:-}
- HF_TOKEN=${HF_TOKEN:-}
command:
- --model-id
- ${EMBED_MODEL}
- --served-model-name
- ${EMBED_MODEL}
- --runner
- pooling
- --host
- 0.0.0.0
- --port
- "8000"
- --gpu-memory-utilization
- ${EMBED_GPU_MEM_UTIL}
- --max-model-len
- ${EMBED_MAX_MODEL_LEN}
# TEI on CUDA is float16-only; parity vs the bf16 vLLM vectors was measured.
- --dtype
- auto
- float16
# Default 32; vLLM had no cap and callers batch 64.
- --max-client-batch-size
- "${MAX_CLIENT_BATCH_SIZE}"
- --auto-truncate
- "false"
- --max-batch-tokens
- "32768"
deploy:
resources:
reservations:
@@ -66,49 +69,39 @@ services:
device_ids: ["0"]
capabilities: [gpu]
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
test: ["CMD", "curl", "-fsS", "http://localhost:80/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 180s
start_period: 60s
labels:
- homepage.group=AI - Eval & Retrieval
- homepage.name=vLLM Embed (esh-ml1)
- homepage.name=Embed — Qwen3 0.6B (TEI, esh-ml1)
- homepage.icon=mdi-vector-arrange-below
- homepage.description=Qwen3 Embedding 0.6B via vLLM (esh-ml1, RTX 2000E Ada)
- homepage.description=Fleet embeddings (qwen3-embedding) via TEI on esh-ml1
- homepage.href=http://10.0.50.80:${EMBED_PORT}/docs
vllm-rerank-bge:
image: vllm/vllm-openai:${VLLM_VERSION}
container_name: vllm-rerank-bge
tei-rerank:
image: ghcr.io/huggingface/text-embeddings-inference:${TEI_TAG}
container_name: tei-rerank
restart: unless-stopped
ipc: host
ports:
- "${RERANK_PORT}:8000"
- "${RERANK_PORT}:80"
volumes:
- /opt/aimodels/huggingface:/hfcache
- /opt/aimodels/tei-cache:/data
environment:
- HF_HOME=/hfcache
- HF_HUB_CACHE=/hfcache/hub
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
- VLLM_API_KEY=${API_KEY:-}
- HF_TOKEN=${HF_TOKEN:-}
command:
- --model-id
- ${RERANK_MODEL}
- --served-model-name
- ${RERANK_MODEL}
- --runner
- pooling
# bge-reranker-v2-m3 is natively a cross-encoder; no --hf-overrides.
- --host
- 0.0.0.0
- --port
- "8000"
- --gpu-memory-utilization
- ${RERANK_GPU_MEM_UTIL}
- --max-model-len
- ${RERANK_MAX_MODEL_LEN}
- --dtype
- auto
- float16
- --max-client-batch-size
- "${MAX_CLIENT_BATCH_SIZE}"
# Fail-closed; bge-reranker-v2-m3's max input (8,192) fits the default
# max-batch-tokens (16,384).
- --auto-truncate
- "false"
deploy:
resources:
reservations:
@@ -117,14 +110,14 @@ services:
device_ids: ["0"]
capabilities: [gpu]
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
test: ["CMD", "curl", "-fsS", "http://localhost:80/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 180s
start_period: 60s
labels:
- homepage.group=AI - Eval & Retrieval
- homepage.name=vLLM Rerank bge-v2-m3 (esh-ml1)
- homepage.name=Rerank — bge-v2-m3 (TEI, esh-ml1)
- homepage.icon=mdi-sort-variant
- homepage.description=bge-reranker-v2-m3 via vLLM (esh-ml1, RTX 2000E Ada)
- homepage.description=Fleet reranker (reranker) via TEI on esh-ml1
- homepage.href=http://10.0.50.80:${RERANK_PORT}/docs