feat(embed-rerank): TEI is the fleet embed/rerank engine; esh-ml1 sole backend; retire fv-ml1 seats

Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet
from now on; fv-ml1 retires both once esh-ml1 is up.

- stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports
  8001/8013, fail-closed truncation (--auto-truncate false; embed
  --max-batch-tokens 32768).
- litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address);
  reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's
  `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target.
- Verified via the gateway against the retiring fv-ml1 seats: embed cosine
  median 0.999927 (n=203); rerank top-1/top-3 29/30.
- stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on
  fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged.
- Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md;
  CLAUDE.md gains the TEI convention.
This commit is contained in:
vh
2026-09-25 08:30:53 -07:00
parent 49b4bf0177
commit 7bdac80878
17 changed files with 271 additions and 418 deletions
+8 -18
View File
@@ -1,29 +1,19 @@
# embed-rerank tunables (esh-ml1). Copy to `.env` on the server.
#
# Everything model-shaped here MUST match fv-ml1's stacks/vllm .env: the same
# models, the same vLLM version, the same max-model-len. A drift in the
# embedding model or its version makes this seat's vectors incompatible with
# every index built against fv-ml1's.
# embed-rerank tunables (esh-ml1, TEI). Copy to `.env` on the server.
# PINNED to fv-ml1's version. Bump both sites together.
VLLM_VERSION=v0.24.0
# TEI image tag. `89-` = the Ada Lovelace (sm_89) build; a different GPU
# generation needs a different prefix (see the TEI README's image table).
# PINNED to an exact release.
TEI_TAG=89-1.9.4
# Same host ports as fv-ml1 (container listens on 8000).
# Kept from the vLLM era so the embedding gateway entry did not have to change.
EMBED_PORT=8001
RERANK_PORT=8013
# ⚠ Changing EMBED_MODEL invalidates every index built on it.
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
RERANK_MODEL=BAAI/bge-reranker-v2-m3
# Fractions of the RTX 2000E Ada's 16,380 MiB. fv-ml1 runs 0.03 of a 96 GB
# card (~2.9 GB each); 0.20 here is ~3.2 GB each — the same budget plus a
# little, leaving ~9.5 GB free.
EMBED_GPU_MEM_UTIL=0.20
RERANK_GPU_MEM_UTIL=0.20
EMBED_MAX_MODEL_LEN=8192
RERANK_MAX_MODEL_LEN=8192
MAX_CLIENT_BATCH_SIZE=128
# fv-ml1's seats run with no API key (LiteLLM fronts them); match that.
API_KEY=
# Both models are public; no token needed.
HF_TOKEN=