feat(embed-rerank): TEI is the fleet embed/rerank engine; esh-ml1 sole backend; retire fv-ml1 seats
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet from now on; fv-ml1 retires both once esh-ml1 is up. - stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports 8001/8013, fail-closed truncation (--auto-truncate false; embed --max-batch-tokens 32768). - litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address); reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target. - Verified via the gateway against the retiring fv-ml1 seats: embed cosine median 0.999927 (n=203); rerank top-1/top-3 29/30. - stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged. - Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md; CLAUDE.md gains the TEI convention.
This commit is contained in:
@@ -12,23 +12,15 @@
|
||||
VLLM_VERSION=v0.24.0
|
||||
|
||||
# Host ports (container always listens on 8000 internally)
|
||||
EMBED_PORT=8001
|
||||
# 8013 — the reranker moved here 2026-08-20 when bge-v2-m3 (the R43 winner, which
|
||||
# had been running as a throwaway `docker run` on this port) was promoted into
|
||||
# this stack and the Qwen incumbent on :8002 was retired.
|
||||
RERANK_PORT=8013
|
||||
# (EMBED_PORT 8001 / RERANK_PORT 8013 retired 2026-09-25 with their seats —
|
||||
# embedding + reranking moved to TEI on esh-ml1, stacks/embed-rerank.)
|
||||
REWARD_PORT=8003
|
||||
|
||||
# GPU assignment — all services share this GPU
|
||||
# (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work in llama-swap)
|
||||
GPU_ID=1
|
||||
|
||||
# Models — reference by full repo name in API requests
|
||||
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
|
||||
# bge-reranker-v2-m3 — the R43 bake-off winner, replacing Qwen3-Reranker-0.6B
|
||||
# (measured HARMING 80/90 fleet queries). Multilingual cross-encoder; needs no
|
||||
# --hf-overrides, unlike the Qwen reranker it displaced.
|
||||
RERANK_MODEL=BAAI/bge-reranker-v2-m3
|
||||
# Models
|
||||
# Skywork is a local-path AWQ output, not from HF Hub. Bind-mounted into the
|
||||
# reward container at /local-models — see compose.yaml. No env var here for
|
||||
# the model path itself since it's hard-coded in the compose command.
|
||||
@@ -62,16 +54,12 @@ RERANK_MODEL=BAAI/bge-reranker-v2-m3
|
||||
# each at 0.05 (mostly util-reservation waste); 0.03 (~3.6 GB) fits weights +
|
||||
# CUDA context with room, freeing ~4 GB back to granite. Recreate them ONE AT A
|
||||
# TIME — concurrent recreate races the memory-profiling assertion.
|
||||
EMBED_GPU_MEM_UTIL=0.03
|
||||
RERANK_GPU_MEM_UTIL=0.03
|
||||
REWARD_GPU_MEM_UTIL=0.10
|
||||
|
||||
# Context length caps — lower these if VRAM is tight.
|
||||
# Qwen3-Embedding supports up to 32k; reranker up to 32k.
|
||||
# Skywork capped at 16k server-side as defense-in-depth; JudgeClient also
|
||||
# enforces the cap at dispatch time per spec.
|
||||
EMBED_MAX_MODEL_LEN=8192
|
||||
RERANK_MAX_MODEL_LEN=8192
|
||||
REWARD_MAX_MODEL_LEN=16384
|
||||
|
||||
# Optional API key — leave blank for no auth (fine on the internal network).
|
||||
|
||||
Reference in New Issue
Block a user