feat(litellm): load-share embed/rerank across esh-ml1 + nh3-ml1 with intra-group failover
- qwen3-embedding and reranker: second deployment on nh3-ml1 (config); reranker-a3-bge-v2-m3: second DB deployment via /model/new. - router_settings.enable_weighted_failover: true. Only the three multi-deployment groups are affected. - Measured by stopping nh3-ml1 TEI: without failover 7/40 embeds 500'd; with it rerank 80/80, embed 38/40 at onset and 60/60 over a 34 s outage; nh3 rejoins rotation after restart. Split 12/28 embed, 22/18 rerank.
This commit is contained in:
@@ -2,11 +2,11 @@
|
||||
|
||||
**The fleet's embedding + reranking service**, on **esh-ml1** (CT 110 on esh-pve,
|
||||
RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**.
|
||||
Since 2026-09-25 it is the only backend behind the gateway names
|
||||
`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3`. A second instance
|
||||
runs on **nh3-ml1** (CT 109 on nh3-pve, the same card), parity-verified against
|
||||
esh-ml1 on 2026-09-25 (`servers/nh3-ml1/README.md`). It is not in the gateway
|
||||
yet; that is Prime's call.
|
||||
It runs on **two hosts**, esh-ml1 and **nh3-ml1** (CT 109 on nh3-pve, the same
|
||||
card). The two are parity-verified as indistinguishable, and since 2026-09-26
|
||||
(Prime) they are **load-shared** behind the gateway names `qwen3-embedding`,
|
||||
`reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover
|
||||
(`servers/nh3-ml1/README.md` has the measurements).
|
||||
|
||||
**TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or
|
||||
reranking seats go on TEI, not vLLM. Why, and the measurements behind it:
|
||||
|
||||
@@ -443,15 +443,14 @@ model_list:
|
||||
# explicitly. Operator ruling 2026-08-23. ---
|
||||
|
||||
# --- Qwen3 embeddings ---
|
||||
# ONE backend: esh-ml1 (RTX 2000E Ada), served by Hugging Face TEI — the fleet's
|
||||
# embed/rerank engine from 2026-09-25 (Prime). fv-ml1's vLLM seat was retired the
|
||||
# same day. TEI's /v1/embeddings is OpenAI-compatible, so the provider stays
|
||||
# `hosted_vllm/`. Same model as before; TEI vectors match the old vLLM ones
|
||||
# (cosine median 0.999925, inside vLLM's own noise; old-index retrieval overlap
|
||||
# 0.988 vs 0.986 self-noise), so no index needed re-embedding.
|
||||
# ⚠ Single backend until the second RTX 2000 lands (Prime, 2026-09-25): if
|
||||
# esh-ml1, esh-pve or the ESH mesh route is down, fleet embeddings are down.
|
||||
# ⚠ Never point this name at a DIFFERENT embedding model — vectors are
|
||||
# TWO backends, load-shared (simple-shuffle), since 2026-09-26 (Prime): esh-ml1
|
||||
# (ESH) and nh3-ml1 (NH3). Both are an RTX 2000E Ada running Hugging Face TEI
|
||||
# 89-1.9.4 with the same model. Parity measured 2026-09-25: cosine esh vs nh3
|
||||
# min 0.999993 = each host's own floor; overlap@10 1.000
|
||||
# (servers/nh3-ml1/README.md). TEI's /v1/embeddings is OpenAI-compatible, so the
|
||||
# provider is `hosted_vllm/`. TEI vectors match the retired vLLM seats (cosine
|
||||
# median 0.999925), so no index needed re-embedding.
|
||||
# ⚠ Never point this name at a DIFFERENT embedding model: vectors are
|
||||
# model-specific. Engine bake-off: docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md.
|
||||
- model_name: qwen3-embedding
|
||||
litellm_params:
|
||||
@@ -460,6 +459,13 @@ model_list:
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
model_info:
|
||||
mode: embedding
|
||||
- model_name: qwen3-embedding
|
||||
litellm_params:
|
||||
model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
|
||||
api_base: http://10.100.50.80:8001/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
model_info:
|
||||
mode: embedding
|
||||
|
||||
# --- qwen3-reranker RETIRED 2026-08-20. It named Qwen3-Reranker-0.6B on :8002,
|
||||
# the incumbent the R43 bake-off replaced on 2026-08-06 after measuring it
|
||||
@@ -498,12 +504,13 @@ model_list:
|
||||
model_info:
|
||||
mode: chat
|
||||
# reranker → generic capability name for rerank (bge-reranker-v2-m3 since the
|
||||
# R43 cutover). ONE backend: TEI on esh-ml1 (fv-ml1's vLLM seat retired
|
||||
# 2026-09-25). ⚠ TEI's /rerank takes `texts`, not `documents`: this MUST use the
|
||||
# R43 cutover). TWO backends, load-shared, since 2026-09-26 (Prime): TEI on
|
||||
# esh-ml1 and nh3-ml1. esh vs nh3 max score diff 0.0014, inside esh's own
|
||||
# 0.0020 run-to-run floor; top-1 identical. The DB-only alias
|
||||
# `reranker-a3-bge-v2-m3` carries the same two deployments.
|
||||
# ⚠ TEI's /rerank takes `texts`, not `documents`: this MUST use the
|
||||
# `huggingface/` provider. `hosted_vllm/` gets a 422 "missing field texts".
|
||||
# api_base has NO /v1. api_key is a placeholder; TEI runs without auth.
|
||||
# Parity vs the retired vLLM seat: top-1/top-3 identical (129/130 top-1 over
|
||||
# two runs); scores move up to 0.019, reshuffling only near-zero tail docs.
|
||||
- model_name: reranker
|
||||
litellm_params:
|
||||
model: huggingface/BAAI/bge-reranker-v2-m3
|
||||
@@ -511,6 +518,13 @@ model_list:
|
||||
api_key: "none"
|
||||
model_info:
|
||||
mode: rerank
|
||||
- model_name: reranker
|
||||
litellm_params:
|
||||
model: huggingface/BAAI/bge-reranker-v2-m3
|
||||
api_base: http://10.100.50.80:8013
|
||||
api_key: "none"
|
||||
model_info:
|
||||
mode: rerank
|
||||
|
||||
# --- lfm25-vl-3b → LiquidAI LFM2.5-VL-3B (Q5_K_M + mmproj Q8_0), llama.cpp server on
|
||||
# nh3-ml1 :8030 (stacks/lfm-vl-seat, 2026-09-26). Image understanding for
|
||||
@@ -1077,6 +1091,18 @@ general_settings:
|
||||
forward_headers: true
|
||||
include_subpath: true
|
||||
|
||||
router_settings:
|
||||
# Intra-group failover for simple-shuffle. When one deployment fails, the SAME
|
||||
# request re-picks among the group's other healthy deployments, excluding the
|
||||
# failed one. Without it, a dead backend in a 2-deployment group kept failing
|
||||
# a share of requests: the router's default 2 retries re-picked at random and
|
||||
# hit the dead one again. Measured 2026-09-26 with nh3-ml1 TEI stopped: 7/40
|
||||
# embeds returned 500.
|
||||
# Affects ONLY multi-deployment groups (qwen3-embedding, reranker,
|
||||
# reranker-a3-bge-v2-m3 as of 2026-09-26). A single-deployment group has no
|
||||
# other deployment to re-pick, so its behaviour is unchanged.
|
||||
enable_weighted_failover: true
|
||||
|
||||
litellm_settings:
|
||||
# vLLM rejects some OpenAI params other backends accept; drop silently
|
||||
# rather than 400 the caller.
|
||||
|
||||
@@ -110,11 +110,11 @@ monitors:
|
||||
|
||||
- name: Embed — Qwen3 0.6B (TEI, nh3-ml1)
|
||||
url: http://10.100.50.80:8001/health
|
||||
description: second `qwen3-embedding` backend (nh3-ml1)
|
||||
description: gateway `qwen3-embedding` backend (nh3-ml1)
|
||||
|
||||
- name: Rerank — bge-v2-m3 (TEI, nh3-ml1)
|
||||
url: http://10.100.50.80:8013/health
|
||||
description: second `reranker` backend (nh3-ml1)
|
||||
description: gateway `reranker` backend (nh3-ml1)
|
||||
|
||||
- name: talk
|
||||
url: https://talk.nh3.phasefinal.com:8092/
|
||||
|
||||
Reference in New Issue
Block a user