feat(litellm): load-share embed/rerank across esh-ml1 + nh3-ml1 with intra-group failover
- qwen3-embedding and reranker: second deployment on nh3-ml1 (config); reranker-a3-bge-v2-m3: second DB deployment via /model/new. - router_settings.enable_weighted_failover: true. Only the three multi-deployment groups are affected. - Measured by stopping nh3-ml1 TEI: without failover 7/40 embeds 500'd; with it rerank 80/80, embed 38/40 at onset and 60/60 over a 34 s outage; nh3 rejoins rotation after restart. Split 12/28 embed, 22/18 rerank.
This commit is contained in:
@@ -2,11 +2,11 @@
|
||||
|
||||
**The fleet's embedding + reranking service**, on **esh-ml1** (CT 110 on esh-pve,
|
||||
RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**.
|
||||
Since 2026-09-25 it is the only backend behind the gateway names
|
||||
`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3`. A second instance
|
||||
runs on **nh3-ml1** (CT 109 on nh3-pve, the same card), parity-verified against
|
||||
esh-ml1 on 2026-09-25 (`servers/nh3-ml1/README.md`). It is not in the gateway
|
||||
yet; that is Prime's call.
|
||||
It runs on **two hosts**, esh-ml1 and **nh3-ml1** (CT 109 on nh3-pve, the same
|
||||
card). The two are parity-verified as indistinguishable, and since 2026-09-26
|
||||
(Prime) they are **load-shared** behind the gateway names `qwen3-embedding`,
|
||||
`reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover
|
||||
(`servers/nh3-ml1/README.md` has the measurements).
|
||||
|
||||
**TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or
|
||||
reranking seats go on TEI, not vLLM. Why, and the measurements behind it:
|
||||
|
||||
Reference in New Issue
Block a user