feat(litellm): load-share embed/rerank across esh-ml1 + nh3-ml1 with intra-group failover
- qwen3-embedding and reranker: second deployment on nh3-ml1 (config); reranker-a3-bge-v2-m3: second DB deployment via /model/new. - router_settings.enable_weighted_failover: true. Only the three multi-deployment groups are affected. - Measured by stopping nh3-ml1 TEI: without failover 7/40 embeds 500'd; with it rerank 80/80, embed 38/40 at onset and 60/60 over a 34 s outage; nh3 rejoins rotation after restart. Split 12/28 embed, 22/18 rerank.
This commit is contained in:
@@ -110,11 +110,11 @@ monitors:
|
||||
|
||||
- name: Embed — Qwen3 0.6B (TEI, nh3-ml1)
|
||||
url: http://10.100.50.80:8001/health
|
||||
description: second `qwen3-embedding` backend (nh3-ml1)
|
||||
description: gateway `qwen3-embedding` backend (nh3-ml1)
|
||||
|
||||
- name: Rerank — bge-v2-m3 (TEI, nh3-ml1)
|
||||
url: http://10.100.50.80:8013/health
|
||||
description: second `reranker` backend (nh3-ml1)
|
||||
description: gateway `reranker` backend (nh3-ml1)
|
||||
|
||||
- name: talk
|
||||
url: https://talk.nh3.phasefinal.com:8092/
|
||||
|
||||
Reference in New Issue
Block a user