feat(litellm): load-share embed/rerank across esh-ml1 + nh3-ml1 with intra-group failover

- qwen3-embedding and reranker: second deployment on nh3-ml1 (config);
  reranker-a3-bge-v2-m3: second DB deployment via /model/new.
- router_settings.enable_weighted_failover: true. Only the three
  multi-deployment groups are affected.
- Measured by stopping nh3-ml1 TEI: without failover 7/40 embeds 500'd;
  with it rerank 80/80, embed 38/40 at onset and 60/60 over a 34 s outage;
  nh3 rejoins rotation after restart. Split 12/28 embed, 22/18 rerank.
This commit is contained in:
vh
2026-09-26 00:54:23 -07:00
parent f2792183d4
commit 50c85e0e8a
6 changed files with 73 additions and 33 deletions
+2 -2
View File
@@ -110,11 +110,11 @@ monitors:
- name: Embed — Qwen3 0.6B (TEI, nh3-ml1)
url: http://10.100.50.80:8001/health
description: second `qwen3-embedding` backend (nh3-ml1)
description: gateway `qwen3-embedding` backend (nh3-ml1)
- name: Rerank — bge-v2-m3 (TEI, nh3-ml1)
url: http://10.100.50.80:8013/health
description: second `reranker` backend (nh3-ml1)
description: gateway `reranker` backend (nh3-ml1)
- name: talk
url: https://talk.nh3.phasefinal.com:8092/