feat(litellm): load-share embed/rerank across esh-ml1 + nh3-ml1 with intra-group failover
- qwen3-embedding and reranker: second deployment on nh3-ml1 (config); reranker-a3-bge-v2-m3: second DB deployment via /model/new. - router_settings.enable_weighted_failover: true. Only the three multi-deployment groups are affected. - Measured by stopping nh3-ml1 TEI: without failover 7/40 embeds 500'd; with it rerank 80/80, embed 38/40 at onset and 60/60 over a 34 s outage; nh3 rejoins rotation after restart. Split 12/28 embed, 22/18 rerank.
This commit is contained in:
@@ -5,12 +5,11 @@ GPU LXC for the ESH home lab: **CT 110 on esh-pve**, holding the **NVIDIA RTX
|
||||
reranking service** — the only backend behind the gateway's `qwen3-embedding`,
|
||||
`reranker` and `reranker-a3-bge-v2-m3` since 2026-09-25. Built 2026-09-24.
|
||||
|
||||
⚠ **Still the single gateway backend.** The second RTX 2000 is live as
|
||||
[nh3-ml1](../nh3-ml1/README.md) (2026-09-25, parity-verified: the two hosts cannot
|
||||
be told apart), but LiteLLM does not route to it yet; that is Prime's call. Until
|
||||
it does, if esh-ml1, esh-pve, or ESH's mesh route (esh-scale, CT 108) is down,
|
||||
fleet embeddings and reranking are down: Worldtree recall, nevermore clustering,
|
||||
Open WebUI RAG.
|
||||
**Load-shared with [nh3-ml1](../nh3-ml1/README.md) since 2026-09-26 (Prime).** The
|
||||
gateway splits `qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` across both
|
||||
TEI hosts, which are parity-verified as indistinguishable. Losing either host no
|
||||
longer takes fleet embeddings down; the failover test results are in the nh3-ml1
|
||||
README. The reward seat (`/scalar-judge`) is still esh-ml1 only.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
|
||||
Reference in New Issue
Block a user