feat(litellm): load-share embed/rerank across esh-ml1 + nh3-ml1 with intra-group failover

- qwen3-embedding and reranker: second deployment on nh3-ml1 (config);
  reranker-a3-bge-v2-m3: second DB deployment via /model/new.
- router_settings.enable_weighted_failover: true. Only the three
  multi-deployment groups are affected.
- Measured by stopping nh3-ml1 TEI: without failover 7/40 embeds 500'd;
  with it rerank 80/80, embed 38/40 at onset and 60/60 over a 34 s outage;
  nh3 rejoins rotation after restart. Split 12/28 embed, 22/18 rerank.
This commit is contained in:
vh
2026-09-26 00:54:23 -07:00
parent f2792183d4
commit 50c85e0e8a
6 changed files with 73 additions and 33 deletions
+5 -6
View File
@@ -5,12 +5,11 @@ GPU LXC for the ESH home lab: **CT 110 on esh-pve**, holding the **NVIDIA RTX
reranking service** — the only backend behind the gateway's `qwen3-embedding`,
`reranker` and `reranker-a3-bge-v2-m3` since 2026-09-25. Built 2026-09-24.
⚠ **Still the single gateway backend.** The second RTX 2000 is live as
[nh3-ml1](../nh3-ml1/README.md) (2026-09-25, parity-verified: the two hosts cannot
be told apart), but LiteLLM does not route to it yet; that is Prime's call. Until
it does, if esh-ml1, esh-pve, or ESH's mesh route (esh-scale, CT 108) is down,
fleet embeddings and reranking are down: Worldtree recall, nevermore clustering,
Open WebUI RAG.
**Load-shared with [nh3-ml1](../nh3-ml1/README.md) since 2026-09-26 (Prime).** The
gateway splits `qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` across both
TEI hosts, which are parity-verified as indistinguishable. Losing either host no
longer takes fleet embeddings down; the failover test results are in the nh3-ml1
README. The reward seat (`/scalar-judge`) is still esh-ml1 only.
| | |
|---|---|