feat(litellm): load-share embed/rerank across esh-ml1 + nh3-ml1 with intra-group failover

- qwen3-embedding and reranker: second deployment on nh3-ml1 (config);
  reranker-a3-bge-v2-m3: second DB deployment via /model/new.
- router_settings.enable_weighted_failover: true. Only the three
  multi-deployment groups are affected.
- Measured by stopping nh3-ml1 TEI: without failover 7/40 embeds 500'd;
  with it rerank 80/80, embed 38/40 at onset and 60/60 over a 34 s outage;
  nh3 rejoins rotation after restart. Split 12/28 embed, 22/18 rerank.
This commit is contained in:
vh
2026-09-26 00:54:23 -07:00
parent f2792183d4
commit 50c85e0e8a
6 changed files with 73 additions and 33 deletions
+5 -5
View File
@@ -2,11 +2,11 @@
**The fleet's embedding + reranking service**, on **esh-ml1** (CT 110 on esh-pve,
RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**.
Since 2026-09-25 it is the only backend behind the gateway names
`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3`. A second instance
runs on **nh3-ml1** (CT 109 on nh3-pve, the same card), parity-verified against
esh-ml1 on 2026-09-25 (`servers/nh3-ml1/README.md`). It is not in the gateway
yet; that is Prime's call.
It runs on **two hosts**, esh-ml1 and **nh3-ml1** (CT 109 on nh3-pve, the same
card). The two are parity-verified as indistinguishable, and since 2026-09-26
(Prime) they are **load-shared** behind the gateway names `qwen3-embedding`,
`reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover
(`servers/nh3-ml1/README.md` has the measurements).
**TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or
reranking seats go on TEI, not vLLM. Why, and the measurements behind it: