feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1
Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics read at source in v1.97.0): lowest order among healthy deployments, and a failed request retries at the next order. Verified after the restart (43 s gateway outage): 6/6 calls per alias landed on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6 from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1. The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does not (403).
This commit is contained in:
@@ -6,7 +6,13 @@ reranking backend**, the twin of [esh-ml1](../esh-ml1/README.md): same card, sam
|
||||
driver, same TEI image and models. Built 2026-09-25, after the NH3 site visit
|
||||
turned Secure Boot off on nh3-pve.
|
||||
|
||||
**Load-shared with esh-ml1 since 2026-09-26 (Prime).** The gateway splits
|
||||
**Embedding routing since 2026-10-05 0030 (Prime): primary + failover, no longer shuffled.** `qwen3-embedding`
|
||||
(fleet default) = esh-ml1 order 1, nh3-ml1 order 2; `qwen3-embedding-bulk` (albok-service's write path) = nh3-ml1 order 1,
|
||||
esh-ml1 order 2. Measured 0730: 6/6 of each alias on its primary; with nh3-ml1's `tei-embed` stopped, bulk went 6/6 to esh-ml1
|
||||
with 200s and fleet stayed on esh-ml1, then bulk returned 6/6 to nh3-ml1 once it was back. (esh-ml1-down → fleet failover is
|
||||
the same code path, not separately tested.) The reranker groups are still load-shared as below.
|
||||
|
||||
**Load-shared with esh-ml1 since 2026-09-26 (Prime); embeddings moved to primary/failover 2026-10-05, above.** The gateway split
|
||||
`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` across both hosts at
|
||||
random, one `simple-shuffle` deployment each. Measured with 40 + 40 requests: embed
|
||||
12 esh / 28 nh3, rerank 22 / 18.
|
||||
|
||||
Reference in New Issue
Block a user