feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1
Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics read at source in v1.97.0): lowest order among healthy deployments, and a failed request retries at the next order. Verified after the restart (43 s gateway outage): 6/6 calls per alias landed on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6 from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1. The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does not (403).
This commit is contained in:
@@ -4,9 +4,10 @@
|
||||
RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**.
|
||||
It runs on **two hosts**, esh-ml1 and **nh3-ml1** (CT 109 on nh3-pve, the same
|
||||
card). The two are parity-verified as indistinguishable, and since 2026-09-26
|
||||
(Prime) they are **load-shared** behind the gateway names `qwen3-embedding`,
|
||||
`reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover
|
||||
(`servers/nh3-ml1/README.md` has the measurements).
|
||||
(Prime) they are **load-shared** behind `reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover
|
||||
(`servers/nh3-ml1/README.md` has the measurements). **Embeddings are primary + failover since 2026-10-05
|
||||
(Prime):** `qwen3-embedding` esh-ml1 first, `qwen3-embedding-bulk` (albok writes) nh3-ml1 first, set with LiteLLM
|
||||
`order` in `stacks/litellm/conf/config.yaml`.
|
||||
|
||||
**TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or
|
||||
reranking seats go on TEI, not vLLM. Why, and the measurements behind it:
|
||||
|
||||
Reference in New Issue
Block a user