feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1

Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces
the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker
groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics
read at source in v1.97.0): lowest order among healthy deployments, and a
failed request retries at the next order.

Verified after the restart (43 s gateway outage): 6/6 calls per alias landed
on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6
from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1.
The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does
not (403).
This commit is contained in:
vh
2026-10-05 00:32:42 -07:00
parent a7a068f56f
commit c0d3384c68
5 changed files with 65 additions and 19 deletions
+4 -3
View File
@@ -4,9 +4,10 @@
RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**.
It runs on **two hosts**, esh-ml1 and **nh3-ml1** (CT 109 on nh3-pve, the same
card). The two are parity-verified as indistinguishable, and since 2026-09-26
(Prime) they are **load-shared** behind the gateway names `qwen3-embedding`,
`reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover
(`servers/nh3-ml1/README.md` has the measurements).
(Prime) they are **load-shared** behind `reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover
(`servers/nh3-ml1/README.md` has the measurements). **Embeddings are primary + failover since 2026-10-05
(Prime):** `qwen3-embedding` esh-ml1 first, `qwen3-embedding-bulk` (albok writes) nh3-ml1 first, set with LiteLLM
`order` in `stacks/litellm/conf/config.yaml`.
**TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or
reranking seats go on TEI, not vLLM. Why, and the measurements behind it: