vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1

Browse Source 
Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces
the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker
groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics
read at source in v1.97.0): lowest order among healthy deployments, and a
failed request retries at the next order.

Verified after the restart (43 s gateway outage): 6/6 calls per alias landed
on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6
from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1.
The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does
not (403).
This commit is contained in:
vh
2026-10-05 00:32:42 -07:00
parent a7a068f56f
commit c0d3384c68
5 changed files with 65 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
servers/esh-ml1/README.md
+7 -1
View File
@@ -5,7 +5,13 @@ GPU LXC for the ESH home lab: **CT 110 on esh-pve**, holding the **NVIDIA RTX
reranking service** — the only backend behind the gateway's `qwen3-embedding`,
`reranker` and `reranker-a3-bge-v2-m3` since 2026-09-25. Built 2026-09-24.
**Load-shared with [nh3-ml1](../nh3-ml1/README.md) since 2026-09-26 (Prime).** The
**Embedding routing since 2026-10-05 0030 (Prime): primary + failover, no longer shuffled.** `qwen3-embedding`
(fleet default) = esh-ml1 order 1, nh3-ml1 order 2; `qwen3-embedding-bulk` (albok-service's write path) = nh3-ml1 order 1,
esh-ml1 order 2. Measured 0730: 6/6 of each alias on its primary; with nh3-ml1's `tei-embed` stopped, bulk went 6/6 to esh-ml1
with 200s and fleet stayed on esh-ml1, then bulk returned 6/6 to nh3-ml1 once it was back. (esh-ml1-down → fleet failover is
the same code path, not separately tested.) The reranker groups are still load-shared as below.
**Load-shared with [nh3-ml1](../nh3-ml1/README.md) since 2026-09-26 (Prime), rerank only since 2026-10-05.** The
gateway splits `qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` across both
TEI hosts, which are parity-verified as indistinguishable. Losing either host no
longer takes fleet embeddings down; the failover test results are in the nh3-ml1
servers/nh3-ml1/README.md
+7 -1
View File
@@ -6,7 +6,13 @@ reranking backend**, the twin of [esh-ml1](../esh-ml1/README.md): same card, sam
driver, same TEI image and models. Built 2026-09-25, after the NH3 site visit
turned Secure Boot off on nh3-pve.
**Load-shared with esh-ml1 since 2026-09-26 (Prime).** The gateway splits
**Embedding routing since 2026-10-05 0030 (Prime): primary + failover, no longer shuffled.** `qwen3-embedding`
(fleet default) = esh-ml1 order 1, nh3-ml1 order 2; `qwen3-embedding-bulk` (albok-service's write path) = nh3-ml1 order 1,
esh-ml1 order 2. Measured 0730: 6/6 of each alias on its primary; with nh3-ml1's `tei-embed` stopped, bulk went 6/6 to esh-ml1
with 200s and fleet stayed on esh-ml1, then bulk returned 6/6 to nh3-ml1 once it was back. (esh-ml1-down → fleet failover is
the same code path, not separately tested.) The reranker groups are still load-shared as below.
**Load-shared with esh-ml1 since 2026-09-26 (Prime); embeddings moved to primary/failover 2026-10-05, above.** The gateway split
`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` across both hosts at
random, one `simple-shuffle` deployment each. Measured with 40 + 40 requests: embed
12 esh / 28 nh3, rerank 22 / 18.