feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1
Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics read at source in v1.97.0): lowest order among healthy deployments, and a failed request retries at the next order. Verified after the restart (43 s gateway outage): 6/6 calls per alias landed on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6 from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1. The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does not (403).
This commit is contained in:
@@ -4,9 +4,10 @@
|
||||
RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**.
|
||||
It runs on **two hosts**, esh-ml1 and **nh3-ml1** (CT 109 on nh3-pve, the same
|
||||
card). The two are parity-verified as indistinguishable, and since 2026-09-26
|
||||
(Prime) they are **load-shared** behind the gateway names `qwen3-embedding`,
|
||||
`reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover
|
||||
(`servers/nh3-ml1/README.md` has the measurements).
|
||||
(Prime) they are **load-shared** behind `reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover
|
||||
(`servers/nh3-ml1/README.md` has the measurements). **Embeddings are primary + failover since 2026-10-05
|
||||
(Prime):** `qwen3-embedding` esh-ml1 first, `qwen3-embedding-bulk` (albok writes) nh3-ml1 first, set with LiteLLM
|
||||
`order` in `stacks/litellm/conf/config.yaml`.
|
||||
|
||||
**TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or
|
||||
reranking seats go on TEI, not vLLM. Why, and the measurements behind it:
|
||||
|
||||
@@ -7,7 +7,8 @@
|
||||
#
|
||||
# Model-name → upstream mapping:
|
||||
# phi4-mini → vLLM :8004 (generative chat)
|
||||
# qwen3-embedding → TEI on esh-ml1 :8001 (/v1/embeddings)
|
||||
# qwen3-embedding → TEI on esh-ml1 :8001, nh3-ml1 failover (/v1/embeddings)
|
||||
# qwen3-embedding-bulk → TEI on nh3-ml1 :8001, esh-ml1 failover (albok writes)
|
||||
# reranker → TEI on esh-ml1 :8013 (/rerank, huggingface/ provider)
|
||||
# * (wildcard) → llama-swap :9292 (the swappable generative zoo)
|
||||
#
|
||||
@@ -443,20 +444,34 @@ model_list:
|
||||
# explicitly. Operator ruling 2026-08-23. ---
|
||||
|
||||
# --- Qwen3 embeddings ---
|
||||
# TWO backends, load-shared (simple-shuffle), since 2026-09-26 (Prime): esh-ml1
|
||||
# (ESH) and nh3-ml1 (NH3). Both are an RTX 2000E Ada running Hugging Face TEI
|
||||
# 89-1.9.4 with the same model. Parity measured 2026-09-25: cosine esh vs nh3
|
||||
# min 0.999993 = each host's own floor; overlap@10 1.000
|
||||
# (servers/nh3-ml1/README.md). TEI's /v1/embeddings is OpenAI-compatible, so the
|
||||
# provider is `hosted_vllm/`. TEI vectors match the retired vLLM seats (cosine
|
||||
# median 0.999925), so no index needed re-embedding.
|
||||
# ⚠ Never point this name at a DIFFERENT embedding model: vectors are
|
||||
# model-specific. Engine bake-off: docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md.
|
||||
# TWO backends, esh-ml1 (ESH) and nh3-ml1 (NH3). Both are an RTX 2000E Ada running Hugging Face TEI
|
||||
# 89-1.9.4 with the same model. Parity measured 2026-09-25: cosine esh vs nh3 min 0.999993 = each
|
||||
# host's own floor; overlap@10 1.000 (servers/nh3-ml1/README.md). Re-measured 2026-10-05 by albok-dev
|
||||
# (20 texts x 3 runs, direct to each TEI): min cosine 0.999996, the same as each backend against
|
||||
# itself; cross-backend query/document ranking identical 5/5. TEI's /v1/embeddings is
|
||||
# OpenAI-compatible, so the provider is `hosted_vllm/`.
|
||||
#
|
||||
# ROUTING (Prime 2026-10-05, replaces the 2026-09-26 simple-shuffle load-share for embeddings only;
|
||||
# the reranker groups below are unchanged): PRIMARY + FAILOVER via `order`, never a split.
|
||||
# qwen3-embedding fleet default esh-ml1 order 1, nh3-ml1 order 2
|
||||
# qwen3-embedding-bulk albok writes nh3-ml1 order 1, esh-ml1 order 2
|
||||
# so bulk ingest (an albok reindex is ~15k chunks) lands on nh3-ml1 and cannot starve the fleet's
|
||||
# query path on esh-ml1, and either host going down slows its traffic rather than failing it.
|
||||
# `order` semantics, read at source in this image (v1.97.0, litellm/utils.py
|
||||
# _get_order_filtered_deployments + router.py "ORDER-BASED FALLBACKS"): the router picks the lowest
|
||||
# `order` among HEALTHY (not cooled-down) deployments; when a request fails there, it retries the
|
||||
# same group at the next `order` level before any external fallback.
|
||||
# ⚠ Never point either name at a DIFFERENT embedding model: vectors are model-specific, and both
|
||||
# aliases feed the same indexes. Engine bake-off: docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md.
|
||||
# ⚠ TEI /info reports model_sha null on both hosts, so the weights revision is not pinned. A
|
||||
# re-pull on one host could drift the pair apart silently; re-run the parity check after any
|
||||
# re-pull (albok-dev, 2026-10-05).
|
||||
- model_name: qwen3-embedding
|
||||
litellm_params:
|
||||
model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
|
||||
api_base: http://10.0.50.80:8001/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
order: 1
|
||||
model_info:
|
||||
mode: embedding
|
||||
- model_name: qwen3-embedding
|
||||
@@ -464,6 +479,23 @@ model_list:
|
||||
model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
|
||||
api_base: http://10.100.50.80:8001/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
order: 2
|
||||
model_info:
|
||||
mode: embedding
|
||||
- model_name: qwen3-embedding-bulk
|
||||
litellm_params:
|
||||
model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
|
||||
api_base: http://10.100.50.80:8001/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
order: 1
|
||||
model_info:
|
||||
mode: embedding
|
||||
- model_name: qwen3-embedding-bulk
|
||||
litellm_params:
|
||||
model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
|
||||
api_base: http://10.0.50.80:8001/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
order: 2
|
||||
model_info:
|
||||
mode: embedding
|
||||
|
||||
@@ -1107,8 +1139,8 @@ router_settings:
|
||||
# a share of requests: the router's default 2 retries re-picked at random and
|
||||
# hit the dead one again. Measured 2026-09-26 with nh3-ml1 TEI stopped: 7/40
|
||||
# embeds returned 500.
|
||||
# Affects ONLY multi-deployment groups (qwen3-embedding, reranker,
|
||||
# reranker-a3-bge-v2-m3 as of 2026-09-26). A single-deployment group has no
|
||||
# Affects ONLY multi-deployment groups (reranker and reranker-a3-bge-v2-m3; the two
|
||||
# embedding groups route by `order` since 2026-10-05, see the Qwen3 embeddings block). A single-deployment group has no
|
||||
# other deployment to re-pick, so its behaviour is unchanged.
|
||||
enable_weighted_failover: true
|
||||
|
||||
|
||||
Reference in New Issue
Block a user