From c0d3384c68c381e1ceced4fa490f7e04d12c9b33 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 5 Oct 2026 00:32:42 -0700 Subject: [PATCH] =?UTF-8?q?feat(litellm):=20embedder=20primary/failover=20?= =?UTF-8?q?split=20=E2=80=94=20qwen3-embedding=20on=20esh-ml1,=20new=20qwe?= =?UTF-8?q?n3-embedding-bulk=20on=20nh3-ml1?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics read at source in v1.97.0): lowest order among healthy deployments, and a failed request retries at the next order. Verified after the restart (43 s gateway outage): 6/6 calls per alias landed on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6 from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1. The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does not (403). --- persistent-memory.md | 5 +-- servers/esh-ml1/README.md | 8 ++++- servers/nh3-ml1/README.md | 8 ++++- stacks/embed-rerank/README.md | 7 +++-- stacks/litellm/conf/config.yaml | 56 ++++++++++++++++++++++++++------- 5 files changed, 65 insertions(+), 19 deletions(-) diff --git a/persistent-memory.md b/persistent-memory.md index 0b0c839..5c763b5 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -177,8 +177,8 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca `10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None, password in the vault as `nh3-pve/amt-admin`. → `servers/nh3-pve/README.md` - **nh3-ml1** (CT 109 @ `10.100.50.80`): - - TEI embed/rerank is **load-shared with esh-ml1** through the gateway, with - `router_settings.enable_weighted_failover`. + - TEI rerank is **load-shared with esh-ml1** through the gateway (`enable_weighted_failover`). Embeddings are + **primary/failover since 2026-10-05**: `qwen3-embedding` esh-ml1 first, `qwen3-embedding-bulk` (albok writes) nh3-ml1 first. - brokkr's foundry seats: - `lfm-vl` `:8030` (gateway `lfm25-vl-3b`); - `lfm-vl-uncensored` `:8032` (direct; passed brokkr's eval); @@ -299,6 +299,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca ## Recent decisions +- `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml` - `[2026-10-04]` **albok-service moved to nh3-albok, a VM that runs only albok (albok-dev relaying Prime)** — nh3-docker's socket containers and plaintext tcp/2375 were root over the restricted wings; nh3-pve-2 vmbr0 VLAN-aware by reboot; PBS ns `nh3-pve-2`; copy verified by digest (dir st_size differs on ext4, not data). → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md` - `[2026-10-04]` **The whole fleet is on PVE 9.2.21 and PBS 4.2.7** — per-host traps (pfi-pve's grub-pc on UEFI, esh-pve's dead bridge port, nh3-pve's missing 7.0 headers, PBS via guest agent) and what a pfi-pve reboot costs. → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md` - `[2026-10-04]` **pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + `infra-ops-pages` tokens** — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md` diff --git a/servers/esh-ml1/README.md b/servers/esh-ml1/README.md index 9be556d..ecd6f53 100644 --- a/servers/esh-ml1/README.md +++ b/servers/esh-ml1/README.md @@ -5,7 +5,13 @@ GPU LXC for the ESH home lab: **CT 110 on esh-pve**, holding the **NVIDIA RTX reranking service** — the only backend behind the gateway's `qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` since 2026-09-25. Built 2026-09-24. -**Load-shared with [nh3-ml1](../nh3-ml1/README.md) since 2026-09-26 (Prime).** The +**Embedding routing since 2026-10-05 0030 (Prime): primary + failover, no longer shuffled.** `qwen3-embedding` +(fleet default) = esh-ml1 order 1, nh3-ml1 order 2; `qwen3-embedding-bulk` (albok-service's write path) = nh3-ml1 order 1, +esh-ml1 order 2. Measured 0730: 6/6 of each alias on its primary; with nh3-ml1's `tei-embed` stopped, bulk went 6/6 to esh-ml1 +with 200s and fleet stayed on esh-ml1, then bulk returned 6/6 to nh3-ml1 once it was back. (esh-ml1-down → fleet failover is +the same code path, not separately tested.) The reranker groups are still load-shared as below. + +**Load-shared with [nh3-ml1](../nh3-ml1/README.md) since 2026-09-26 (Prime), rerank only since 2026-10-05.** The gateway splits `qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` across both TEI hosts, which are parity-verified as indistinguishable. Losing either host no longer takes fleet embeddings down; the failover test results are in the nh3-ml1 diff --git a/servers/nh3-ml1/README.md b/servers/nh3-ml1/README.md index 90720c5..53b19ef 100644 --- a/servers/nh3-ml1/README.md +++ b/servers/nh3-ml1/README.md @@ -6,7 +6,13 @@ reranking backend**, the twin of [esh-ml1](../esh-ml1/README.md): same card, sam driver, same TEI image and models. Built 2026-09-25, after the NH3 site visit turned Secure Boot off on nh3-pve. -**Load-shared with esh-ml1 since 2026-09-26 (Prime).** The gateway splits +**Embedding routing since 2026-10-05 0030 (Prime): primary + failover, no longer shuffled.** `qwen3-embedding` +(fleet default) = esh-ml1 order 1, nh3-ml1 order 2; `qwen3-embedding-bulk` (albok-service's write path) = nh3-ml1 order 1, +esh-ml1 order 2. Measured 0730: 6/6 of each alias on its primary; with nh3-ml1's `tei-embed` stopped, bulk went 6/6 to esh-ml1 +with 200s and fleet stayed on esh-ml1, then bulk returned 6/6 to nh3-ml1 once it was back. (esh-ml1-down → fleet failover is +the same code path, not separately tested.) The reranker groups are still load-shared as below. + +**Load-shared with esh-ml1 since 2026-09-26 (Prime); embeddings moved to primary/failover 2026-10-05, above.** The gateway split `qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` across both hosts at random, one `simple-shuffle` deployment each. Measured with 40 + 40 requests: embed 12 esh / 28 nh3, rerank 22 / 18. diff --git a/stacks/embed-rerank/README.md b/stacks/embed-rerank/README.md index d20004d..697914c 100644 --- a/stacks/embed-rerank/README.md +++ b/stacks/embed-rerank/README.md @@ -4,9 +4,10 @@ RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**. It runs on **two hosts**, esh-ml1 and **nh3-ml1** (CT 109 on nh3-pve, the same card). The two are parity-verified as indistinguishable, and since 2026-09-26 -(Prime) they are **load-shared** behind the gateway names `qwen3-embedding`, -`reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover -(`servers/nh3-ml1/README.md` has the measurements). +(Prime) they are **load-shared** behind `reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover +(`servers/nh3-ml1/README.md` has the measurements). **Embeddings are primary + failover since 2026-10-05 +(Prime):** `qwen3-embedding` esh-ml1 first, `qwen3-embedding-bulk` (albok writes) nh3-ml1 first, set with LiteLLM +`order` in `stacks/litellm/conf/config.yaml`. **TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or reranking seats go on TEI, not vLLM. Why, and the measurements behind it: diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index 3750fe9..fa5602b 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -7,7 +7,8 @@ # # Model-name → upstream mapping: # phi4-mini → vLLM :8004 (generative chat) -# qwen3-embedding → TEI on esh-ml1 :8001 (/v1/embeddings) +# qwen3-embedding → TEI on esh-ml1 :8001, nh3-ml1 failover (/v1/embeddings) +# qwen3-embedding-bulk → TEI on nh3-ml1 :8001, esh-ml1 failover (albok writes) # reranker → TEI on esh-ml1 :8013 (/rerank, huggingface/ provider) # * (wildcard) → llama-swap :9292 (the swappable generative zoo) # @@ -443,20 +444,34 @@ model_list: # explicitly. Operator ruling 2026-08-23. --- # --- Qwen3 embeddings --- - # TWO backends, load-shared (simple-shuffle), since 2026-09-26 (Prime): esh-ml1 - # (ESH) and nh3-ml1 (NH3). Both are an RTX 2000E Ada running Hugging Face TEI - # 89-1.9.4 with the same model. Parity measured 2026-09-25: cosine esh vs nh3 - # min 0.999993 = each host's own floor; overlap@10 1.000 - # (servers/nh3-ml1/README.md). TEI's /v1/embeddings is OpenAI-compatible, so the - # provider is `hosted_vllm/`. TEI vectors match the retired vLLM seats (cosine - # median 0.999925), so no index needed re-embedding. - # ⚠ Never point this name at a DIFFERENT embedding model: vectors are - # model-specific. Engine bake-off: docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md. + # TWO backends, esh-ml1 (ESH) and nh3-ml1 (NH3). Both are an RTX 2000E Ada running Hugging Face TEI + # 89-1.9.4 with the same model. Parity measured 2026-09-25: cosine esh vs nh3 min 0.999993 = each + # host's own floor; overlap@10 1.000 (servers/nh3-ml1/README.md). Re-measured 2026-10-05 by albok-dev + # (20 texts x 3 runs, direct to each TEI): min cosine 0.999996, the same as each backend against + # itself; cross-backend query/document ranking identical 5/5. TEI's /v1/embeddings is + # OpenAI-compatible, so the provider is `hosted_vllm/`. + # + # ROUTING (Prime 2026-10-05, replaces the 2026-09-26 simple-shuffle load-share for embeddings only; + # the reranker groups below are unchanged): PRIMARY + FAILOVER via `order`, never a split. + # qwen3-embedding fleet default esh-ml1 order 1, nh3-ml1 order 2 + # qwen3-embedding-bulk albok writes nh3-ml1 order 1, esh-ml1 order 2 + # so bulk ingest (an albok reindex is ~15k chunks) lands on nh3-ml1 and cannot starve the fleet's + # query path on esh-ml1, and either host going down slows its traffic rather than failing it. + # `order` semantics, read at source in this image (v1.97.0, litellm/utils.py + # _get_order_filtered_deployments + router.py "ORDER-BASED FALLBACKS"): the router picks the lowest + # `order` among HEALTHY (not cooled-down) deployments; when a request fails there, it retries the + # same group at the next `order` level before any external fallback. + # ⚠ Never point either name at a DIFFERENT embedding model: vectors are model-specific, and both + # aliases feed the same indexes. Engine bake-off: docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md. + # ⚠ TEI /info reports model_sha null on both hosts, so the weights revision is not pinned. A + # re-pull on one host could drift the pair apart silently; re-run the parity check after any + # re-pull (albok-dev, 2026-10-05). - model_name: qwen3-embedding litellm_params: model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B api_base: http://10.0.50.80:8001/v1 api_key: os.environ/VLLM_API_KEY + order: 1 model_info: mode: embedding - model_name: qwen3-embedding @@ -464,6 +479,23 @@ model_list: model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B api_base: http://10.100.50.80:8001/v1 api_key: os.environ/VLLM_API_KEY + order: 2 + model_info: + mode: embedding + - model_name: qwen3-embedding-bulk + litellm_params: + model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B + api_base: http://10.100.50.80:8001/v1 + api_key: os.environ/VLLM_API_KEY + order: 1 + model_info: + mode: embedding + - model_name: qwen3-embedding-bulk + litellm_params: + model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B + api_base: http://10.0.50.80:8001/v1 + api_key: os.environ/VLLM_API_KEY + order: 2 model_info: mode: embedding @@ -1107,8 +1139,8 @@ router_settings: # a share of requests: the router's default 2 retries re-picked at random and # hit the dead one again. Measured 2026-09-26 with nh3-ml1 TEI stopped: 7/40 # embeds returned 500. - # Affects ONLY multi-deployment groups (qwen3-embedding, reranker, - # reranker-a3-bge-v2-m3 as of 2026-09-26). A single-deployment group has no + # Affects ONLY multi-deployment groups (reranker and reranker-a3-bge-v2-m3; the two + # embedding groups route by `order` since 2026-10-05, see the Qwen3 embeddings block). A single-deployment group has no # other deployment to re-pick, so its behaviour is unchanged. enable_weighted_failover: true