feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1

Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces
the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker
groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics
read at source in v1.97.0): lowest order among healthy deployments, and a
failed request retries at the next order.

Verified after the restart (43 s gateway outage): 6/6 calls per alias landed
on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6
from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1.
The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does
not (403).

persistent-memory.md

@@ -177,8 +177,8 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
   `10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None, password in the
   vault as `nh3-pve/amt-admin`. → `servers/nh3-pve/README.md`
 - **nh3-ml1** (CT 109 @ `10.100.50.80`):
-  - TEI embed/rerank is **load-shared with esh-ml1** through the gateway, with
+  - TEI rerank is **load-shared with esh-ml1** through the gateway (`enable_weighted_failover`). Embeddings are
-    `router_settings.enable_weighted_failover`.
+    **primary/failover since 2026-10-05**: `qwen3-embedding` esh-ml1 first, `qwen3-embedding-bulk` (albok writes) nh3-ml1 first.
   - brokkr's foundry seats:
     - `lfm-vl` `:8030` (gateway `lfm25-vl-3b`);
     - `lfm-vl-uncensored` `:8032` (direct; passed brokkr's eval);
@@ -299,6 +299,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
 
 ## Recent decisions
 
+- `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml`
 - `[2026-10-04]` **albok-service moved to nh3-albok, a VM that runs only albok (albok-dev relaying Prime)** — nh3-docker's socket containers and plaintext tcp/2375 were root over the restricted wings; nh3-pve-2 vmbr0 VLAN-aware by reboot; PBS ns `nh3-pve-2`; copy verified by digest (dir st_size differs on ext4, not data). → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md`
 - `[2026-10-04]` **The whole fleet is on PVE 9.2.21 and PBS 4.2.7** — per-host traps (pfi-pve's grub-pc on UEFI, esh-pve's dead bridge port, nh3-pve's missing 7.0 headers, PBS via guest agent) and what a pfi-pve reboot costs. → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`
 - `[2026-10-04]` **pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + `infra-ops-pages` tokens** — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md`

servers/esh-ml1/README.md

@@ -5,7 +5,13 @@ GPU LXC for the ESH home lab: **CT 110 on esh-pve**, holding the **NVIDIA RTX
 reranking service** — the only backend behind the gateway's `qwen3-embedding`,
 `reranker` and `reranker-a3-bge-v2-m3` since 2026-09-25. Built 2026-09-24.
 
-**Load-shared with [nh3-ml1](../nh3-ml1/README.md) since 2026-09-26 (Prime).** The
+**Embedding routing since 2026-10-05 0030 (Prime): primary + failover, no longer shuffled.** `qwen3-embedding`
+(fleet default) = esh-ml1 order 1, nh3-ml1 order 2; `qwen3-embedding-bulk` (albok-service's write path) = nh3-ml1 order 1,
+esh-ml1 order 2. Measured 0730: 6/6 of each alias on its primary; with nh3-ml1's `tei-embed` stopped, bulk went 6/6 to esh-ml1
+with 200s and fleet stayed on esh-ml1, then bulk returned 6/6 to nh3-ml1 once it was back. (esh-ml1-down → fleet failover is
+the same code path, not separately tested.) The reranker groups are still load-shared as below.
+
+**Load-shared with [nh3-ml1](../nh3-ml1/README.md) since 2026-09-26 (Prime), rerank only since 2026-10-05.** The
 gateway splits `qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` across both
 TEI hosts, which are parity-verified as indistinguishable. Losing either host no
 longer takes fleet embeddings down; the failover test results are in the nh3-ml1

servers/nh3-ml1/README.md

@@ -6,7 +6,13 @@ reranking backend**, the twin of [esh-ml1](../esh-ml1/README.md): same card, sam
 driver, same TEI image and models. Built 2026-09-25, after the NH3 site visit
 turned Secure Boot off on nh3-pve.
 
-**Load-shared with esh-ml1 since 2026-09-26 (Prime).** The gateway splits
+**Embedding routing since 2026-10-05 0030 (Prime): primary + failover, no longer shuffled.** `qwen3-embedding`
+(fleet default) = esh-ml1 order 1, nh3-ml1 order 2; `qwen3-embedding-bulk` (albok-service's write path) = nh3-ml1 order 1,
+esh-ml1 order 2. Measured 0730: 6/6 of each alias on its primary; with nh3-ml1's `tei-embed` stopped, bulk went 6/6 to esh-ml1
+with 200s and fleet stayed on esh-ml1, then bulk returned 6/6 to nh3-ml1 once it was back. (esh-ml1-down → fleet failover is
+the same code path, not separately tested.) The reranker groups are still load-shared as below.
+
+**Load-shared with esh-ml1 since 2026-09-26 (Prime); embeddings moved to primary/failover 2026-10-05, above.** The gateway split
 `qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` across both hosts at
 random, one `simple-shuffle` deployment each. Measured with 40 + 40 requests: embed
 12 esh / 28 nh3, rerank 22 / 18.

stacks/embed-rerank/README.md

@@ -4,9 +4,10 @@
 RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**.
 It runs on **two hosts**, esh-ml1 and **nh3-ml1** (CT 109 on nh3-pve, the same
 card). The two are parity-verified as indistinguishable, and since 2026-09-26
-(Prime) they are **load-shared** behind the gateway names `qwen3-embedding`,
+(Prime) they are **load-shared** behind `reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover
-`reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover
+(`servers/nh3-ml1/README.md` has the measurements). **Embeddings are primary + failover since 2026-10-05
-(`servers/nh3-ml1/README.md` has the measurements).
+(Prime):** `qwen3-embedding` esh-ml1 first, `qwen3-embedding-bulk` (albok writes) nh3-ml1 first, set with LiteLLM
+`order` in `stacks/litellm/conf/config.yaml`.
 
 **TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or
 reranking seats go on TEI, not vLLM. Why, and the measurements behind it:

stacks/litellm/conf/config.yaml

@@ -7,7 +7,8 @@
 #
 # Model-name → upstream mapping:
 #   phi4-mini        → vLLM :8004  (generative chat)
-#   qwen3-embedding  → TEI on esh-ml1 :8001  (/v1/embeddings)
+#   qwen3-embedding  → TEI on esh-ml1 :8001, nh3-ml1 failover (/v1/embeddings)
+#   qwen3-embedding-bulk → TEI on nh3-ml1 :8001, esh-ml1 failover (albok writes)
 #   reranker         → TEI on esh-ml1 :8013  (/rerank, huggingface/ provider)
 #   *  (wildcard)    → llama-swap :9292  (the swappable generative zoo)
 #
@@ -443,20 +444,34 @@ model_list:
   #     explicitly. Operator ruling 2026-08-23. ---
 
   # --- Qwen3 embeddings ---
-  # TWO backends, load-shared (simple-shuffle), since 2026-09-26 (Prime): esh-ml1
+  # TWO backends, esh-ml1 (ESH) and nh3-ml1 (NH3). Both are an RTX 2000E Ada running Hugging Face TEI
-  # (ESH) and nh3-ml1 (NH3). Both are an RTX 2000E Ada running Hugging Face TEI
+  # 89-1.9.4 with the same model. Parity measured 2026-09-25: cosine esh vs nh3 min 0.999993 = each
-  # 89-1.9.4 with the same model. Parity measured 2026-09-25: cosine esh vs nh3
+  # host's own floor; overlap@10 1.000 (servers/nh3-ml1/README.md). Re-measured 2026-10-05 by albok-dev
-  # min 0.999993 = each host's own floor; overlap@10 1.000
+  # (20 texts x 3 runs, direct to each TEI): min cosine 0.999996, the same as each backend against
-  # (servers/nh3-ml1/README.md). TEI's /v1/embeddings is OpenAI-compatible, so the
+  # itself; cross-backend query/document ranking identical 5/5. TEI's /v1/embeddings is
-  # provider is `hosted_vllm/`. TEI vectors match the retired vLLM seats (cosine
+  # OpenAI-compatible, so the provider is `hosted_vllm/`.
-  # median 0.999925), so no index needed re-embedding.
+  #
-  # ⚠ Never point this name at a DIFFERENT embedding model: vectors are
+  # ROUTING (Prime 2026-10-05, replaces the 2026-09-26 simple-shuffle load-share for embeddings only;
-  # model-specific. Engine bake-off: docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md.
+  # the reranker groups below are unchanged): PRIMARY + FAILOVER via `order`, never a split.
+  #   qwen3-embedding       fleet default   esh-ml1 order 1, nh3-ml1 order 2
+  #   qwen3-embedding-bulk  albok writes    nh3-ml1 order 1, esh-ml1 order 2
+  # so bulk ingest (an albok reindex is ~15k chunks) lands on nh3-ml1 and cannot starve the fleet's
+  # query path on esh-ml1, and either host going down slows its traffic rather than failing it.
+  # `order` semantics, read at source in this image (v1.97.0, litellm/utils.py
+  # _get_order_filtered_deployments + router.py "ORDER-BASED FALLBACKS"): the router picks the lowest
+  # `order` among HEALTHY (not cooled-down) deployments; when a request fails there, it retries the
+  # same group at the next `order` level before any external fallback.
+  # ⚠ Never point either name at a DIFFERENT embedding model: vectors are model-specific, and both
+  #   aliases feed the same indexes. Engine bake-off: docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md.
+  # ⚠ TEI /info reports model_sha null on both hosts, so the weights revision is not pinned. A
+  #   re-pull on one host could drift the pair apart silently; re-run the parity check after any
+  #   re-pull (albok-dev, 2026-10-05).
   - model_name: qwen3-embedding
     litellm_params:
       model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
       api_base: http://10.0.50.80:8001/v1
       api_key: os.environ/VLLM_API_KEY
+      order: 1
     model_info:
       mode: embedding
   - model_name: qwen3-embedding
@@ -464,6 +479,23 @@ model_list:
       model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
       api_base: http://10.100.50.80:8001/v1
       api_key: os.environ/VLLM_API_KEY
+      order: 2
+    model_info:
+      mode: embedding
+  - model_name: qwen3-embedding-bulk
+    litellm_params:
+      model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
+      api_base: http://10.100.50.80:8001/v1
+      api_key: os.environ/VLLM_API_KEY
+      order: 1
+    model_info:
+      mode: embedding
+  - model_name: qwen3-embedding-bulk
+    litellm_params:
+      model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
+      api_base: http://10.0.50.80:8001/v1
+      api_key: os.environ/VLLM_API_KEY
+      order: 2
     model_info:
       mode: embedding
 
@@ -1107,8 +1139,8 @@ router_settings:
   # a share of requests: the router's default 2 retries re-picked at random and
   # hit the dead one again. Measured 2026-09-26 with nh3-ml1 TEI stopped: 7/40
   # embeds returned 500.
-  # Affects ONLY multi-deployment groups (qwen3-embedding, reranker,
+  # Affects ONLY multi-deployment groups (reranker and reranker-a3-bge-v2-m3; the two
-  # reranker-a3-bge-v2-m3 as of 2026-09-26). A single-deployment group has no
+  # embedding groups route by `order` since 2026-10-05, see the Qwen3 embeddings block). A single-deployment group has no
   # other deployment to re-pick, so its behaviour is unchanged.
   enable_weighted_failover: true