feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1

Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces
the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker
groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics
read at source in v1.97.0): lowest order among healthy deployments, and a
failed request retries at the next order.

Verified after the restart (43 s gateway outage): 6/6 calls per alias landed
on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6
from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1.
The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does
not (403).
This commit is contained in:
vh
2026-10-05 00:32:42 -07:00
parent a7a068f56f
commit c0d3384c68
5 changed files with 65 additions and 19 deletions
+3 -2
View File
@@ -177,8 +177,8 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
`10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None, password in the
vault as `nh3-pve/amt-admin`. → `servers/nh3-pve/README.md`
- **nh3-ml1** (CT 109 @ `10.100.50.80`):
- TEI embed/rerank is **load-shared with esh-ml1** through the gateway, with
`router_settings.enable_weighted_failover`.
- TEI rerank is **load-shared with esh-ml1** through the gateway (`enable_weighted_failover`). Embeddings are
**primary/failover since 2026-10-05**: `qwen3-embedding` esh-ml1 first, `qwen3-embedding-bulk` (albok writes) nh3-ml1 first.
- brokkr's foundry seats:
- `lfm-vl` `:8030` (gateway `lfm25-vl-3b`);
- `lfm-vl-uncensored` `:8032` (direct; passed brokkr's eval);
@@ -299,6 +299,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
## Recent decisions
- `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml`
- `[2026-10-04]` **albok-service moved to nh3-albok, a VM that runs only albok (albok-dev relaying Prime)** — nh3-docker's socket containers and plaintext tcp/2375 were root over the restricted wings; nh3-pve-2 vmbr0 VLAN-aware by reboot; PBS ns `nh3-pve-2`; copy verified by digest (dir st_size differs on ext4, not data). → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md`
- `[2026-10-04]` **The whole fleet is on PVE 9.2.21 and PBS 4.2.7** — per-host traps (pfi-pve's grub-pc on UEFI, esh-pve's dead bridge port, nh3-pve's missing 7.0 headers, PBS via guest agent) and what a pfi-pve reboot costs. → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`
- `[2026-10-04]` **pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + `infra-ops-pages` tokens** — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md`