feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1
Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics read at source in v1.97.0): lowest order among healthy deployments, and a failed request retries at the next order. Verified after the restart (43 s gateway outage): 6/6 calls per alias landed on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6 from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1. The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does not (403).
This commit is contained in:
@@ -177,8 +177,8 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
|
||||
`10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None, password in the
|
||||
vault as `nh3-pve/amt-admin`. → `servers/nh3-pve/README.md`
|
||||
- **nh3-ml1** (CT 109 @ `10.100.50.80`):
|
||||
- TEI embed/rerank is **load-shared with esh-ml1** through the gateway, with
|
||||
`router_settings.enable_weighted_failover`.
|
||||
- TEI rerank is **load-shared with esh-ml1** through the gateway (`enable_weighted_failover`). Embeddings are
|
||||
**primary/failover since 2026-10-05**: `qwen3-embedding` esh-ml1 first, `qwen3-embedding-bulk` (albok writes) nh3-ml1 first.
|
||||
- brokkr's foundry seats:
|
||||
- `lfm-vl` `:8030` (gateway `lfm25-vl-3b`);
|
||||
- `lfm-vl-uncensored` `:8032` (direct; passed brokkr's eval);
|
||||
@@ -299,6 +299,7 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml`
|
||||
- `[2026-10-04]` **albok-service moved to nh3-albok, a VM that runs only albok (albok-dev relaying Prime)** — nh3-docker's socket containers and plaintext tcp/2375 were root over the restricted wings; nh3-pve-2 vmbr0 VLAN-aware by reboot; PBS ns `nh3-pve-2`; copy verified by digest (dir st_size differs on ext4, not data). → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md`
|
||||
- `[2026-10-04]` **The whole fleet is on PVE 9.2.21 and PBS 4.2.7** — per-host traps (pfi-pve's grub-pc on UEFI, esh-pve's dead bridge port, nh3-pve's missing 7.0 headers, PBS via guest agent) and what a pfi-pve reboot costs. → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`
|
||||
- `[2026-10-04]` **pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + `infra-ops-pages` tokens** — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md`
|
||||
|
||||
Reference in New Issue
Block a user