feat(litellm): load-share embed/rerank across esh-ml1 + nh3-ml1 with intra-group failover
- qwen3-embedding and reranker: second deployment on nh3-ml1 (config); reranker-a3-bge-v2-m3: second DB deployment via /model/new. - router_settings.enable_weighted_failover: true. Only the three multi-deployment groups are affected. - Measured by stopping nh3-ml1 TEI: without failover 7/40 embeds 500'd; with it rerank 80/80, embed 38/40 at onset and 60/60 over a 34 s outage; nh3 rejoins rotation after restart. Split 12/28 embed, 22/18 rerank.
This commit is contained in:
@@ -130,8 +130,7 @@ hermes-gateway, Booth, ttyd, zellij-web and dev-backup are running.
|
||||
wired: Beszel `1feeeq61g4mkqre` with 5 alerts, Kuma #29/#30, Homepage cards,
|
||||
Dozzle agent (hub 7→8 clients). DNS `nh3-ml1.nh3.internal` is set.
|
||||
`servers/nh3-ml1/README.md` exists.
|
||||
- ⏳ **Prime's call: gateway routing.** Recommendation: load-share
|
||||
`qwen3-embedding` + `reranker` across esh-ml1 and nh3-ml1.
|
||||
- ✅ **Load-shared behind the gateway since 2026-09-26** (Prime), with intra-group failover; measured in `servers/nh3-ml1/README.md`.
|
||||
- ✅ **AMT on nh3-mgmt, static `10.100.250.61` (2306),** `nh3-pve-amt.nh3.internal`,
|
||||
Homepage *NH3-PVE-AMT*. Password in the vault as `nh3-pve/amt-admin`.
|
||||
- UDM port 6 is native mgmt with tags blocked.
|
||||
@@ -161,8 +160,8 @@ hermes-gateway, Booth, ttyd, zellij-web and dev-backup are running.
|
||||
|
||||
### ⭐ esh-ml1 = the fleet's utility GPU host (2026-09-25): embed/rerank (TEI) + reward (vLLM)
|
||||
|
||||
TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE gateway
|
||||
backend (nh3-ml1 is live and parity-verified; routing it in is Prime's call). The Skywork reward seat moved
|
||||
TEI is the fleet embed/rerank engine (Prime), LOAD-SHARED across esh-ml1 and
|
||||
nh3-ml1 since 2026-09-26. The Skywork reward seat moved
|
||||
there too; the audit found nothing supersedes it. fv-ml1 GPU 1 freed ~15 GB
|
||||
across the day; only `vllm-coder` is left in `stacks/vllm`, and **parakeet stays**
|
||||
on fv-ml1. Monitoring is wired: Beszel with GPU and 5 alerts, Kuma #27/#28,
|
||||
@@ -203,6 +202,7 @@ in `servers/pfi-gx10/README.md`. → `persistent-memory.d/2026-09-24-gx10-ac-res
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-26]` **Embed/rerank LOAD-SHARED across esh-ml1 + nh3-ml1 (Prime).** Second deployments were added for qwen3-embedding and reranker (config) and for reranker-a3-bge-v2-m3 (DB, `/model/new`). Without failover, a dead backend failed 7/40 embeds; `router_settings.enable_weighted_failover: true` (affects only multi-deployment groups) brought it to rerank 80/80 and embed 38/40 at onset, 60/60 sustained. In LiteLLM 1.97 the embedding path does not stamp the failed deployment, so it relies on cooldown. LiteLLM restarted 3× tonight, ~36 s each.
|
||||
- `[2026-09-26]` **Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed):** LFM2.5-VL-3B on llama.cpp `:8030` (gateway `lfm25-vl-3b`, LiteLLM restarted 36 s at 0039) and VibeVoice-ASR-Streaming-1.5B on **audio.cpp** `:8031`, not vibevoice.cpp as specced; the GGUF card names audio.cpp. Controls: VL read a synthetic image exactly, but hallucinates when no image is sent; ASR WER 3/69 on the bundled LibriSpeech clips. libsoxr added (linear resampling misheard a word). First ASR request ~31 s cold. Answered brokkr: the old 27B image seat (10.250.50.54:8015) was retired 09-14; baseline = gateway `image-judge` (Flash-Next).
|
||||
- `[2026-09-26]` **Coder seat STAYS on fv-ml1 (Prime).** The nh3-ml1 copy gave the same quality (teacher-forced true-code logprob diff +0.008 ± 0.019) but ran ~5× slower (64-tok FIM ~1.0 s vs 0.2 s; 63 vs 338 tok/s), and freeing 6.3 GB on fv-ml1 GPU 1 (20 GB spare) bought little. Copy removed; recipe kept in `stacks/coder-seat/`. The RTX 2000E suits embed/rerank/classify, not latency-sensitive generation. Only coder was a candidate: parakeet stays (earlier ruling), voices is a generation seat.
|
||||
- `[2026-09-25]` **nh3-ml1 LIVE after the NH3 visit (SB off, IGFX restored): driver + CT 109 + TEI, parity vs esh-ml1 indistinguishable (cos min 0.999993 = own floor; overlap@10 1.000 vs MRL-256 control 0.684), same speed; monitoring/DNS wired. Autonomous: lxc-pve 6.0.0-2 upgrade on nh3-pve (Docker-in-LXC fix #7006). Gateway routing + AMT follow-up are Prime's.** → `persistent-memory.d/2026-09-25-nh3-ml1-live.md`
|
||||
|
||||
@@ -5,12 +5,11 @@ GPU LXC for the ESH home lab: **CT 110 on esh-pve**, holding the **NVIDIA RTX
|
||||
reranking service** — the only backend behind the gateway's `qwen3-embedding`,
|
||||
`reranker` and `reranker-a3-bge-v2-m3` since 2026-09-25. Built 2026-09-24.
|
||||
|
||||
⚠ **Still the single gateway backend.** The second RTX 2000 is live as
|
||||
[nh3-ml1](../nh3-ml1/README.md) (2026-09-25, parity-verified: the two hosts cannot
|
||||
be told apart), but LiteLLM does not route to it yet; that is Prime's call. Until
|
||||
it does, if esh-ml1, esh-pve, or ESH's mesh route (esh-scale, CT 108) is down,
|
||||
fleet embeddings and reranking are down: Worldtree recall, nevermore clustering,
|
||||
Open WebUI RAG.
|
||||
**Load-shared with [nh3-ml1](../nh3-ml1/README.md) since 2026-09-26 (Prime).** The
|
||||
gateway splits `qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` across both
|
||||
TEI hosts, which are parity-verified as indistinguishable. Losing either host no
|
||||
longer takes fleet embeddings down; the failover test results are in the nh3-ml1
|
||||
README. The reward seat (`/scalar-judge`) is still esh-ml1 only.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
|
||||
@@ -6,9 +6,24 @@ reranking backend**, the twin of [esh-ml1](../esh-ml1/README.md): same card, sam
|
||||
driver, same TEI image and models. Built 2026-09-25, after the NH3 site visit
|
||||
turned Secure Boot off on nh3-pve.
|
||||
|
||||
⚠ **Not behind the gateway yet.** It serves on its own ports and is monitored, but
|
||||
LiteLLM still routes `qwen3-embedding` and `reranker` to esh-ml1 alone. Routing is
|
||||
Prime's call (recommendation: load-share; see below).
|
||||
**Load-shared with esh-ml1 since 2026-09-26 (Prime).** The gateway splits
|
||||
`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` across both hosts at
|
||||
random, one `simple-shuffle` deployment each. Measured with 40 + 40 requests: embed
|
||||
12 esh / 28 nh3, rerank 22 / 18.
|
||||
|
||||
**Failover, measured by stopping this host's TEI** (2026-09-26 0747–0753):
|
||||
- Without failover settings: **7/40 embeds returned 500**. The router's default 2
|
||||
retries re-picked at random and hit the dead backend again.
|
||||
- With `router_settings.enable_weighted_failover: true`, which only affects
|
||||
multi-deployment groups:
|
||||
- rerank **80/80** OK.
|
||||
- embeds **38/40** in the first test (2 failures at outage onset) and **60/60**
|
||||
across a 34 s sustained outage.
|
||||
- Once TEI restarted, nh3 was back in rotation (15/30).
|
||||
- Residual: in LiteLLM 1.97 the embedding path does not stamp the failed
|
||||
deployment, so embeds rely on the router's cooldown rather than per-request
|
||||
re-pick. Expect a couple of failed embeds when a backend first dies. n = 2
|
||||
outages.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
|
||||
@@ -2,11 +2,11 @@
|
||||
|
||||
**The fleet's embedding + reranking service**, on **esh-ml1** (CT 110 on esh-pve,
|
||||
RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**.
|
||||
Since 2026-09-25 it is the only backend behind the gateway names
|
||||
`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3`. A second instance
|
||||
runs on **nh3-ml1** (CT 109 on nh3-pve, the same card), parity-verified against
|
||||
esh-ml1 on 2026-09-25 (`servers/nh3-ml1/README.md`). It is not in the gateway
|
||||
yet; that is Prime's call.
|
||||
It runs on **two hosts**, esh-ml1 and **nh3-ml1** (CT 109 on nh3-pve, the same
|
||||
card). The two are parity-verified as indistinguishable, and since 2026-09-26
|
||||
(Prime) they are **load-shared** behind the gateway names `qwen3-embedding`,
|
||||
`reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover
|
||||
(`servers/nh3-ml1/README.md` has the measurements).
|
||||
|
||||
**TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or
|
||||
reranking seats go on TEI, not vLLM. Why, and the measurements behind it:
|
||||
|
||||
@@ -443,15 +443,14 @@ model_list:
|
||||
# explicitly. Operator ruling 2026-08-23. ---
|
||||
|
||||
# --- Qwen3 embeddings ---
|
||||
# ONE backend: esh-ml1 (RTX 2000E Ada), served by Hugging Face TEI — the fleet's
|
||||
# embed/rerank engine from 2026-09-25 (Prime). fv-ml1's vLLM seat was retired the
|
||||
# same day. TEI's /v1/embeddings is OpenAI-compatible, so the provider stays
|
||||
# `hosted_vllm/`. Same model as before; TEI vectors match the old vLLM ones
|
||||
# (cosine median 0.999925, inside vLLM's own noise; old-index retrieval overlap
|
||||
# 0.988 vs 0.986 self-noise), so no index needed re-embedding.
|
||||
# ⚠ Single backend until the second RTX 2000 lands (Prime, 2026-09-25): if
|
||||
# esh-ml1, esh-pve or the ESH mesh route is down, fleet embeddings are down.
|
||||
# ⚠ Never point this name at a DIFFERENT embedding model — vectors are
|
||||
# TWO backends, load-shared (simple-shuffle), since 2026-09-26 (Prime): esh-ml1
|
||||
# (ESH) and nh3-ml1 (NH3). Both are an RTX 2000E Ada running Hugging Face TEI
|
||||
# 89-1.9.4 with the same model. Parity measured 2026-09-25: cosine esh vs nh3
|
||||
# min 0.999993 = each host's own floor; overlap@10 1.000
|
||||
# (servers/nh3-ml1/README.md). TEI's /v1/embeddings is OpenAI-compatible, so the
|
||||
# provider is `hosted_vllm/`. TEI vectors match the retired vLLM seats (cosine
|
||||
# median 0.999925), so no index needed re-embedding.
|
||||
# ⚠ Never point this name at a DIFFERENT embedding model: vectors are
|
||||
# model-specific. Engine bake-off: docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md.
|
||||
- model_name: qwen3-embedding
|
||||
litellm_params:
|
||||
@@ -460,6 +459,13 @@ model_list:
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
model_info:
|
||||
mode: embedding
|
||||
- model_name: qwen3-embedding
|
||||
litellm_params:
|
||||
model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
|
||||
api_base: http://10.100.50.80:8001/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
model_info:
|
||||
mode: embedding
|
||||
|
||||
# --- qwen3-reranker RETIRED 2026-08-20. It named Qwen3-Reranker-0.6B on :8002,
|
||||
# the incumbent the R43 bake-off replaced on 2026-08-06 after measuring it
|
||||
@@ -498,12 +504,13 @@ model_list:
|
||||
model_info:
|
||||
mode: chat
|
||||
# reranker → generic capability name for rerank (bge-reranker-v2-m3 since the
|
||||
# R43 cutover). ONE backend: TEI on esh-ml1 (fv-ml1's vLLM seat retired
|
||||
# 2026-09-25). ⚠ TEI's /rerank takes `texts`, not `documents`: this MUST use the
|
||||
# R43 cutover). TWO backends, load-shared, since 2026-09-26 (Prime): TEI on
|
||||
# esh-ml1 and nh3-ml1. esh vs nh3 max score diff 0.0014, inside esh's own
|
||||
# 0.0020 run-to-run floor; top-1 identical. The DB-only alias
|
||||
# `reranker-a3-bge-v2-m3` carries the same two deployments.
|
||||
# ⚠ TEI's /rerank takes `texts`, not `documents`: this MUST use the
|
||||
# `huggingface/` provider. `hosted_vllm/` gets a 422 "missing field texts".
|
||||
# api_base has NO /v1. api_key is a placeholder; TEI runs without auth.
|
||||
# Parity vs the retired vLLM seat: top-1/top-3 identical (129/130 top-1 over
|
||||
# two runs); scores move up to 0.019, reshuffling only near-zero tail docs.
|
||||
- model_name: reranker
|
||||
litellm_params:
|
||||
model: huggingface/BAAI/bge-reranker-v2-m3
|
||||
@@ -511,6 +518,13 @@ model_list:
|
||||
api_key: "none"
|
||||
model_info:
|
||||
mode: rerank
|
||||
- model_name: reranker
|
||||
litellm_params:
|
||||
model: huggingface/BAAI/bge-reranker-v2-m3
|
||||
api_base: http://10.100.50.80:8013
|
||||
api_key: "none"
|
||||
model_info:
|
||||
mode: rerank
|
||||
|
||||
# --- lfm25-vl-3b → LiquidAI LFM2.5-VL-3B (Q5_K_M + mmproj Q8_0), llama.cpp server on
|
||||
# nh3-ml1 :8030 (stacks/lfm-vl-seat, 2026-09-26). Image understanding for
|
||||
@@ -1077,6 +1091,18 @@ general_settings:
|
||||
forward_headers: true
|
||||
include_subpath: true
|
||||
|
||||
router_settings:
|
||||
# Intra-group failover for simple-shuffle. When one deployment fails, the SAME
|
||||
# request re-picks among the group's other healthy deployments, excluding the
|
||||
# failed one. Without it, a dead backend in a 2-deployment group kept failing
|
||||
# a share of requests: the router's default 2 retries re-picked at random and
|
||||
# hit the dead one again. Measured 2026-09-26 with nh3-ml1 TEI stopped: 7/40
|
||||
# embeds returned 500.
|
||||
# Affects ONLY multi-deployment groups (qwen3-embedding, reranker,
|
||||
# reranker-a3-bge-v2-m3 as of 2026-09-26). A single-deployment group has no
|
||||
# other deployment to re-pick, so its behaviour is unchanged.
|
||||
enable_weighted_failover: true
|
||||
|
||||
litellm_settings:
|
||||
# vLLM rejects some OpenAI params other backends accept; drop silently
|
||||
# rather than 400 the caller.
|
||||
|
||||
@@ -110,11 +110,11 @@ monitors:
|
||||
|
||||
- name: Embed — Qwen3 0.6B (TEI, nh3-ml1)
|
||||
url: http://10.100.50.80:8001/health
|
||||
description: second `qwen3-embedding` backend (nh3-ml1)
|
||||
description: gateway `qwen3-embedding` backend (nh3-ml1)
|
||||
|
||||
- name: Rerank — bge-v2-m3 (TEI, nh3-ml1)
|
||||
url: http://10.100.50.80:8013/health
|
||||
description: second `reranker` backend (nh3-ml1)
|
||||
description: gateway `reranker` backend (nh3-ml1)
|
||||
|
||||
- name: talk
|
||||
url: https://talk.nh3.phasefinal.com:8092/
|
||||
|
||||
Reference in New Issue
Block a user