diff --git a/persistent-memory.md b/persistent-memory.md index ebf9022..5f71786 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -130,8 +130,7 @@ hermes-gateway, Booth, ttyd, zellij-web and dev-backup are running. wired: Beszel `1feeeq61g4mkqre` with 5 alerts, Kuma #29/#30, Homepage cards, Dozzle agent (hub 7→8 clients). DNS `nh3-ml1.nh3.internal` is set. `servers/nh3-ml1/README.md` exists. -- ⏳ **Prime's call: gateway routing.** Recommendation: load-share - `qwen3-embedding` + `reranker` across esh-ml1 and nh3-ml1. +- ✅ **Load-shared behind the gateway since 2026-09-26** (Prime), with intra-group failover; measured in `servers/nh3-ml1/README.md`. - ✅ **AMT on nh3-mgmt, static `10.100.250.61` (2306),** `nh3-pve-amt.nh3.internal`, Homepage *NH3-PVE-AMT*. Password in the vault as `nh3-pve/amt-admin`. - UDM port 6 is native mgmt with tags blocked. @@ -161,8 +160,8 @@ hermes-gateway, Booth, ttyd, zellij-web and dev-backup are running. ### ⭐ esh-ml1 = the fleet's utility GPU host (2026-09-25): embed/rerank (TEI) + reward (vLLM) -TEI is the fleet embed/rerank engine (Prime) and esh-ml1 is the SOLE gateway -backend (nh3-ml1 is live and parity-verified; routing it in is Prime's call). The Skywork reward seat moved +TEI is the fleet embed/rerank engine (Prime), LOAD-SHARED across esh-ml1 and +nh3-ml1 since 2026-09-26. The Skywork reward seat moved there too; the audit found nothing supersedes it. fv-ml1 GPU 1 freed ~15 GB across the day; only `vllm-coder` is left in `stacks/vllm`, and **parakeet stays** on fv-ml1. Monitoring is wired: Beszel with GPU and 5 alerts, Kuma #27/#28, @@ -203,6 +202,7 @@ in `servers/pfi-gx10/README.md`. → `persistent-memory.d/2026-09-24-gx10-ac-res ## Recent decisions +- `[2026-09-26]` **Embed/rerank LOAD-SHARED across esh-ml1 + nh3-ml1 (Prime).** Second deployments were added for qwen3-embedding and reranker (config) and for reranker-a3-bge-v2-m3 (DB, `/model/new`). Without failover, a dead backend failed 7/40 embeds; `router_settings.enable_weighted_failover: true` (affects only multi-deployment groups) brought it to rerank 80/80 and embed 38/40 at onset, 60/60 sustained. In LiteLLM 1.97 the embedding path does not stamp the failed deployment, so it relies on cooldown. LiteLLM restarted 3× tonight, ~36 s each. - `[2026-09-26]` **Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed):** LFM2.5-VL-3B on llama.cpp `:8030` (gateway `lfm25-vl-3b`, LiteLLM restarted 36 s at 0039) and VibeVoice-ASR-Streaming-1.5B on **audio.cpp** `:8031`, not vibevoice.cpp as specced; the GGUF card names audio.cpp. Controls: VL read a synthetic image exactly, but hallucinates when no image is sent; ASR WER 3/69 on the bundled LibriSpeech clips. libsoxr added (linear resampling misheard a word). First ASR request ~31 s cold. Answered brokkr: the old 27B image seat (10.250.50.54:8015) was retired 09-14; baseline = gateway `image-judge` (Flash-Next). - `[2026-09-26]` **Coder seat STAYS on fv-ml1 (Prime).** The nh3-ml1 copy gave the same quality (teacher-forced true-code logprob diff +0.008 ± 0.019) but ran ~5× slower (64-tok FIM ~1.0 s vs 0.2 s; 63 vs 338 tok/s), and freeing 6.3 GB on fv-ml1 GPU 1 (20 GB spare) bought little. Copy removed; recipe kept in `stacks/coder-seat/`. The RTX 2000E suits embed/rerank/classify, not latency-sensitive generation. Only coder was a candidate: parakeet stays (earlier ruling), voices is a generation seat. - `[2026-09-25]` **nh3-ml1 LIVE after the NH3 visit (SB off, IGFX restored): driver + CT 109 + TEI, parity vs esh-ml1 indistinguishable (cos min 0.999993 = own floor; overlap@10 1.000 vs MRL-256 control 0.684), same speed; monitoring/DNS wired. Autonomous: lxc-pve 6.0.0-2 upgrade on nh3-pve (Docker-in-LXC fix #7006). Gateway routing + AMT follow-up are Prime's.** → `persistent-memory.d/2026-09-25-nh3-ml1-live.md` diff --git a/servers/esh-ml1/README.md b/servers/esh-ml1/README.md index e905c90..887e153 100644 --- a/servers/esh-ml1/README.md +++ b/servers/esh-ml1/README.md @@ -5,12 +5,11 @@ GPU LXC for the ESH home lab: **CT 110 on esh-pve**, holding the **NVIDIA RTX reranking service** — the only backend behind the gateway's `qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` since 2026-09-25. Built 2026-09-24. -⚠ **Still the single gateway backend.** The second RTX 2000 is live as -[nh3-ml1](../nh3-ml1/README.md) (2026-09-25, parity-verified: the two hosts cannot -be told apart), but LiteLLM does not route to it yet; that is Prime's call. Until -it does, if esh-ml1, esh-pve, or ESH's mesh route (esh-scale, CT 108) is down, -fleet embeddings and reranking are down: Worldtree recall, nevermore clustering, -Open WebUI RAG. +**Load-shared with [nh3-ml1](../nh3-ml1/README.md) since 2026-09-26 (Prime).** The +gateway splits `qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` across both +TEI hosts, which are parity-verified as indistinguishable. Losing either host no +longer takes fleet embeddings down; the failover test results are in the nh3-ml1 +README. The reward seat (`/scalar-judge`) is still esh-ml1 only. | | | |---|---| diff --git a/servers/nh3-ml1/README.md b/servers/nh3-ml1/README.md index 4b633d4..816ebbd 100644 --- a/servers/nh3-ml1/README.md +++ b/servers/nh3-ml1/README.md @@ -6,9 +6,24 @@ reranking backend**, the twin of [esh-ml1](../esh-ml1/README.md): same card, sam driver, same TEI image and models. Built 2026-09-25, after the NH3 site visit turned Secure Boot off on nh3-pve. -⚠ **Not behind the gateway yet.** It serves on its own ports and is monitored, but -LiteLLM still routes `qwen3-embedding` and `reranker` to esh-ml1 alone. Routing is -Prime's call (recommendation: load-share; see below). +**Load-shared with esh-ml1 since 2026-09-26 (Prime).** The gateway splits +`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3` across both hosts at +random, one `simple-shuffle` deployment each. Measured with 40 + 40 requests: embed +12 esh / 28 nh3, rerank 22 / 18. + +**Failover, measured by stopping this host's TEI** (2026-09-26 0747–0753): +- Without failover settings: **7/40 embeds returned 500**. The router's default 2 + retries re-picked at random and hit the dead backend again. +- With `router_settings.enable_weighted_failover: true`, which only affects + multi-deployment groups: + - rerank **80/80** OK. + - embeds **38/40** in the first test (2 failures at outage onset) and **60/60** + across a 34 s sustained outage. + - Once TEI restarted, nh3 was back in rotation (15/30). +- Residual: in LiteLLM 1.97 the embedding path does not stamp the failed + deployment, so embeds rely on the router's cooldown rather than per-request + re-pick. Expect a couple of failed embeds when a backend first dies. n = 2 + outages. | | | |---|---| diff --git a/stacks/embed-rerank/README.md b/stacks/embed-rerank/README.md index 6f1d304..d20004d 100644 --- a/stacks/embed-rerank/README.md +++ b/stacks/embed-rerank/README.md @@ -2,11 +2,11 @@ **The fleet's embedding + reranking service**, on **esh-ml1** (CT 110 on esh-pve, RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**. -Since 2026-09-25 it is the only backend behind the gateway names -`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3`. A second instance -runs on **nh3-ml1** (CT 109 on nh3-pve, the same card), parity-verified against -esh-ml1 on 2026-09-25 (`servers/nh3-ml1/README.md`). It is not in the gateway -yet; that is Prime's call. +It runs on **two hosts**, esh-ml1 and **nh3-ml1** (CT 109 on nh3-pve, the same +card). The two are parity-verified as indistinguishable, and since 2026-09-26 +(Prime) they are **load-shared** behind the gateway names `qwen3-embedding`, +`reranker` and `reranker-a3-bge-v2-m3`, with intra-group failover +(`servers/nh3-ml1/README.md` has the measurements). **TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or reranking seats go on TEI, not vLLM. Why, and the measurements behind it: diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index 698cef3..93a501c 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -443,15 +443,14 @@ model_list: # explicitly. Operator ruling 2026-08-23. --- # --- Qwen3 embeddings --- - # ONE backend: esh-ml1 (RTX 2000E Ada), served by Hugging Face TEI — the fleet's - # embed/rerank engine from 2026-09-25 (Prime). fv-ml1's vLLM seat was retired the - # same day. TEI's /v1/embeddings is OpenAI-compatible, so the provider stays - # `hosted_vllm/`. Same model as before; TEI vectors match the old vLLM ones - # (cosine median 0.999925, inside vLLM's own noise; old-index retrieval overlap - # 0.988 vs 0.986 self-noise), so no index needed re-embedding. - # ⚠ Single backend until the second RTX 2000 lands (Prime, 2026-09-25): if - # esh-ml1, esh-pve or the ESH mesh route is down, fleet embeddings are down. - # ⚠ Never point this name at a DIFFERENT embedding model — vectors are + # TWO backends, load-shared (simple-shuffle), since 2026-09-26 (Prime): esh-ml1 + # (ESH) and nh3-ml1 (NH3). Both are an RTX 2000E Ada running Hugging Face TEI + # 89-1.9.4 with the same model. Parity measured 2026-09-25: cosine esh vs nh3 + # min 0.999993 = each host's own floor; overlap@10 1.000 + # (servers/nh3-ml1/README.md). TEI's /v1/embeddings is OpenAI-compatible, so the + # provider is `hosted_vllm/`. TEI vectors match the retired vLLM seats (cosine + # median 0.999925), so no index needed re-embedding. + # ⚠ Never point this name at a DIFFERENT embedding model: vectors are # model-specific. Engine bake-off: docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md. - model_name: qwen3-embedding litellm_params: @@ -460,6 +459,13 @@ model_list: api_key: os.environ/VLLM_API_KEY model_info: mode: embedding + - model_name: qwen3-embedding + litellm_params: + model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B + api_base: http://10.100.50.80:8001/v1 + api_key: os.environ/VLLM_API_KEY + model_info: + mode: embedding # --- qwen3-reranker RETIRED 2026-08-20. It named Qwen3-Reranker-0.6B on :8002, # the incumbent the R43 bake-off replaced on 2026-08-06 after measuring it @@ -498,12 +504,13 @@ model_list: model_info: mode: chat # reranker → generic capability name for rerank (bge-reranker-v2-m3 since the - # R43 cutover). ONE backend: TEI on esh-ml1 (fv-ml1's vLLM seat retired - # 2026-09-25). ⚠ TEI's /rerank takes `texts`, not `documents`: this MUST use the + # R43 cutover). TWO backends, load-shared, since 2026-09-26 (Prime): TEI on + # esh-ml1 and nh3-ml1. esh vs nh3 max score diff 0.0014, inside esh's own + # 0.0020 run-to-run floor; top-1 identical. The DB-only alias + # `reranker-a3-bge-v2-m3` carries the same two deployments. + # ⚠ TEI's /rerank takes `texts`, not `documents`: this MUST use the # `huggingface/` provider. `hosted_vllm/` gets a 422 "missing field texts". # api_base has NO /v1. api_key is a placeholder; TEI runs without auth. - # Parity vs the retired vLLM seat: top-1/top-3 identical (129/130 top-1 over - # two runs); scores move up to 0.019, reshuffling only near-zero tail docs. - model_name: reranker litellm_params: model: huggingface/BAAI/bge-reranker-v2-m3 @@ -511,6 +518,13 @@ model_list: api_key: "none" model_info: mode: rerank + - model_name: reranker + litellm_params: + model: huggingface/BAAI/bge-reranker-v2-m3 + api_base: http://10.100.50.80:8013 + api_key: "none" + model_info: + mode: rerank # --- lfm25-vl-3b → LiquidAI LFM2.5-VL-3B (Q5_K_M + mmproj Q8_0), llama.cpp server on # nh3-ml1 :8030 (stacks/lfm-vl-seat, 2026-09-26). Image understanding for @@ -1077,6 +1091,18 @@ general_settings: forward_headers: true include_subpath: true +router_settings: + # Intra-group failover for simple-shuffle. When one deployment fails, the SAME + # request re-picks among the group's other healthy deployments, excluding the + # failed one. Without it, a dead backend in a 2-deployment group kept failing + # a share of requests: the router's default 2 retries re-picked at random and + # hit the dead one again. Measured 2026-09-26 with nh3-ml1 TEI stopped: 7/40 + # embeds returned 500. + # Affects ONLY multi-deployment groups (qwen3-embedding, reranker, + # reranker-a3-bge-v2-m3 as of 2026-09-26). A single-deployment group has no + # other deployment to re-pick, so its behaviour is unchanged. + enable_weighted_failover: true + litellm_settings: # vLLM rejects some OpenAI params other backends accept; drop silently # rather than 400 the caller. diff --git a/stacks/uptimekuma/monitors.yaml b/stacks/uptimekuma/monitors.yaml index 572f86d..5ebbcfc 100644 --- a/stacks/uptimekuma/monitors.yaml +++ b/stacks/uptimekuma/monitors.yaml @@ -110,11 +110,11 @@ monitors: - name: Embed — Qwen3 0.6B (TEI, nh3-ml1) url: http://10.100.50.80:8001/health - description: second `qwen3-embedding` backend (nh3-ml1) + description: gateway `qwen3-embedding` backend (nh3-ml1) - name: Rerank — bge-v2-m3 (TEI, nh3-ml1) url: http://10.100.50.80:8013/health - description: second `reranker` backend (nh3-ml1) + description: gateway `reranker` backend (nh3-ml1) - name: talk url: https://talk.nh3.phasefinal.com:8092/