From 7bdac8087887f9c745e059172024db93cd088755 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 25 Sep 2026 08:30:53 -0700 Subject: [PATCH] feat(embed-rerank): TEI is the fleet embed/rerank engine; esh-ml1 sole backend; retire fv-ml1 seats Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet from now on; fv-ml1 retires both once esh-ml1 is up. - stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports 8001/8013, fail-closed truncation (--auto-truncate false; embed --max-batch-tokens 32768). - litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address); reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target. - Verified via the gateway against the retiring fv-ml1 seats: embed cosine median 0.999927 (n=203); rerank top-1/top-3 29/30. - stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged. - Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md; CLAUDE.md gains the TEI convention. --- CLAUDE.md | 6 +- docs/fleettools/litellm.md | 4 +- .../pfi/embed-rerank-tei-vs-vllm-bakeoff.md | 18 +- docs/pfi/fv-ml1-gpu-seat-inventory.md | 6 +- docs/pfi/reranker-selection-ledger.md | 9 + .../2026-09-25-tei-fleet-embed-rerank.md | 43 +++++ persistent-memory.md | 14 +- playbooks/esh-ml1-lxc.yaml | 2 +- servers/esh-ml1/README.md | 55 +++++-- stacks/embed-rerank/.env.example | 26 +-- stacks/embed-rerank/README.md | 49 ++++-- stacks/embed-rerank/compose.yaml | 139 ++++++++-------- stacks/litellm/conf/config.yaml | 58 +++---- stacks/tei-bakeoff/compose.yaml | 81 --------- stacks/vllm/.env.example | 18 +- stacks/vllm/README.md | 6 + stacks/vllm/compose.yaml | 155 ++---------------- 17 files changed, 271 insertions(+), 418 deletions(-) rename stacks/tei-bakeoff/README.md => docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md (87%) create mode 100644 persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md delete mode 100644 stacks/tei-bakeoff/compose.yaml diff --git a/CLAUDE.md b/CLAUDE.md index 75e7609..3ab04a2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -331,6 +331,10 @@ Observed and standardized across servers: the new content serves immediately — it is the route table that is frozen at start, not the file contents. Reference it as `icon: /images/`. - **Healthchecks** on services that expose HTTP +- **Embedding + reranking models are served with Hugging Face TEI, not vLLM** (Prime, + 2026-09-25). The fleet service is `stacks/embed-rerank` on esh-ml1. Why, plus the + parity bar any engine or model change must clear: + `docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`. ## Servers @@ -362,7 +366,7 @@ Observed and standardized across servers: | esh-pve | 10.0.250.35 | ESH home lab | Proxmox VE hypervisor | `servers/esh-pve/README.md` | | esh-pve-nas | 10.0.50.55 | ESH home lab | Proxmox VE hypervisor (storage / media) | `servers/esh-pve-nas/README.md` | | esh-vm-db | 10.0.50.60 | ESH home lab | DB VM — PostgreSQL (paperless-ng) + MongoDB; bare-metal VM, no Docker | `servers/esh-vm-db/README.md` | -| esh-ml1 | 10.0.50.80 | ESH home lab | GPU LXC (CT 110 on esh-pve, RTX 2000E Ada 16 GB, host-side NVIDIA driver) — embedding + reranking, LiteLLM order-2 failover behind fv-ml1 | `servers/esh-ml1/README.md` | +| esh-ml1 | 10.0.50.80 | ESH home lab | GPU LXC (CT 110 on esh-pve, RTX 2000E Ada 16 GB, host-side NVIDIA driver) — **the fleet's embedding + reranking service** (TEI), sole gateway backend since 2026-09-25 | `servers/esh-ml1/README.md` | **Placement rules:** - GPU-required stacks → `fv-ml1` (primary, Fountain Valley) or `irv-ml1` diff --git a/docs/fleettools/litellm.md b/docs/fleettools/litellm.md index a39d739..6a5eb38 100644 --- a/docs/fleettools/litellm.md +++ b/docs/fleettools/litellm.md @@ -33,8 +33,8 @@ on someone else's tab" matters, get a project-scoped key. `summarizer` / `summarizer-large` / `classifier` — the always-available local seat for summarization, naming/labeling, classification and triage. Essentially free compute; reach for it before spending premium API tokens on work a small local model handles. -Embeddings and rerankers (`qwen3-embedding`, `qwen3-reranker`, `reranker`) are local -and free too. Anything named `*-frontier`, `glm-*` or `kimi-*` is **PAID**. +Embeddings and reranking (`qwen3-embedding`, `reranker`) are local and free too — +served by TEI on esh-ml1 since 2026-09-25 (`qwen3-reranker` was retired 2026-08-20). Anything named `*-frontier`, `glm-*` or `kimi-*` is **PAID**. ## Traps diff --git a/stacks/tei-bakeoff/README.md b/docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md similarity index 87% rename from stacks/tei-bakeoff/README.md rename to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md index fbd45ba..f10683a 100644 --- a/stacks/tei-bakeoff/README.md +++ b/docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md @@ -1,10 +1,16 @@ -# tei-bakeoff — TEI vs vLLM for embedding + reranking on esh-ml1 +# Embed/rerank engine bake-off — TEI vs vLLM on esh-ml1 (2026-09-25) Prime, 2026-09-25: *"run the TEI bake-off."* Hugging Face **Text Embeddings -Inference 1.9.4** (`89-1.9.4`, the Ada build) serving the same two models as -[`embed-rerank`](../embed-rerank/) (vLLM v0.24.0), side by side on the same RTX -2000E Ada. Not wired into the gateway. This stack either becomes the -embed-rerank stack or is torn down. +Inference 1.9.4** (`89-1.9.4`, the Ada build) served the same two models as the +then-vLLM `embed-rerank` stack (vLLM v0.24.0), side by side on the same RTX 2000E +Ada, as a temporary `tei-bakeoff` stack. + +**Outcome (Prime, 2026-09-25): "TEI is embed/reranker server for esh-ml1 and the +FLEET in general, in future."** The same day TEI replaced vLLM in +[`stacks/embed-rerank`](../../stacks/embed-rerank/) on ports 8001/8013. The +gateway was repointed with esh-ml1 as the sole backend, and fv-ml1's +`vllm-embed` + `vllm-rerank-a3` seats were retired, freeing ~6.1 GB on fv-ml1 +GPU 1. The bake-off stack was removed; this file is its record. ## Results (2026-09-25 0744–0805 PT) @@ -58,7 +64,7 @@ On-box, 2 interleaved runs × 3 reps each, medians: TEI ran its fused `FlashQwen3` path. `--max-batch-tokens 32768` did not change the novel time (39–40 s), so it stays at the default. **Through the gateway these gaps mostly vanish**: LiteLLM is the bottleneck there (see -`servers/esh-ml1/README.md`). +[`servers/esh-ml1/README.md`](../../servers/esh-ml1/README.md)). ### Footprint — TEI is much lighter diff --git a/docs/pfi/fv-ml1-gpu-seat-inventory.md b/docs/pfi/fv-ml1-gpu-seat-inventory.md index 3c721ed..fee363b 100644 --- a/docs/pfi/fv-ml1-gpu-seat-inventory.md +++ b/docs/pfi/fv-ml1-gpu-seat-inventory.md @@ -124,8 +124,6 @@ context. | `gen-small-reasoning` | 8026 | | `image-judge` | 8022 | | `qwen-image-bench` | 8022 | -| `qwen3-embedding` | 8001 | -| `reranker` | 8013 | | `sec` | 8025 | | `sec-reasoning` | 8025 | | `summarizer` | 8026 | @@ -138,3 +136,7 @@ context. *Regenerate with `scripts/seat-inventory.py` after ANY seat change — model swap, quant change, context or utilization edit, or speculative-decoding change. Run `--check` in CI to catch a stale document.* + +> **2026-09-25:** `qwen3-embedding` (:8001) and `reranker` (:8013) LEFT fv-ml1 — +> their vLLM seats were retired and the names now route to TEI on esh-ml1 +> (`stacks/embed-rerank`). ~6.1 GB freed on GPU 1. diff --git a/docs/pfi/reranker-selection-ledger.md b/docs/pfi/reranker-selection-ledger.md index 2015f21..af4c2eb 100644 --- a/docs/pfi/reranker-selection-ledger.md +++ b/docs/pfi/reranker-selection-ledger.md @@ -243,3 +243,12 @@ on fv-ml1; `/model/delete` the two aliases (DB-persisted); weights retained in H form, not load-balancing: fv-ml1 serves every request while it is up. Parity: |score| FV-vs-ESH max 0.000145 vs FV-vs-FV floor 0.000181, identical ranking over 11 documents. See `servers/esh-ml1/README.md`. + +## 2026-09-25 — the reranker moved to TEI on esh-ml1 (Prime) + +- TEI adopted as the fleet embed/rerank engine after a bake-off + (`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`). `vllm-rerank-a3` on fv-ml1 was + **retired**; `reranker` and the DB alias `reranker-a3-bge-v2-m3` now both route to + `tei-rerank` on esh-ml1 :8013 via LiteLLM's `huggingface/` provider. Same model + (bge-reranker-v2-m3): top-1/top-3 identical to the retired seat, and scores move + up to 0.019 in the near-zero tail. diff --git a/persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md b/persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md new file mode 100644 index 0000000..f2b7b6f --- /dev/null +++ b/persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md @@ -0,0 +1,43 @@ +# `[2026-09-25]` TEI adopted as the fleet embed/rerank engine; esh-ml1 is the sole backend; fv-ml1 seats retired + +**Prime's rulings (2026-09-25):** +- Utility seats move off fv-ml1 to a utility host like esh-ml1, accepting slower + service; the Blackwells are kept for higher-requirement work. +- **parakeet STAYS on fv-ml1** (real-time chat path, every ms counts). +- **"TEI is embed/reranker server for esh-ml1 and the FLEET in general, in + future."** +- fv-ml1 retires both embed + rerank once esh-ml1 is up. A **second RTX 2000 is + coming to the fleet soon**; until then embed/rerank has one backend (accepted). + +**Evidence chain:** speed A/B vs fv-ml1 (single queries a wash, bulk 3–10× +slower on the Ada); whole-novel test (*The Stand*, 12,814 paragraphs) through +the gateway ~12 min one per request on either site, because the gateway is the +bottleneck; TEI bake-off (`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`): +matches vLLM vectors, no re-embed, slower on bulk, much lighter. Dvalin +(Grok research peer) reviewed: agree with caveats. Two caveats were actioned: +fail-closed truncation (TEI's default silently embedded a 16,384-token prefix of a +40k input and returned 200) and nevermore's 0.3 rerank threshold (0 flips in 2,000 +scores; low sensitivity, since few scores sit near 0.3). + +**Cut-over, 0820–0830 PT:** +1. esh-ml1: vLLM pair down, TEI `tei-embed` :8001 + `tei-rerank` :8013 up (same + ports). The old `.env` is kept as `.env.vllm-retired-2026-09-25`. The vLLM image + (~30 GB) is kept in case the reward classifier (vLLM-only) moves here. +2. Gateway: `qwen3-embedding` → esh-ml1 only (`hosted_vllm/`, unchanged + address); `reranker` → **`huggingface/`** provider (hosted_vllm 422s on TEI's + `texts` body); `order` fields removed; LiteLLM restarted (~58 s). The DB + alias `reranker-a3-bge-v2-m3` was PATCHed to the same TEI target. +3. Verified through the gateway against the still-running FV seats: embed + cosine median 0.999927 / min 0.999881 (n=203); rerank top-1/top-3 29/30 (2 of + 30 lists had a duplicate-source tie; the disagreeing query wasn't logged). +4. **No direct (non-gateway) consumers existed**: per day, fv-ml1 seat POSTs = + gateway spend-log rows exactly (09-20→09-24). After the repoint, the only FV hits + were my own verification calls. +5. fv-ml1: `vllm-embed` + `vllm-rerank-a3` stopped and removed; `stacks/vllm` + compose + `.env.example` updated (reward + coder untouched, 10–11 days up); + GPU 1 93,086 → 86,956 MiB (~6.1 GB freed). + +**Open:** esh-ml1 is now load-bearing with no monitoring (no Beszel, Kuma or +Homepage). Next: "what else moves". vllm-reward (9.2 GB, vLLM-only) is the +candidate and fits beside TEI's ~2.6 GB on 16 GB; vllm-coder (6.3 GB) is +latency-sensitive. diff --git a/persistent-memory.md b/persistent-memory.md index f5b4573..0d78b81 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -117,14 +117,13 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-09-24 ~2245 PT._ -### ⭐ esh-ml1: TEI vs vLLM decision pending (Prime), then "what else moves" +### ⭐ TEI is the fleet embed/rerank engine; esh-ml1 is the SOLE backend (2026-09-25) -Prime's direction 2026-09-25: move the utility seats off fv-ml1 to esh-ml1 -(accepting slower); **parakeet STAYS on fv-ml1** (real-time chat path). TEI -bake-off done — results `stacks/tei-bakeoff/README.md`; tei-embed :8081 + -tei-rerank :8083 running beside vLLM, NOT gateway-wired. Next: Prime picks the -engine, then the rest of the move (reward model is the open candidate; it needs -vLLM — TEI can't serve a Llama classifier). +Cut-over done 0830 PT: gateway `qwen3-embedding` + `reranker` (+ DB alias a3) → +TEI on esh-ml1; fv-ml1's two vLLM seats retired. ⚠ **Single backend until the +second RTX 2000 arrives**, and **esh-ml1 has no monitoring yet**: wire Beszel + +Kuma next. Next conversation with Prime: "what else moves" (vllm-reward is the +candidate; **parakeet stays**). → Recent decisions. ### ✅ esh-ml1 built (2026-09-24) — two follow-ups @@ -170,6 +169,7 @@ a shutdown (stays off by design). Outcomes and revert in ## Recent decisions +- `[2026-09-25]` **TEI adopted as the fleet embed/rerank engine (Prime); esh-ml1 is the sole gateway backend for `qwen3-embedding` + `reranker`; fv-ml1's vLLM embed/rerank seats retired (~6.1 GB freed); parakeet stays on fv-ml1.** → `persistent-memory.d/2026-09-25-tei-fleet-embed-rerank.md` - `[2026-09-24]` **esh-ml1 built — RTX 2000E Ada as CT 110 on esh-pve (host driver 580.178.04 DKMS, loaded live, no reboot), vLLM embed+rerank at measured parity with fv-ml1, wired as LiteLLM `order: 2` failover; dead DB alias `reranker-a3-bge-v2-m3` (pre-relocation IP) repaired.** → `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md` - `[2026-09-24]` **esh-pve: VM 102 retired, a 14-day hung VFIO process found and cleared, T400 → RTX 2000 Ada; Prime chose LXC + host driver for embed/rerank** (implementation deferred to next session, tracked here + `servers/esh-pve/README.md`). → `persistent-memory.d/2026-09-24-esh-pve-vm102-retired-gpu-swap.md` - `[2026-09-24]` **pfi-gx10 AC-restore UEFI patch applied but NOT validated; box OFF until Prime's AC pull 2026-09-25** (tracking: `servers/pfi-gx10/README.md`, `e5197a3`). → `persistent-memory.d/2026-09-24-gx10-ac-restore-patch.md` diff --git a/playbooks/esh-ml1-lxc.yaml b/playbooks/esh-ml1-lxc.yaml index 5e19129..1055b97 100644 --- a/playbooks/esh-ml1-lxc.yaml +++ b/playbooks/esh-ml1-lxc.yaml @@ -59,7 +59,7 @@ steps: --onboot 1 --startup order=30,up=10 \ --dev0 /dev/nvidia0 --dev1 /dev/nvidiactl \ --dev2 /dev/nvidia-uvm --dev3 /dev/nvidia-uvm-tools \ - --description "esh-ml1 — RTX 2000E Ada embedding + reranking (vLLM). Built by eshpfi playbooks/esh-ml1-lxc.yaml; stack stacks/embed-rerank. NOT in the vzdump job on purpose (rebuildable)." + --description "esh-ml1 — RTX 2000E Ada: the fleet embedding + reranking service (TEI). Built by eshpfi playbooks/esh-ml1-lxc.yaml; stack stacks/embed-rerank. NOT in the vzdump job on purpose (rebuildable)." when: "! pct status {{ ctid }} >/dev/null 2>&1" - name: Start the container diff --git a/servers/esh-ml1/README.md b/servers/esh-ml1/README.md index d7260a7..785ed3e 100644 --- a/servers/esh-ml1/README.md +++ b/servers/esh-ml1/README.md @@ -1,8 +1,14 @@ # esh-ml1 GPU LXC for the ESH home lab: **CT 110 on esh-pve**, holding the **NVIDIA RTX -2000E Ada** (16 GB, 50 W, `01:00.0`). It serves the fleet's embedding and -reranking models locally at ESH. Built 2026-09-24. +2000E Ada** (16 GB, 50 W, `01:00.0`). **It is the fleet's embedding and +reranking service** — the only backend behind the gateway's `qwen3-embedding`, +`reranker` and `reranker-a3-bge-v2-m3` since 2026-09-25. Built 2026-09-24. + +⚠ **Single backend until the second RTX 2000 arrives** (Prime, 2026-09-25). If +esh-ml1, esh-pve, or ESH's mesh route (esh-scale, CT 108) is down, fleet +embeddings and reranking are down: Worldtree recall, nevermore clustering, Open +WebUI RAG. | | | |---|---| @@ -16,17 +22,31 @@ reranking models locally at ESH. Built 2026-09-24. ## What it serves -`stacks/embed-rerank` (`/opt/docker/compose/embed-rerank`): +`stacks/embed-rerank` (`/opt/docker/compose/embed-rerank`), **Hugging Face TEI +1.9.4** — the fleet's embed/rerank engine from 2026-09-25 (Prime). It replaced +vLLM here after a bake-off; see +[`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`](../../docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md). -| container | model | port | gateway name | +| container | model | port | gateway name (LiteLLM provider) | |---|---|---|---| -| `vllm-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 | `qwen3-embedding` (order 2) | -| `vllm-rerank-bge` | `BAAI/bge-reranker-v2-m3` | 8013 | `reranker` (order 2) | +| `tei-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 | `qwen3-embedding` (`hosted_vllm/`, `/v1`) | +| `tei-rerank` | `BAAI/bge-reranker-v2-m3` | 8013 | `reranker`, `reranker-a3-bge-v2-m3` (`huggingface/`, no `/v1`) | -The same models, vLLM version (`v0.24.0`, digest `251eba5cc7c1`) and flags as -fv-ml1's `vllm` stack, so the two sites are interchangeable. In LiteLLM they -are the **order-2 failover** behind fv-ml1: fv-ml1 serves every request while -it is up. +VRAM ~2.6 GB for both; the rest of the 16 GB is free for further utility seats. +The vLLM image (`vllm/vllm-openai:v0.24.0`, ~30 GB) is still on disk: kept in +case a vLLM-only seat (the reward classifier) moves here. + +**Cut-over verified 2026-09-25** through the gateway against fv-ml1's vLLM seats +just before they were retired: embed cosine median 0.999927 / min 0.999881 +(n=203); rerank top-1 and top-3 agreement 29/30. Two of the 30 lists contained +the source paragraph twice, which makes #1 a tie either engine may break either +way. The disagreeing query wasn't logged, so that is the likely cause, not a +proven one. + +### History — the vLLM era (2026-09-24 → 25) + +These measurements are of vLLM on this card vs vLLM on fv-ml1, and are kept for +reference. **Parity, measured 2026-09-24** (11 texts incl. CJK, code, a 6k-char passage; 2 runs per site): @@ -62,7 +82,7 @@ on small-query latency on fv-ml1, so the small-query rows cannot rank the sites. Positive control: 2k-token inputs were slower than 512-token ones on both sites. One fv-ml1 rep of the gateway-vantage rerank burst stalled to 1.0 req/s (64 requests in ~64 s); it did not recur in 11 other reps — unexplained, n=1. -**Consequence: keep esh-ml1 as failover, don't load-share** — sharing would +**Consequence at the time: keep esh-ml1 as failover, don't load-share** (superseded 2026-09-25 by the move) — sharing would roughly double rerank latency for half the fleet's calls. **Whole-novel embedding, measured 2026-09-25 0653–0717 PT** — *The Stand* @@ -113,10 +133,15 @@ build and this CT will fail to start at the next boot. ## Not yet wired +⚠ **esh-ml1 became load-bearing on 2026-09-25, so these are no longer +optional.** Nothing alerts if it dies today. The gateway will just start +returning errors for `qwen3-embedding` and `reranker`. + +- **Beszel**: no agent yet. +- **Uptime Kuma**: no check on `:8001/health` / `:8013/health` yet. - **Homepage**: the compose carries labels, but esh-ml1 is not in `stacks/homepage/conf/docker.yaml` (it would need dockerd on tcp/2375 like the other hosts). -- **Beszel**: no agent yet. -- **ESH consumers** (Open WebUI RAG, Paperless) still go through the gateway - at ana-docker, so they do not survive a mesh outage. See persistent-memory - for the open decision. +- **Path:** every consumer reaches esh-ml1 through the gateway at ana-docker, so + ESH callers hairpin ESH → Anaheim → ESH over the mesh. A mesh outage cuts + every consumer off, the ones at ESH included. diff --git a/stacks/embed-rerank/.env.example b/stacks/embed-rerank/.env.example index ca3070b..e004530 100644 --- a/stacks/embed-rerank/.env.example +++ b/stacks/embed-rerank/.env.example @@ -1,29 +1,19 @@ -# embed-rerank tunables (esh-ml1). Copy to `.env` on the server. -# -# Everything model-shaped here MUST match fv-ml1's stacks/vllm .env: the same -# models, the same vLLM version, the same max-model-len. A drift in the -# embedding model or its version makes this seat's vectors incompatible with -# every index built against fv-ml1's. +# embed-rerank tunables (esh-ml1, TEI). Copy to `.env` on the server. -# PINNED to fv-ml1's version. Bump both sites together. -VLLM_VERSION=v0.24.0 +# TEI image tag. `89-` = the Ada Lovelace (sm_89) build; a different GPU +# generation needs a different prefix (see the TEI README's image table). +# PINNED to an exact release. +TEI_TAG=89-1.9.4 -# Same host ports as fv-ml1 (container listens on 8000). +# Kept from the vLLM era so the embedding gateway entry did not have to change. EMBED_PORT=8001 RERANK_PORT=8013 +# ⚠ Changing EMBED_MODEL invalidates every index built on it. EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B RERANK_MODEL=BAAI/bge-reranker-v2-m3 -# Fractions of the RTX 2000E Ada's 16,380 MiB. fv-ml1 runs 0.03 of a 96 GB -# card (~2.9 GB each); 0.20 here is ~3.2 GB each — the same budget plus a -# little, leaving ~9.5 GB free. -EMBED_GPU_MEM_UTIL=0.20 -RERANK_GPU_MEM_UTIL=0.20 -EMBED_MAX_MODEL_LEN=8192 -RERANK_MAX_MODEL_LEN=8192 +MAX_CLIENT_BATCH_SIZE=128 -# fv-ml1's seats run with no API key (LiteLLM fronts them); match that. -API_KEY= # Both models are public; no token needed. HF_TOKEN= diff --git a/stacks/embed-rerank/README.md b/stacks/embed-rerank/README.md index 1329869..687c1a7 100644 --- a/stacks/embed-rerank/README.md +++ b/stacks/embed-rerank/README.md @@ -1,18 +1,40 @@ # embed-rerank -The fleet's embedding + reranking models on **esh-ml1** (CT 110 on esh-pve, -RTX 2000E Ada). The second site for `qwen3-embedding` and `reranker`; fv-ml1's -[`vllm`](../vllm/) stack is the first. +**The fleet's embedding + reranking service**, on **esh-ml1** (CT 110 on esh-pve, +RTX 2000E Ada), served by **Hugging Face Text Embeddings Inference (TEI)**. +Since 2026-09-25 it is the only backend behind the gateway names +`qwen3-embedding`, `reranker` and `reranker-a3-bge-v2-m3`. -| container | model | port | -|---|---|---| -| `vllm-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 | -| `vllm-rerank-bge` | `BAAI/bge-reranker-v2-m3` | 8013 | +**TEI is the fleet's embed/rerank engine** (Prime, 2026-09-25). New embedding or +reranking seats go on TEI, not vLLM. Why, and the measurements behind it: +[`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`](../../docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md). -**Keep it in lockstep with `stacks/vllm`:** same models, same `VLLM_VERSION`, -same `--max-model-len`. The LiteLLM groups treat the two sites as one model, -so any drift in the embedding model or its version silently mixes -incompatible vectors into consumers' indexes. +| container | model | port | endpoint | +|---|---|---|---| +| `tei-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 | `/v1/embeddings` (OpenAI), `/embed` | +| `tei-rerank` | `BAAI/bge-reranker-v2-m3` | 8013 | `/rerank` — body `{"query", "texts"}` | + +## Gateway wiring (stacks/litellm/conf/config.yaml) + +- `qwen3-embedding` → `hosted_vllm/Qwen/Qwen3-Embedding-0.6B`, `api_base: http://10.0.50.80:8001/v1`. +- `reranker` → **`huggingface/`**`BAAI/bge-reranker-v2-m3`, `api_base: http://10.0.50.80:8013` (**no `/v1`**). + ⚠ `hosted_vllm/` sends `documents` and TEI answers **422 "missing field texts"**. +- `reranker-a3-bge-v2-m3` is a DB-only alias (not in config.yaml) with the same target. + +## ⚠ Invariants + +- **Changing `EMBED_MODEL` invalidates every index built on it** (Worldtree, + nevermore, Open WebUI). An engine change is allowed only with a parity + measurement against the current vectors. The TEI switch measured cosine median + 0.999925 and old-index retrieval overlap 0.988. +- **Truncation is fail-closed** (`--auto-truncate false`). TEI's default silently + embeds a prefix of an over-length input and returns 200. Now: embed rejects + more than 32,768 tokens and rerank more than 8,192, both with 422. +- **The image tag is GPU-generation specific**: `89-` is Ada. A different card + needs a different prefix (TEI README image table). +- nevermore thresholds rerank scores at 0.3. TEI scores differ from the old vLLM + seat by up to 0.019, which produced 0 flips in 2,000 scores. Recheck + thresholding consumers after any engine or version change. ## Deploy @@ -30,8 +52,5 @@ Host prerequisites (driver, LXC, docker, toolkit) are in curl -s http://10.0.50.80:8001/v1/embeddings -H 'content-type: application/json' \ -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"hello"}' | jq '.data[0].embedding | length' # 1024 curl -s http://10.0.50.80:8013/rerank -H 'content-type: application/json' \ - -d '{"model":"BAAI/bge-reranker-v2-m3","query":"cat","documents":["a cat","a car"]}' | jq '.results[0]' + -d '{"query":"cat","texts":["a cat","a car"]}' # index 0 scores ~0.9986 ``` - -Parity against fv-ml1 is recorded in the host README; re-measure after any -version bump on either side. diff --git a/stacks/embed-rerank/compose.yaml b/stacks/embed-rerank/compose.yaml index 29db57a..3e4da5f 100644 --- a/stacks/embed-rerank/compose.yaml +++ b/stacks/embed-rerank/compose.yaml @@ -1,63 +1,66 @@ -# embed-rerank — the fleet's embedding + reranking models, served locally at ESH -# on esh-ml1 (CT 110 on esh-pve, RTX 2000E Ada, 16 GB). +# embed-rerank — THE fleet's embedding + reranking service, on esh-ml1 (CT 110 on +# esh-pve, RTX 2000E Ada, 16 GB). Served by Hugging Face Text Embeddings +# Inference (TEI). # -# WHY: Prime's decision 2026-09-24 — the Ada card offloads embedding and -# reranking so ESH consumers (Open WebUI RAG, Paperless) are not tied to one -# seat on fv-ml1. It serves the SAME models as fv-ml1's `vllm` stack, with the -# SAME vLLM version and flags, because embedding vectors are model-specific: -# a different embedding model here would silently poison every index built -# against fv-ml1's. Mirror stacks/vllm/compose.yaml when that changes. +# Prime, 2026-09-25: "TEI is embed/reranker server for esh-ml1 and the FLEET in +# general, in future." It replaced vLLM here the same day, after a side-by-side +# bake-off on this card (docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md). TEI +# gives the same vectors as the old vLLM seats (no re-embedding), but it is +# ~1.3x slower on bulk work on this card. It is much lighter (2.6 GB VRAM for +# both, 8 GB image, ~4 s restart). fv-ml1's vLLM embed/rerank seats were +# retired after this went live. # -# vllm-embed Qwen/Qwen3-Embedding-0.6B → /v1/embeddings :8001 -# vllm-rerank-bge BAAI/bge-reranker-v2-m3 → /rerank, /score :8013 +# tei-embed Qwen/Qwen3-Embedding-0.6B → /v1/embeddings (OpenAI), /embed :8001 +# tei-rerank BAAI/bge-reranker-v2-m3 → /rerank (body: query + texts) :8013 # -# Ports match fv-ml1 on purpose, so a LiteLLM deployment for either site -# differs only in the host part of api_base. +# Ports kept from the vLLM era (and fv-ml1), so the embedding gateway entry did +# not change address. ⚠ The RERANK gateway entry must use LiteLLM's +# `huggingface/` provider: `hosted_vllm/` sends `documents` and TEI answers 422 +# "missing field texts". # -# `vllm-rerank-bge`, not fv-ml1's `vllm-rerank-a3`: that name carries R43 -# bake-off provenance for THAT container; and plain `vllm-rerank` was the -# retired Qwen3-Reranker that measured harmful. Name the model instead. +# ⚠ Embedding vectors are model-specific. Never change EMBED_MODEL without a +# re-embedding plan for every index built on it (Worldtree, nevermore, Open WebUI). # -# NO `tnet`/traefik-net: esh-ml1 runs no traefik, and an external network -# that does not exist would stop the stack from starting. Consumers reach the -# published ports directly. +# FAIL-CLOSED truncation (--auto-truncate false). TEI's default silently +# embedded the first 16,384 tokens of a ~40k-token input and returned 200. +# Turning it off requires --max-batch-tokens >= the model's max input (32,768 +# for Qwen3-Embedding), or TEI refuses to start. # -# Host setup (driver, LXC, docker, toolkit): playbooks/esh-pve-nvidia-host.yaml -# then playbooks/esh-ml1-lxc.yaml. Tunables live in .env — edit that, not this. +# NO `tnet`/traefik-net: esh-ml1 runs no traefik; consumers reach the published +# ports, and in practice only the LiteLLM gateway does (verified from the seats' +# logs 2026-09-25: every request matched a gateway spend-log row). +# +# Host setup: playbooks/esh-pve-nvidia-host.yaml, then playbooks/esh-ml1-lxc.yaml. +# Tunables live in .env. name: embed-rerank services: - vllm-embed: - image: vllm/vllm-openai:${VLLM_VERSION} - container_name: vllm-embed + tei-embed: + image: ghcr.io/huggingface/text-embeddings-inference:${TEI_TAG} + container_name: tei-embed restart: unless-stopped - ipc: host ports: - - "${EMBED_PORT}:8000" + - "${EMBED_PORT}:80" volumes: - - /opt/aimodels/huggingface:/hfcache + - /opt/aimodels/tei-cache:/data environment: - - HF_HOME=/hfcache - - HF_HUB_CACHE=/hfcache/hub - - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-} - - VLLM_API_KEY=${API_KEY:-} + - HF_TOKEN=${HF_TOKEN:-} command: + - --model-id - ${EMBED_MODEL} - --served-model-name - ${EMBED_MODEL} - - --runner - - pooling - - --host - - 0.0.0.0 - - --port - - "8000" - - --gpu-memory-utilization - - ${EMBED_GPU_MEM_UTIL} - - --max-model-len - - ${EMBED_MAX_MODEL_LEN} + # TEI on CUDA is float16-only; parity vs the bf16 vLLM vectors was measured. - --dtype - - auto + - float16 + # Default 32; vLLM had no cap and callers batch 64. + - --max-client-batch-size + - "${MAX_CLIENT_BATCH_SIZE}" + - --auto-truncate + - "false" + - --max-batch-tokens + - "32768" deploy: resources: reservations: @@ -66,49 +69,39 @@ services: device_ids: ["0"] capabilities: [gpu] healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + test: ["CMD", "curl", "-fsS", "http://localhost:80/health"] interval: 30s timeout: 10s retries: 3 - start_period: 180s + start_period: 60s labels: - homepage.group=AI - Eval & Retrieval - - homepage.name=vLLM Embed (esh-ml1) + - homepage.name=Embed — Qwen3 0.6B (TEI, esh-ml1) - homepage.icon=mdi-vector-arrange-below - - homepage.description=Qwen3 Embedding 0.6B via vLLM (esh-ml1, RTX 2000E Ada) + - homepage.description=Fleet embeddings (qwen3-embedding) via TEI on esh-ml1 - homepage.href=http://10.0.50.80:${EMBED_PORT}/docs - vllm-rerank-bge: - image: vllm/vllm-openai:${VLLM_VERSION} - container_name: vllm-rerank-bge + tei-rerank: + image: ghcr.io/huggingface/text-embeddings-inference:${TEI_TAG} + container_name: tei-rerank restart: unless-stopped - ipc: host ports: - - "${RERANK_PORT}:8000" + - "${RERANK_PORT}:80" volumes: - - /opt/aimodels/huggingface:/hfcache + - /opt/aimodels/tei-cache:/data environment: - - HF_HOME=/hfcache - - HF_HUB_CACHE=/hfcache/hub - - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-} - - VLLM_API_KEY=${API_KEY:-} + - HF_TOKEN=${HF_TOKEN:-} command: + - --model-id - ${RERANK_MODEL} - - --served-model-name - - ${RERANK_MODEL} - - --runner - - pooling - # bge-reranker-v2-m3 is natively a cross-encoder; no --hf-overrides. - - --host - - 0.0.0.0 - - --port - - "8000" - - --gpu-memory-utilization - - ${RERANK_GPU_MEM_UTIL} - - --max-model-len - - ${RERANK_MAX_MODEL_LEN} - --dtype - - auto + - float16 + - --max-client-batch-size + - "${MAX_CLIENT_BATCH_SIZE}" + # Fail-closed; bge-reranker-v2-m3's max input (8,192) fits the default + # max-batch-tokens (16,384). + - --auto-truncate + - "false" deploy: resources: reservations: @@ -117,14 +110,14 @@ services: device_ids: ["0"] capabilities: [gpu] healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + test: ["CMD", "curl", "-fsS", "http://localhost:80/health"] interval: 30s timeout: 10s retries: 3 - start_period: 180s + start_period: 60s labels: - homepage.group=AI - Eval & Retrieval - - homepage.name=vLLM Rerank bge-v2-m3 (esh-ml1) + - homepage.name=Rerank — bge-v2-m3 (TEI, esh-ml1) - homepage.icon=mdi-sort-variant - - homepage.description=bge-reranker-v2-m3 via vLLM (esh-ml1, RTX 2000E Ada) + - homepage.description=Fleet reranker (reranker) via TEI on esh-ml1 - homepage.href=http://10.0.50.80:${RERANK_PORT}/docs diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index c9a1e8d..f0a9a7a 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -7,8 +7,8 @@ # # Model-name → upstream mapping: # phi4-mini → vLLM :8004 (generative chat) -# qwen3-embedding → vLLM :8001 (/v1/embeddings) -# qwen3-reranker → vLLM :8002 (/rerank) +# qwen3-embedding → TEI on esh-ml1 :8001 (/v1/embeddings) +# reranker → TEI on esh-ml1 :8013 (/rerank, huggingface/ provider) # * (wildcard) → llama-swap :9292 (the swappable generative zoo) # # The wildcard fronts llama-swap so its whole model zoo logs through the @@ -443,31 +443,21 @@ model_list: # explicitly. Operator ruling 2026-08-23. --- # --- Qwen3 embeddings --- - # TWO deployments, one name, ordered: fv-ml1 serves (order 1); esh-ml1 takes - # over only when fv-ml1 fails (order 2 — LiteLLM's order-based fallback, v1.97 - # router.py). Same model, same vLLM version and flags at both sites, so the - # vectors are interchangeable: measured 2026-09-24, cosine FV-vs-ESH median - # 0.999908 (min 0.999772) over 11 texts, inside the FV-vs-FV self-noise floor - # (median 0.999927, min 0.999791); different-text negative control 0.07–0.38. - # ⚠ Failover cost, measured n=3 each: primary REFUSING → +0.15 s; primary - # HOST DOWN (no ARP) → ~18.7 s on EVERY call (no cooldown kicked in). Slow, - # not broken. ⚠ Never add a deployment here that serves a DIFFERENT embedding - # model — vectors are model-specific and a mixed group corrupts indexes. - # esh-ml1: stacks/embed-rerank, playbooks/esh-ml1-lxc.yaml. - - model_name: qwen3-embedding - litellm_params: - model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B - api_base: http://10.251.50.54:8001/v1 - api_key: os.environ/VLLM_API_KEY - order: 1 - model_info: - mode: embedding + # ONE backend: esh-ml1 (RTX 2000E Ada), served by Hugging Face TEI — the fleet's + # embed/rerank engine from 2026-09-25 (Prime). fv-ml1's vLLM seat was retired the + # same day. TEI's /v1/embeddings is OpenAI-compatible, so the provider stays + # `hosted_vllm/`. Same model as before; TEI vectors match the old vLLM ones + # (cosine median 0.999925, inside vLLM's own noise; old-index retrieval overlap + # 0.988 vs 0.986 self-noise), so no index needed re-embedding. + # ⚠ Single backend until the second RTX 2000 lands (Prime, 2026-09-25): if + # esh-ml1, esh-pve or the ESH mesh route is down, fleet embeddings are down. + # ⚠ Never point this name at a DIFFERENT embedding model — vectors are + # model-specific. Engine bake-off: docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md. - model_name: qwen3-embedding litellm_params: model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B api_base: http://10.0.50.80:8001/v1 api_key: os.environ/VLLM_API_KEY - order: 2 model_info: mode: embedding @@ -508,23 +498,17 @@ model_list: model_info: mode: chat # reranker → generic capability name for rerank (bge-reranker-v2-m3 since the - # R43 cutover). fv-ml1 order 1, esh-ml1 order 2 — same failover shape as - # qwen3-embedding above. Parity measured 2026-09-24: |score| FV-vs-ESH max - # 0.000145 vs FV-vs-FV floor 0.000181, identical ranking over 11 documents. + # R43 cutover). ONE backend: TEI on esh-ml1 (fv-ml1's vLLM seat retired + # 2026-09-25). ⚠ TEI's /rerank takes `texts`, not `documents`: this MUST use the + # `huggingface/` provider. `hosted_vllm/` gets a 422 "missing field texts". + # api_base has NO /v1. api_key is a placeholder; TEI runs without auth. + # Parity vs the retired vLLM seat: top-1/top-3 identical (129/130 top-1 over + # two runs); scores move up to 0.019, reshuffling only near-zero tail docs. - model_name: reranker litellm_params: - model: hosted_vllm/BAAI/bge-reranker-v2-m3 - api_base: http://10.251.50.54:8013/v1 - api_key: os.environ/VLLM_API_KEY - order: 1 - model_info: - mode: rerank - - model_name: reranker - litellm_params: - model: hosted_vllm/BAAI/bge-reranker-v2-m3 - api_base: http://10.0.50.80:8013/v1 - api_key: os.environ/VLLM_API_KEY - order: 2 + model: huggingface/BAAI/bge-reranker-v2-m3 + api_base: http://10.0.50.80:8013 + api_key: "none" model_info: mode: rerank diff --git a/stacks/tei-bakeoff/compose.yaml b/stacks/tei-bakeoff/compose.yaml deleted file mode 100644 index 87a68ad..0000000 --- a/stacks/tei-bakeoff/compose.yaml +++ /dev/null @@ -1,81 +0,0 @@ -# tei-bakeoff — Hugging Face Text Embeddings Inference (TEI) serving the SAME two -# models as stacks/embed-rerank (vLLM), side by side on esh-ml1, so the two -# engines can be compared on identical hardware. Prime, 2026-09-25: "run the -# TEI bake-off". Not wired into the gateway. Temporary: either this becomes the -# embed-rerank stack or it is torn down. -# -# tei-embed Qwen/Qwen3-Embedding-0.6B → /v1/embeddings, /embed :8081 -# tei-rerank BAAI/bge-reranker-v2-m3 → /rerank :8083 -# -# 89-* = the Ada Lovelace (sm_89) build. Pinned to an exact release. -# TEI on CUDA offers float16/float32 only (no bfloat16), while vLLM serves -# Qwen3-Embedding in bf16 — the parity measurement is what decides whether -# that matters. -# (--max-batch-tokens 32768 was tried on tei-embed: whole-novel time unchanged, -# 39.2–40.4 s vs 39.9–40.3 s, so it stays at the default.) -# --max-client-batch-size 128: the default is 32; vLLM has no such cap and the -# bake-off sends 64 per request. -# Separate HF cache from vLLM's, so neither engine can disturb the other's files. - -name: tei-bakeoff - -services: - tei-embed: - image: ghcr.io/huggingface/text-embeddings-inference:89-1.9.4 - container_name: tei-embed - restart: unless-stopped - ports: - - "8081:80" - volumes: - - /opt/aimodels/tei-cache:/data - command: - - --model-id - - Qwen/Qwen3-Embedding-0.6B - - --served-model-name - - Qwen/Qwen3-Embedding-0.6B - - --dtype - - float16 - - --max-client-batch-size - - "128" - # FAIL-CLOSED on over-length input (Dvalin review 2026-09-25; measured: - # TEI's default silently cut a ~40k-token input to 16,384 and returned - # 200, where vLLM returns 400). Truncation off requires max-batch-tokens - # >= the model's max input (Qwen3-Embedding: 32,768) or TEI refuses to start. - - --auto-truncate - - "false" - - --max-batch-tokens - - "32768" - deploy: - resources: - reservations: - devices: - - driver: nvidia - device_ids: ["0"] - capabilities: [gpu] - - tei-rerank: - image: ghcr.io/huggingface/text-embeddings-inference:89-1.9.4 - container_name: tei-rerank - restart: unless-stopped - ports: - - "8083:80" - volumes: - - /opt/aimodels/tei-cache:/data - command: - - --model-id - - BAAI/bge-reranker-v2-m3 - - --dtype - - float16 - - --max-client-batch-size - - "128" - # Fail-closed, as tei-embed. bge-reranker-v2-m3 max input (8,192) fits the - # default max-batch-tokens (16,384). - - --auto-truncate - - "false" - deploy: - resources: - reservations: - devices: - - driver: nvidia - device_ids: ["0"] - capabilities: [gpu] diff --git a/stacks/vllm/.env.example b/stacks/vllm/.env.example index 6126433..60177b4 100644 --- a/stacks/vllm/.env.example +++ b/stacks/vllm/.env.example @@ -12,23 +12,15 @@ VLLM_VERSION=v0.24.0 # Host ports (container always listens on 8000 internally) -EMBED_PORT=8001 -# 8013 — the reranker moved here 2026-08-20 when bge-v2-m3 (the R43 winner, which -# had been running as a throwaway `docker run` on this port) was promoted into -# this stack and the Qwen incumbent on :8002 was retired. -RERANK_PORT=8013 +# (EMBED_PORT 8001 / RERANK_PORT 8013 retired 2026-09-25 with their seats — +# embedding + reranking moved to TEI on esh-ml1, stacks/embed-rerank.) REWARD_PORT=8003 # GPU assignment — all services share this GPU # (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work in llama-swap) GPU_ID=1 -# Models — reference by full repo name in API requests -EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B -# bge-reranker-v2-m3 — the R43 bake-off winner, replacing Qwen3-Reranker-0.6B -# (measured HARMING 80/90 fleet queries). Multilingual cross-encoder; needs no -# --hf-overrides, unlike the Qwen reranker it displaced. -RERANK_MODEL=BAAI/bge-reranker-v2-m3 +# Models # Skywork is a local-path AWQ output, not from HF Hub. Bind-mounted into the # reward container at /local-models — see compose.yaml. No env var here for # the model path itself since it's hard-coded in the compose command. @@ -62,16 +54,12 @@ RERANK_MODEL=BAAI/bge-reranker-v2-m3 # each at 0.05 (mostly util-reservation waste); 0.03 (~3.6 GB) fits weights + # CUDA context with room, freeing ~4 GB back to granite. Recreate them ONE AT A # TIME — concurrent recreate races the memory-profiling assertion. -EMBED_GPU_MEM_UTIL=0.03 -RERANK_GPU_MEM_UTIL=0.03 REWARD_GPU_MEM_UTIL=0.10 # Context length caps — lower these if VRAM is tight. # Qwen3-Embedding supports up to 32k; reranker up to 32k. # Skywork capped at 16k server-side as defense-in-depth; JudgeClient also # enforces the cap at dispatch time per spec. -EMBED_MAX_MODEL_LEN=8192 -RERANK_MAX_MODEL_LEN=8192 REWARD_MAX_MODEL_LEN=16384 # Optional API key — leave blank for no auth (fine on the internal network). diff --git a/stacks/vllm/README.md b/stacks/vllm/README.md index 3e0111c..534b3ae 100644 --- a/stacks/vllm/README.md +++ b/stacks/vllm/README.md @@ -1,5 +1,11 @@ # vllm +> ⚠ **2026-09-25: embedding + reranking LEFT this stack.** `vllm-embed` and +> `vllm-rerank-a3` were retired; the fleet's embed/rerank now runs on **TEI on +> esh-ml1** (`stacks/embed-rerank`), and TEI is the fleet engine for those from +> now on (Prime). What remains here is `vllm-reward` and `vllm-coder` on fv-ml1. +> The history below predates that. + Multi-service vLLM stack on ana-ml2. Started as Qwen3 embedding + rerank (replacing the unmaintained Infinity stack); generalized to host any vLLM model on the box, currently three services: diff --git a/stacks/vllm/compose.yaml b/stacks/vllm/compose.yaml index 4b35046..17ab2e1 100644 --- a/stacks/vllm/compose.yaml +++ b/stacks/vllm/compose.yaml @@ -1,162 +1,27 @@ -# vLLM — Qwen3 Embedding + Reranker + Skywork Reward-V2 classifier. +# vLLM — utility seats on fv-ml1: Skywork Reward-V2 classifier + the coder FIM seat. # # Originally created to replace the unmaintained Infinity stack (embed + # rerank); generalized 2026-05-13 to host any vLLM-served model on fv-ml1, # starting with the Skywork-Reward-V2-Llama-3.1-8B reward classifier # (AWQ-quantized locally, served from /tank/aimodels/llm/). # -# vLLM runs one model per process, so this stack brings up three containers -# sharing a single GPU: -# -# vllm-embed — Qwen3-Embedding served as an OpenAI /v1/embeddings server -# vllm-rerank — Qwen3-Reranker served as a /rerank + /score server -# vllm-reward — Skywork-Reward-V2-Llama-3.1-8B-AWQ served as a /classify scorer -# -# The reranker is a causal-LM checkpoint; --hf-overrides re-maps it to -# Qwen3ForSequenceClassification so vLLM's reranking endpoints work and the -# model only emits two class logits (no/yes) instead of the full 151k vocab. +# ⚠ Embedding + reranking LEFT this stack 2026-09-25 (Prime): they now run on TEI +# on esh-ml1 (stacks/embed-rerank), and TEI is the fleet's embed/rerank engine +# from now on. Do not re-add them here. # # All tunables live in .env — edit that, not this file. # -# Pre-download models to avoid first-run delay: -# scripts/elway fv-ml1 --playbook playbooks/pull-hf-repo.yaml \ -# --var hf_repo=Qwen/Qwen3-Embedding-0.6B -# scripts/elway fv-ml1 --playbook playbooks/pull-hf-repo.yaml \ -# --var hf_repo=Qwen/Qwen3-Reranker-0.6B -# # Skywork-Reward-V2-Llama-3.1-8B-AWQ is a locally-quantized model — lives at # /tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ on fv-ml1 and is # bind-mounted into the reward service at /local-models. Not from HF Hub. services: - vllm-embed: - image: vllm/vllm-openai:${VLLM_VERSION} - container_name: vllm-embed - restart: unless-stopped - ipc: host - ports: - - "${EMBED_PORT}:8000" - volumes: - - /tank/aimodels/huggingface:/hfcache - environment: - - HF_HOME=/hfcache - - HF_HUB_CACHE=/hfcache/hub - - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-} - - VLLM_API_KEY=${API_KEY:-} - command: - - ${EMBED_MODEL} - - --served-model-name - - ${EMBED_MODEL} - - --runner - - pooling - - --host - - 0.0.0.0 - - --port - - "8000" - - --gpu-memory-utilization - - ${EMBED_GPU_MEM_UTIL} - - --max-model-len - - ${EMBED_MAX_MODEL_LEN} - - --dtype - - auto - deploy: - resources: - reservations: - devices: - - driver: nvidia - device_ids: - - "${GPU_ID}" - capabilities: - - gpu - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8000/health"] - interval: 30s - timeout: 10s - retries: 3 - start_period: 180s - networks: - - tnet - labels: - - homepage.group=AI - Eval & Retrieval - - homepage.name=vLLM Embed (Qwen3) - - homepage.icon=mdi-vector-arrange-below - - homepage.description=Qwen3 Embedding via vLLM (fv-ml1) - - homepage.href=http://10.251.50.54:${EMBED_PORT}/docs - - # THE fleet reranker. Backs the LiteLLM `reranker` alias, which is what every - # consumer should name — never the model, never a bake-off arm name. - # - # It won the Brokkr R43 bake-off (docs/pfi/reranker-selection-ledger.md) and - # replaced Qwen3-Reranker-0.6B, which was measured HARMING 80/90 fleet queries - # (no-reranker beat it 89/90 vs 56/90). The R42 v13 acceptance gate went - # 56/90 -> 90/90 on the cutover, its first-ever PASS. - # - # Promoted from a throwaway `docker run` to this service 2026-08-20 (the - # ledger's own open follow-up). It carried the bake-off's arm name from the - # start and KEEPS it: the ledger, persistent-memory and the R43 record all say - # `vllm-rerank-a3`, and renaming for tidiness would orphan every one of those - # references. The name carries its provenance. - # - # Retired alongside this promotion: `vllm-rerank` (Qwen3-Reranker-0.6B, :8002 — - # the rollback path, kept warm 13 days) and `vllm-rerank-a4` - # (gte-reranker-modernbert, :8014 — a documented throughput fallback that was - # never given a gateway alias, so it was unreachable the whole time). - vllm-rerank-a3: - image: vllm/vllm-openai:${VLLM_VERSION} - container_name: vllm-rerank-a3 - restart: unless-stopped - ipc: host - ports: - - "${RERANK_PORT}:8000" - volumes: - - /tank/aimodels/huggingface:/hfcache - environment: - - HF_HOME=/hfcache - - HF_HUB_CACHE=/hfcache/hub - - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-} - - VLLM_API_KEY=${API_KEY:-} - command: - - ${RERANK_MODEL} - - --served-model-name - - ${RERANK_MODEL} - - --runner - - pooling - # No --hf-overrides here. The Qwen reranker needed one to be coerced into a - # sequence-classification head; bge-reranker-v2-m3 is natively a - # cross-encoder and vLLM resolves it directly. - - --host - - 0.0.0.0 - - --port - - "8000" - - --gpu-memory-utilization - - ${RERANK_GPU_MEM_UTIL} - - --max-model-len - - ${RERANK_MAX_MODEL_LEN} - - --dtype - - auto - deploy: - resources: - reservations: - devices: - - driver: nvidia - device_ids: - - "${GPU_ID}" - capabilities: - - gpu - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8000/health"] - interval: 30s - timeout: 10s - retries: 3 - start_period: 180s - networks: - - tnet - labels: - - homepage.group=AI - Eval & Retrieval - - homepage.name=vLLM Rerank (bge-v2-m3) - - homepage.icon=mdi-sort-variant - - homepage.description=BAAI bge-reranker-v2-m3 — the fleet reranker, backs the `reranker` alias (fv-ml1) - - homepage.href=http://10.251.50.54:${RERANK_PORT}/docs + # vllm-embed (Qwen3-Embedding-0.6B, :8001) and vllm-rerank-a3 + # (bge-reranker-v2-m3, :8013) were RETIRED 2026-09-25 (Prime): embedding and + # reranking moved to TEI on esh-ml1 (stacks/embed-rerank), and TEI is now the + # fleet's embed/rerank engine. The gateway names `qwen3-embedding`, `reranker` + # and `reranker-a3-bge-v2-m3` did not change. Their definitions are in git + # history before this commit if a rollback is ever needed. vllm-reward: image: vllm/vllm-openai:${VLLM_VERSION}