vh
7bdac80878
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet from now on; fv-ml1 retires both once esh-ml1 is up. - stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports 8001/8013, fail-closed truncation (--auto-truncate false; embed --max-batch-tokens 32768). - litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address); reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target. - Verified via the gateway against the retiring fv-ml1 seats: embed cosine median 0.999927 (n=203); rerank top-1/top-3 29/30. - stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged. - Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md; CLAUDE.md gains the TEI convention.
44 lines
2.7 KiB
Markdown
44 lines
2.7 KiB
Markdown
# `[2026-09-25]` TEI adopted as the fleet embed/rerank engine; esh-ml1 is the sole backend; fv-ml1 seats retired
|
||
|
||
**Prime's rulings (2026-09-25):**
|
||
- Utility seats move off fv-ml1 to a utility host like esh-ml1, accepting slower
|
||
service; the Blackwells are kept for higher-requirement work.
|
||
- **parakeet STAYS on fv-ml1** (real-time chat path, every ms counts).
|
||
- **"TEI is embed/reranker server for esh-ml1 and the FLEET in general, in
|
||
future."**
|
||
- fv-ml1 retires both embed + rerank once esh-ml1 is up. A **second RTX 2000 is
|
||
coming to the fleet soon**; until then embed/rerank has one backend (accepted).
|
||
|
||
**Evidence chain:** speed A/B vs fv-ml1 (single queries a wash, bulk 3–10×
|
||
slower on the Ada); whole-novel test (*The Stand*, 12,814 paragraphs) through
|
||
the gateway ~12 min one per request on either site, because the gateway is the
|
||
bottleneck; TEI bake-off (`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`):
|
||
matches vLLM vectors, no re-embed, slower on bulk, much lighter. Dvalin
|
||
(Grok research peer) reviewed: agree with caveats. Two caveats were actioned:
|
||
fail-closed truncation (TEI's default silently embedded a 16,384-token prefix of a
|
||
40k input and returned 200) and nevermore's 0.3 rerank threshold (0 flips in 2,000
|
||
scores; low sensitivity, since few scores sit near 0.3).
|
||
|
||
**Cut-over, 0820–0830 PT:**
|
||
1. esh-ml1: vLLM pair down, TEI `tei-embed` :8001 + `tei-rerank` :8013 up (same
|
||
ports). The old `.env` is kept as `.env.vllm-retired-2026-09-25`. The vLLM image
|
||
(~30 GB) is kept in case the reward classifier (vLLM-only) moves here.
|
||
2. Gateway: `qwen3-embedding` → esh-ml1 only (`hosted_vllm/`, unchanged
|
||
address); `reranker` → **`huggingface/`** provider (hosted_vllm 422s on TEI's
|
||
`texts` body); `order` fields removed; LiteLLM restarted (~58 s). The DB
|
||
alias `reranker-a3-bge-v2-m3` was PATCHed to the same TEI target.
|
||
3. Verified through the gateway against the still-running FV seats: embed
|
||
cosine median 0.999927 / min 0.999881 (n=203); rerank top-1/top-3 29/30 (2 of
|
||
30 lists had a duplicate-source tie; the disagreeing query wasn't logged).
|
||
4. **No direct (non-gateway) consumers existed**: per day, fv-ml1 seat POSTs =
|
||
gateway spend-log rows exactly (09-20→09-24). After the repoint, the only FV hits
|
||
were my own verification calls.
|
||
5. fv-ml1: `vllm-embed` + `vllm-rerank-a3` stopped and removed; `stacks/vllm`
|
||
compose + `.env.example` updated (reward + coder untouched, 10–11 days up);
|
||
GPU 1 93,086 → 86,956 MiB (~6.1 GB freed).
|
||
|
||
**Open:** esh-ml1 is now load-bearing with no monitoring (no Beszel, Kuma or
|
||
Homepage). Next: "what else moves". vllm-reward (9.2 GB, vLLM-only) is the
|
||
candidate and fits beside TEI's ~2.6 GB on 16 GB; vllm-coder (6.3 GB) is
|
||
latency-sensitive.
|