Files

T

vh 7bdac80878 feat(embed-rerank): TEI is the fleet embed/rerank engine; esh-ml1 sole backend; retire fv-ml1 seats

Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet
from now on; fv-ml1 retires both once esh-ml1 is up.

- stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports
  8001/8013, fail-closed truncation (--auto-truncate false; embed
  --max-batch-tokens 32768).
- litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address);
  reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's
  `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target.
- Verified via the gateway against the retiring fv-ml1 seats: embed cosine
  median 0.999927 (n=203); rerank top-1/top-3 29/30.
- stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on
  fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged.
- Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md;
  CLAUDE.md gains the TEI convention.

2026-09-25 08:30:53 -07:00

2.7 KiB

Raw Blame History

`[2026-09-25]` TEI adopted as the fleet embed/rerank engine; esh-ml1 is the sole backend; fv-ml1 seats retired

Prime's rulings (2026-09-25):

Utility seats move off fv-ml1 to a utility host like esh-ml1, accepting slower service; the Blackwells are kept for higher-requirement work.
parakeet STAYS on fv-ml1 (real-time chat path, every ms counts).
"TEI is embed/reranker server for esh-ml1 and the FLEET in general, in future."
fv-ml1 retires both embed + rerank once esh-ml1 is up. A second RTX 2000 is coming to the fleet soon; until then embed/rerank has one backend (accepted).

Evidence chain: speed A/B vs fv-ml1 (single queries a wash, bulk 3–10× slower on the Ada); whole-novel test (The Stand, 12,814 paragraphs) through the gateway ~12 min one per request on either site, because the gateway is the bottleneck; TEI bake-off (docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md): matches vLLM vectors, no re-embed, slower on bulk, much lighter. Dvalin (Grok research peer) reviewed: agree with caveats. Two caveats were actioned: fail-closed truncation (TEI's default silently embedded a 16,384-token prefix of a 40k input and returned 200) and nevermore's 0.3 rerank threshold (0 flips in 2,000 scores; low sensitivity, since few scores sit near 0.3).

Cut-over, 0820–0830 PT:

esh-ml1: vLLM pair down, TEI tei-embed :8001 + tei-rerank :8013 up (same ports). The old .env is kept as .env.vllm-retired-2026-09-25. The vLLM image (~30 GB) is kept in case the reward classifier (vLLM-only) moves here.
Gateway: qwen3-embedding → esh-ml1 only (hosted_vllm/, unchanged address); reranker → huggingface/ provider (hosted_vllm 422s on TEI's texts body); order fields removed; LiteLLM restarted (~58 s). The DB alias reranker-a3-bge-v2-m3 was PATCHed to the same TEI target.
Verified through the gateway against the still-running FV seats: embed cosine median 0.999927 / min 0.999881 (n=203); rerank top-1/top-3 29/30 (2 of 30 lists had a duplicate-source tie; the disagreeing query wasn't logged).
No direct (non-gateway) consumers existed: per day, fv-ml1 seat POSTs = gateway spend-log rows exactly (09-20→09-24). After the repoint, the only FV hits were my own verification calls.
fv-ml1: vllm-embed + vllm-rerank-a3 stopped and removed; stacks/vllm compose + .env.example updated (reward + coder untouched, 10–11 days up); GPU 1 93,086 → 86,956 MiB (~6.1 GB freed).

Open: esh-ml1 is now load-bearing with no monitoring (no Beszel, Kuma or Homepage). Next: "what else moves". vllm-reward (9.2 GB, vLLM-only) is the candidate and fits beside TEI's ~2.6 GB on 16 GB; vllm-coder (6.3 GB) is latency-sensitive.

2.7 KiB Raw Blame History Unescape Escape

[2026-09-25] TEI adopted as the fleet embed/rerank engine; esh-ml1 is the sole backend; fv-ml1 seats retired

2.7 KiB

Raw Blame History

`[2026-09-25]` TEI adopted as the fleet embed/rerank engine; esh-ml1 is the sole backend; fv-ml1 seats retired