Files

T

vh c0d3384c68 feat(litellm): embedder primary/failover split — qwen3-embedding on esh-ml1, new qwen3-embedding-bulk on nh3-ml1

Prime's ruling 2026-10-05 (relayed by albok-dev, confirmed directly) replaces
the 2026-09-26 simple-shuffle load-share for embeddings only; the reranker
groups are unchanged. Routing uses LiteLLM per-deployment `order` (semantics
read at source in v1.97.0): lowest order among healthy deployments, and a
failed request retries at the next order.

Verified after the restart (43 s gateway outage): 6/6 calls per alias landed
on its primary. With nh3-ml1 tei-embed stopped, qwen3-embedding-bulk served 6/6
from esh-ml1 with 200s; once nh3-ml1 was back, bulk returned 6/6 to nh3-ml1.
The albok-service key now covers qwen3-embedding-bulk (200); the Nemi key does
not (403).

2026-10-05 00:32:42 -07:00

3.3 KiB

Raw Permalink Blame History

embed-rerank

The fleet's embedding + reranking service, on esh-ml1 (CT 110 on esh-pve, RTX 2000E Ada), served by Hugging Face Text Embeddings Inference (TEI). It runs on two hosts, esh-ml1 and nh3-ml1 (CT 109 on nh3-pve, the same card). The two are parity-verified as indistinguishable, and since 2026-09-26 (Prime) they are load-shared behind reranker and reranker-a3-bge-v2-m3, with intra-group failover (servers/nh3-ml1/README.md has the measurements). Embeddings are primary + failover since 2026-10-05 (Prime): qwen3-embedding esh-ml1 first, qwen3-embedding-bulk (albok writes) nh3-ml1 first, set with LiteLLM order in stacks/litellm/conf/config.yaml.

TEI is the fleet's embed/rerank engine (Prime, 2026-09-25). New embedding or reranking seats go on TEI, not vLLM. Why, and the measurements behind it: docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md.

container	model	port	endpoint
`tei-embed`	`Qwen/Qwen3-Embedding-0.6B`	8001	`/v1/embeddings` (OpenAI), `/embed`
`tei-rerank`	`BAAI/bge-reranker-v2-m3`	8013	`/rerank` — body `{"query", "texts"}`

Gateway wiring (stacks/litellm/conf/config.yaml)

qwen3-embedding → hosted_vllm/Qwen/Qwen3-Embedding-0.6B, api_base: http://10.0.50.80:8001/v1.
reranker → huggingface/BAAI/bge-reranker-v2-m3, api_base: http://10.0.50.80:8013 (no /v1). ⚠ hosted_vllm/ sends documents and TEI answers 422 "missing field texts".
reranker-a3-bge-v2-m3 is a DB-only alias (not in config.yaml) with the same target.

⚠ Invariants

Changing EMBED_MODEL invalidates every index built on it (Worldtree, nevermore, Open WebUI). An engine change is allowed only with a parity measurement against the current vectors. The TEI switch measured cosine median 0.999925 and old-index retrieval overlap 0.988.
Truncation is fail-closed (--auto-truncate false). TEI's default silently embeds a prefix of an over-length input and returns 200. Now: embed rejects more than 32,768 tokens and rerank more than 8,192, both with 422.
The image tag is GPU-generation specific: 89- is Ada. A different card needs a different prefix (TEI README image table).
nevermore thresholds rerank scores at 0.3. TEI scores differ from the old vLLM seat by up to 0.019, which produced 0 flips in 2,000 scores. Recheck thresholding consumers after any engine or version change.

Deploy

scripts/deploy-stack.sh esh-ml1 embed-rerank
ssh esh-ml1 'cd /opt/docker/compose/embed-rerank && cp -n .env.example .env && docker compose config -q && docker compose up -d'

nh3-ml1 is the same, plus HOST_NAME=nh3-ml1 and HOST_IP=10.100.50.80 in its .env (they only feed the Homepage labels).

Host prerequisites (driver, LXC, docker, toolkit) are in servers/esh-ml1/README.md.

Smoke test

curl -s http://10.0.50.80:8001/v1/embeddings -H 'content-type: application/json' \
  -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"hello"}' | jq '.data[0].embedding | length'   # 1024
curl -s http://10.0.50.80:8013/rerank -H 'content-type: application/json' \
  -d '{"query":"cat","texts":["a cat","a car"]}'                      # index 0 scores ~0.9986

3.3 KiB Raw Permalink Blame History