vh
bc278d4ba8
- esh-pve-nvidia-host -> pve-nvidia-host: headers/dkms/build-essential step, nouveau blacklist + guarded unload (refuses if nouveau bound a device) - esh-ml1-lxc -> gpu-lxc: host vars have no defaults (elway aborts on undefined), rootfs storage/startup order parameterized, CT kept out of all-guests vzdump jobs - embed-rerank: Homepage labels take HOST_NAME/HOST_IP, defaults = esh-ml1
embed-rerank
The fleet's embedding + reranking service, on esh-ml1 (CT 110 on esh-pve,
RTX 2000E Ada), served by Hugging Face Text Embeddings Inference (TEI).
Since 2026-09-25 it is the only backend behind the gateway names
qwen3-embedding, reranker and reranker-a3-bge-v2-m3.
TEI is the fleet's embed/rerank engine (Prime, 2026-09-25). New embedding or
reranking seats go on TEI, not vLLM. Why, and the measurements behind it:
docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md.
| container | model | port | endpoint |
|---|---|---|---|
tei-embed |
Qwen/Qwen3-Embedding-0.6B |
8001 | /v1/embeddings (OpenAI), /embed |
tei-rerank |
BAAI/bge-reranker-v2-m3 |
8013 | /rerank — body {"query", "texts"} |
Gateway wiring (stacks/litellm/conf/config.yaml)
qwen3-embedding→hosted_vllm/Qwen/Qwen3-Embedding-0.6B,api_base: http://10.0.50.80:8001/v1.reranker→huggingface/BAAI/bge-reranker-v2-m3,api_base: http://10.0.50.80:8013(no/v1). ⚠hosted_vllm/sendsdocumentsand TEI answers 422 "missing field texts".reranker-a3-bge-v2-m3is a DB-only alias (not in config.yaml) with the same target.
⚠ Invariants
- Changing
EMBED_MODELinvalidates every index built on it (Worldtree, nevermore, Open WebUI). An engine change is allowed only with a parity measurement against the current vectors. The TEI switch measured cosine median 0.999925 and old-index retrieval overlap 0.988. - Truncation is fail-closed (
--auto-truncate false). TEI's default silently embeds a prefix of an over-length input and returns 200. Now: embed rejects more than 32,768 tokens and rerank more than 8,192, both with 422. - The image tag is GPU-generation specific:
89-is Ada. A different card needs a different prefix (TEI README image table). - nevermore thresholds rerank scores at 0.3. TEI scores differ from the old vLLM seat by up to 0.019, which produced 0 flips in 2,000 scores. Recheck thresholding consumers after any engine or version change.
Deploy
scripts/deploy-stack.sh esh-ml1 embed-rerank
ssh esh-ml1 'cd /opt/docker/compose/embed-rerank && cp -n .env.example .env && docker compose config -q && docker compose up -d'
Host prerequisites (driver, LXC, docker, toolkit) are in
servers/esh-ml1/README.md.
Smoke test
curl -s http://10.0.50.80:8001/v1/embeddings -H 'content-type: application/json' \
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"hello"}' | jq '.data[0].embedding | length' # 1024
curl -s http://10.0.50.80:8013/rerank -H 'content-type: application/json' \
-d '{"query":"cat","texts":["a cat","a car"]}' # index 0 scores ~0.9986