vh
5960526c3f
NH3 site visit done: Secure Boot off, iGPU restored as boot VGA, AMT port cabled. - nh3-pve: NVIDIA 580.178.04 (DKMS, open modules) via pve-nvidia-host.yaml. - nh3-ml1 = CT 109 @ 10.100.50.80 via gpu-lxc.yaml; embed-rerank (TEI 1.9.4) deployed with HOST_NAME/HOST_IP labels. - Parity vs esh-ml1 (1,126 texts, 2 runs/host, controls): embed cosine min 0.999993 = own noise floor; overlap@10 1.000 vs MRL-256 positive control 0.684; rerank top-1 1.00, max diff 0.0014 vs floor 0.0020. On-box speed identical within rep spread. - gpu-lxc.yaml: first step upgrades lxc-pve to >= 6.0.0-2 (Proxmox fix #7006). With 6.0.0-1 every docker run in a nesting CT failed on runc 1.5's sysctl reopen; applied on nh3-pve (one package). - pve-nvidia-host.yaml: document that the headers meta drags in the newest kernel (nh3-pve went 6.8.12-11 -> -43 at the next reboot). - Monitoring: Beszel NVIDIA agent + 5 alerts, Kuma #29/#30, Homepage nh3-ml1-docker, Dozzle agent (hub 8 clients). DNS nh3-ml1.nh3.internal. - nh3-pve README: SB/IGFX/driver/kernel state, btmtk oops on -4x kernels, AMT cabled but unreachable on the network. Gateway routing to nh3-ml1 is not changed.
embed-rerank
The fleet's embedding + reranking service, on esh-ml1 (CT 110 on esh-pve,
RTX 2000E Ada), served by Hugging Face Text Embeddings Inference (TEI).
Since 2026-09-25 it is the only backend behind the gateway names
qwen3-embedding, reranker and reranker-a3-bge-v2-m3. A second instance
runs on nh3-ml1 (CT 109 on nh3-pve, the same card), parity-verified against
esh-ml1 on 2026-09-25 (servers/nh3-ml1/README.md). It is not in the gateway
yet; that is Prime's call.
TEI is the fleet's embed/rerank engine (Prime, 2026-09-25). New embedding or
reranking seats go on TEI, not vLLM. Why, and the measurements behind it:
docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md.
| container | model | port | endpoint |
|---|---|---|---|
tei-embed |
Qwen/Qwen3-Embedding-0.6B |
8001 | /v1/embeddings (OpenAI), /embed |
tei-rerank |
BAAI/bge-reranker-v2-m3 |
8013 | /rerank — body {"query", "texts"} |
Gateway wiring (stacks/litellm/conf/config.yaml)
qwen3-embedding→hosted_vllm/Qwen/Qwen3-Embedding-0.6B,api_base: http://10.0.50.80:8001/v1.reranker→huggingface/BAAI/bge-reranker-v2-m3,api_base: http://10.0.50.80:8013(no/v1). ⚠hosted_vllm/sendsdocumentsand TEI answers 422 "missing field texts".reranker-a3-bge-v2-m3is a DB-only alias (not in config.yaml) with the same target.
⚠ Invariants
- Changing
EMBED_MODELinvalidates every index built on it (Worldtree, nevermore, Open WebUI). An engine change is allowed only with a parity measurement against the current vectors. The TEI switch measured cosine median 0.999925 and old-index retrieval overlap 0.988. - Truncation is fail-closed (
--auto-truncate false). TEI's default silently embeds a prefix of an over-length input and returns 200. Now: embed rejects more than 32,768 tokens and rerank more than 8,192, both with 422. - The image tag is GPU-generation specific:
89-is Ada. A different card needs a different prefix (TEI README image table). - nevermore thresholds rerank scores at 0.3. TEI scores differ from the old vLLM seat by up to 0.019, which produced 0 flips in 2,000 scores. Recheck thresholding consumers after any engine or version change.
Deploy
scripts/deploy-stack.sh esh-ml1 embed-rerank
ssh esh-ml1 'cd /opt/docker/compose/embed-rerank && cp -n .env.example .env && docker compose config -q && docker compose up -d'
nh3-ml1 is the same, plus HOST_NAME=nh3-ml1 and HOST_IP=10.100.50.80 in its
.env (they only feed the Homepage labels).
Host prerequisites (driver, LXC, docker, toolkit) are in
servers/esh-ml1/README.md.
Smoke test
curl -s http://10.0.50.80:8001/v1/embeddings -H 'content-type: application/json' \
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"hello"}' | jq '.data[0].embedding | length' # 1024
curl -s http://10.0.50.80:8013/rerank -H 'content-type: application/json' \
-d '{"query":"cat","texts":["a cat","a car"]}' # index 0 scores ~0.9986