vh
5402568b76
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia` retired. playbooks/esh-pve-nvidia-host.yaml. esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN, NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce + nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the vzdump job on purpose. DNS esh-ml1.esh.internal. stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking. litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2 behind fv-ml1 (order 1). Order fallback proven with throwaway groups: refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500. Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the fv-ml1 relocation (still named 10.250.50.54); documented the third unkillable homepage wedge on esh-docker-vm.
38 lines
1.4 KiB
Markdown
38 lines
1.4 KiB
Markdown
# embed-rerank
|
|
|
|
The fleet's embedding + reranking models on **esh-ml1** (CT 110 on esh-pve,
|
|
RTX 2000E Ada). The second site for `qwen3-embedding` and `reranker`; fv-ml1's
|
|
[`vllm`](../vllm/) stack is the first.
|
|
|
|
| container | model | port |
|
|
|---|---|---|
|
|
| `vllm-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 |
|
|
| `vllm-rerank-bge` | `BAAI/bge-reranker-v2-m3` | 8013 |
|
|
|
|
**Keep it in lockstep with `stacks/vllm`:** same models, same `VLLM_VERSION`,
|
|
same `--max-model-len`. The LiteLLM groups treat the two sites as one model,
|
|
so any drift in the embedding model or its version silently mixes
|
|
incompatible vectors into consumers' indexes.
|
|
|
|
## Deploy
|
|
|
|
```bash
|
|
scripts/deploy-stack.sh esh-ml1 embed-rerank
|
|
ssh esh-ml1 'cd /opt/docker/compose/embed-rerank && cp -n .env.example .env && docker compose config -q && docker compose up -d'
|
|
```
|
|
|
|
Host prerequisites (driver, LXC, docker, toolkit) are in
|
|
[`servers/esh-ml1/README.md`](../../servers/esh-ml1/README.md).
|
|
|
|
## Smoke test
|
|
|
|
```bash
|
|
curl -s http://10.0.50.80:8001/v1/embeddings -H 'content-type: application/json' \
|
|
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"hello"}' | jq '.data[0].embedding | length' # 1024
|
|
curl -s http://10.0.50.80:8013/rerank -H 'content-type: application/json' \
|
|
-d '{"model":"BAAI/bge-reranker-v2-m3","query":"cat","documents":["a cat","a car"]}' | jq '.results[0]'
|
|
```
|
|
|
|
Parity against fv-ml1 is recorded in the host README; re-measure after any
|
|
version bump on either side.
|