feat(esh-ml1): RTX 2000E Ada on esh-pve serves embed + rerank as a LiteLLM failover
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia` retired. playbooks/esh-pve-nvidia-host.yaml. esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN, NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce + nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the vzdump job on purpose. DNS esh-ml1.esh.internal. stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking. litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2 behind fv-ml1 (order 1). Order fallback proven with throwaway groups: refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500. Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the fv-ml1 relocation (still named 10.250.50.54); documented the third unkillable homepage wedge on esh-docker-vm.
This commit is contained in:
@@ -0,0 +1,29 @@
|
||||
# embed-rerank tunables (esh-ml1). Copy to `.env` on the server.
|
||||
#
|
||||
# Everything model-shaped here MUST match fv-ml1's stacks/vllm .env: the same
|
||||
# models, the same vLLM version, the same max-model-len. A drift in the
|
||||
# embedding model or its version makes this seat's vectors incompatible with
|
||||
# every index built against fv-ml1's.
|
||||
|
||||
# PINNED to fv-ml1's version. Bump both sites together.
|
||||
VLLM_VERSION=v0.24.0
|
||||
|
||||
# Same host ports as fv-ml1 (container listens on 8000).
|
||||
EMBED_PORT=8001
|
||||
RERANK_PORT=8013
|
||||
|
||||
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
|
||||
RERANK_MODEL=BAAI/bge-reranker-v2-m3
|
||||
|
||||
# Fractions of the RTX 2000E Ada's 16,380 MiB. fv-ml1 runs 0.03 of a 96 GB
|
||||
# card (~2.9 GB each); 0.20 here is ~3.2 GB each — the same budget plus a
|
||||
# little, leaving ~9.5 GB free.
|
||||
EMBED_GPU_MEM_UTIL=0.20
|
||||
RERANK_GPU_MEM_UTIL=0.20
|
||||
EMBED_MAX_MODEL_LEN=8192
|
||||
RERANK_MAX_MODEL_LEN=8192
|
||||
|
||||
# fv-ml1's seats run with no API key (LiteLLM fronts them); match that.
|
||||
API_KEY=
|
||||
# Both models are public; no token needed.
|
||||
HF_TOKEN=
|
||||
@@ -0,0 +1,37 @@
|
||||
# embed-rerank
|
||||
|
||||
The fleet's embedding + reranking models on **esh-ml1** (CT 110 on esh-pve,
|
||||
RTX 2000E Ada). The second site for `qwen3-embedding` and `reranker`; fv-ml1's
|
||||
[`vllm`](../vllm/) stack is the first.
|
||||
|
||||
| container | model | port |
|
||||
|---|---|---|
|
||||
| `vllm-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 |
|
||||
| `vllm-rerank-bge` | `BAAI/bge-reranker-v2-m3` | 8013 |
|
||||
|
||||
**Keep it in lockstep with `stacks/vllm`:** same models, same `VLLM_VERSION`,
|
||||
same `--max-model-len`. The LiteLLM groups treat the two sites as one model,
|
||||
so any drift in the embedding model or its version silently mixes
|
||||
incompatible vectors into consumers' indexes.
|
||||
|
||||
## Deploy
|
||||
|
||||
```bash
|
||||
scripts/deploy-stack.sh esh-ml1 embed-rerank
|
||||
ssh esh-ml1 'cd /opt/docker/compose/embed-rerank && cp -n .env.example .env && docker compose config -q && docker compose up -d'
|
||||
```
|
||||
|
||||
Host prerequisites (driver, LXC, docker, toolkit) are in
|
||||
[`servers/esh-ml1/README.md`](../../servers/esh-ml1/README.md).
|
||||
|
||||
## Smoke test
|
||||
|
||||
```bash
|
||||
curl -s http://10.0.50.80:8001/v1/embeddings -H 'content-type: application/json' \
|
||||
-d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"hello"}' | jq '.data[0].embedding | length' # 1024
|
||||
curl -s http://10.0.50.80:8013/rerank -H 'content-type: application/json' \
|
||||
-d '{"model":"BAAI/bge-reranker-v2-m3","query":"cat","documents":["a cat","a car"]}' | jq '.results[0]'
|
||||
```
|
||||
|
||||
Parity against fv-ml1 is recorded in the host README; re-measure after any
|
||||
version bump on either side.
|
||||
@@ -0,0 +1,130 @@
|
||||
# embed-rerank — the fleet's embedding + reranking models, served locally at ESH
|
||||
# on esh-ml1 (CT 110 on esh-pve, RTX 2000E Ada, 16 GB).
|
||||
#
|
||||
# WHY: Prime's decision 2026-09-24 — the Ada card offloads embedding and
|
||||
# reranking so ESH consumers (Open WebUI RAG, Paperless) are not tied to one
|
||||
# seat on fv-ml1. It serves the SAME models as fv-ml1's `vllm` stack, with the
|
||||
# SAME vLLM version and flags, because embedding vectors are model-specific:
|
||||
# a different embedding model here would silently poison every index built
|
||||
# against fv-ml1's. Mirror stacks/vllm/compose.yaml when that changes.
|
||||
#
|
||||
# vllm-embed Qwen/Qwen3-Embedding-0.6B → /v1/embeddings :8001
|
||||
# vllm-rerank-bge BAAI/bge-reranker-v2-m3 → /rerank, /score :8013
|
||||
#
|
||||
# Ports match fv-ml1 on purpose, so a LiteLLM deployment for either site
|
||||
# differs only in the host part of api_base.
|
||||
#
|
||||
# `vllm-rerank-bge`, not fv-ml1's `vllm-rerank-a3`: that name carries R43
|
||||
# bake-off provenance for THAT container; and plain `vllm-rerank` was the
|
||||
# retired Qwen3-Reranker that measured harmful. Name the model instead.
|
||||
#
|
||||
# NO `tnet`/traefik-net: esh-ml1 runs no traefik, and an external network
|
||||
# that does not exist would stop the stack from starting. Consumers reach the
|
||||
# published ports directly.
|
||||
#
|
||||
# Host setup (driver, LXC, docker, toolkit): playbooks/esh-pve-nvidia-host.yaml
|
||||
# then playbooks/esh-ml1-lxc.yaml. Tunables live in .env — edit that, not this.
|
||||
|
||||
name: embed-rerank
|
||||
|
||||
services:
|
||||
vllm-embed:
|
||||
image: vllm/vllm-openai:${VLLM_VERSION}
|
||||
container_name: vllm-embed
|
||||
restart: unless-stopped
|
||||
ipc: host
|
||||
ports:
|
||||
- "${EMBED_PORT}:8000"
|
||||
volumes:
|
||||
- /opt/aimodels/huggingface:/hfcache
|
||||
environment:
|
||||
- HF_HOME=/hfcache
|
||||
- HF_HUB_CACHE=/hfcache/hub
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||||
- VLLM_API_KEY=${API_KEY:-}
|
||||
command:
|
||||
- ${EMBED_MODEL}
|
||||
- --served-model-name
|
||||
- ${EMBED_MODEL}
|
||||
- --runner
|
||||
- pooling
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- "8000"
|
||||
- --gpu-memory-utilization
|
||||
- ${EMBED_GPU_MEM_UTIL}
|
||||
- --max-model-len
|
||||
- ${EMBED_MAX_MODEL_LEN}
|
||||
- --dtype
|
||||
- auto
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids: ["0"]
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 180s
|
||||
labels:
|
||||
- homepage.group=AI - Eval & Retrieval
|
||||
- homepage.name=vLLM Embed (esh-ml1)
|
||||
- homepage.icon=mdi-vector-arrange-below
|
||||
- homepage.description=Qwen3 Embedding 0.6B via vLLM (esh-ml1, RTX 2000E Ada)
|
||||
- homepage.href=http://10.0.50.80:${EMBED_PORT}/docs
|
||||
|
||||
vllm-rerank-bge:
|
||||
image: vllm/vllm-openai:${VLLM_VERSION}
|
||||
container_name: vllm-rerank-bge
|
||||
restart: unless-stopped
|
||||
ipc: host
|
||||
ports:
|
||||
- "${RERANK_PORT}:8000"
|
||||
volumes:
|
||||
- /opt/aimodels/huggingface:/hfcache
|
||||
environment:
|
||||
- HF_HOME=/hfcache
|
||||
- HF_HUB_CACHE=/hfcache/hub
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||||
- VLLM_API_KEY=${API_KEY:-}
|
||||
command:
|
||||
- ${RERANK_MODEL}
|
||||
- --served-model-name
|
||||
- ${RERANK_MODEL}
|
||||
- --runner
|
||||
- pooling
|
||||
# bge-reranker-v2-m3 is natively a cross-encoder; no --hf-overrides.
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- "8000"
|
||||
- --gpu-memory-utilization
|
||||
- ${RERANK_GPU_MEM_UTIL}
|
||||
- --max-model-len
|
||||
- ${RERANK_MAX_MODEL_LEN}
|
||||
- --dtype
|
||||
- auto
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids: ["0"]
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 180s
|
||||
labels:
|
||||
- homepage.group=AI - Eval & Retrieval
|
||||
- homepage.name=vLLM Rerank bge-v2-m3 (esh-ml1)
|
||||
- homepage.icon=mdi-sort-variant
|
||||
- homepage.description=bge-reranker-v2-m3 via vLLM (esh-ml1, RTX 2000E Ada)
|
||||
- homepage.href=http://10.0.50.80:${RERANK_PORT}/docs
|
||||
@@ -443,11 +443,31 @@ model_list:
|
||||
# explicitly. Operator ruling 2026-08-23. ---
|
||||
|
||||
# --- Qwen3 embeddings ---
|
||||
# TWO deployments, one name, ordered: fv-ml1 serves (order 1); esh-ml1 takes
|
||||
# over only when fv-ml1 fails (order 2 — LiteLLM's order-based fallback, v1.97
|
||||
# router.py). Same model, same vLLM version and flags at both sites, so the
|
||||
# vectors are interchangeable: measured 2026-09-24, cosine FV-vs-ESH median
|
||||
# 0.999908 (min 0.999772) over 11 texts, inside the FV-vs-FV self-noise floor
|
||||
# (median 0.999927, min 0.999791); different-text negative control 0.07–0.38.
|
||||
# ⚠ Failover cost, measured n=3 each: primary REFUSING → +0.15 s; primary
|
||||
# HOST DOWN (no ARP) → ~18.7 s on EVERY call (no cooldown kicked in). Slow,
|
||||
# not broken. ⚠ Never add a deployment here that serves a DIFFERENT embedding
|
||||
# model — vectors are model-specific and a mixed group corrupts indexes.
|
||||
# esh-ml1: stacks/embed-rerank, playbooks/esh-ml1-lxc.yaml.
|
||||
- model_name: qwen3-embedding
|
||||
litellm_params:
|
||||
model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
|
||||
api_base: http://10.251.50.54:8001/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
order: 1
|
||||
model_info:
|
||||
mode: embedding
|
||||
- model_name: qwen3-embedding
|
||||
litellm_params:
|
||||
model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
|
||||
api_base: http://10.0.50.80:8001/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
order: 2
|
||||
model_info:
|
||||
mode: embedding
|
||||
|
||||
@@ -487,12 +507,24 @@ model_list:
|
||||
enable_thinking: false
|
||||
model_info:
|
||||
mode: chat
|
||||
# reranker → generic capability name for rerank (currently qwen3-reranker).
|
||||
# reranker → generic capability name for rerank (bge-reranker-v2-m3 since the
|
||||
# R43 cutover). fv-ml1 order 1, esh-ml1 order 2 — same failover shape as
|
||||
# qwen3-embedding above. Parity measured 2026-09-24: |score| FV-vs-ESH max
|
||||
# 0.000145 vs FV-vs-FV floor 0.000181, identical ranking over 11 documents.
|
||||
- model_name: reranker
|
||||
litellm_params:
|
||||
model: hosted_vllm/BAAI/bge-reranker-v2-m3
|
||||
api_base: http://10.251.50.54:8013/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
order: 1
|
||||
model_info:
|
||||
mode: rerank
|
||||
- model_name: reranker
|
||||
litellm_params:
|
||||
model: hosted_vllm/BAAI/bge-reranker-v2-m3
|
||||
api_base: http://10.0.50.80:8013/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
order: 2
|
||||
model_info:
|
||||
mode: rerank
|
||||
|
||||
|
||||
Reference in New Issue
Block a user