feat(esh-ml1): RTX 2000E Ada on esh-pve serves embed + rerank as a LiteLLM failover

esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia` retired. playbooks/esh-pve-nvidia-host.yaml. esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN, NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce + nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the vzdump job on purpose. DNS esh-ml1.esh.internal. stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking. litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2 behind fv-ml1 (order 1). Order fallback proven with throwaway groups: refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500. Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the fv-ml1 relocation (still named 10.250.50.54); documented the third unkillable homepage wedge on esh-docker-vm.
2026-09-24 22:38:55 -07:00
parent eb46973051
commit 5402568b76
16 changed files with 1053 additions and 30 deletions
@@ -0,0 +1,29 @@
+# embed-rerank tunables (esh-ml1). Copy to `.env` on the server.
+#
+# Everything model-shaped here MUST match fv-ml1's stacks/vllm .env: the same
+# models, the same vLLM version, the same max-model-len. A drift in the
+# embedding model or its version makes this seat's vectors incompatible with
+# every index built against fv-ml1's.
+
+# PINNED to fv-ml1's version. Bump both sites together.
+VLLM_VERSION=v0.24.0
+
+# Same host ports as fv-ml1 (container listens on 8000).
+EMBED_PORT=8001
+RERANK_PORT=8013
+
+EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
+RERANK_MODEL=BAAI/bge-reranker-v2-m3
+
+# Fractions of the RTX 2000E Ada's 16,380 MiB. fv-ml1 runs 0.03 of a 96 GB
+# card (~2.9 GB each); 0.20 here is ~3.2 GB each — the same budget plus a
+# little, leaving ~9.5 GB free.
+EMBED_GPU_MEM_UTIL=0.20
+RERANK_GPU_MEM_UTIL=0.20
+EMBED_MAX_MODEL_LEN=8192
+RERANK_MAX_MODEL_LEN=8192
+
+# fv-ml1's seats run with no API key (LiteLLM fronts them); match that.
+API_KEY=
+# Both models are public; no token needed.
+HF_TOKEN=
@@ -0,0 +1,37 @@
+# embed-rerank
+
+The fleet's embedding + reranking models on **esh-ml1** (CT 110 on esh-pve,
+RTX 2000E Ada). The second site for `qwen3-embedding` and `reranker`; fv-ml1's
+[`vllm`](../vllm/) stack is the first.
+
+| container | model | port |
+|---|---|---|
+| `vllm-embed` | `Qwen/Qwen3-Embedding-0.6B` | 8001 |
+| `vllm-rerank-bge` | `BAAI/bge-reranker-v2-m3` | 8013 |
+
+**Keep it in lockstep with `stacks/vllm`:** same models, same `VLLM_VERSION`,
+same `--max-model-len`. The LiteLLM groups treat the two sites as one model,
+so any drift in the embedding model or its version silently mixes
+incompatible vectors into consumers' indexes.
+
+## Deploy
+
+```bash
+scripts/deploy-stack.sh esh-ml1 embed-rerank
+ssh esh-ml1 'cd /opt/docker/compose/embed-rerank && cp -n .env.example .env && docker compose config -q && docker compose up -d'
+```
+
+Host prerequisites (driver, LXC, docker, toolkit) are in
+[`servers/esh-ml1/README.md`](../../servers/esh-ml1/README.md).
+
+## Smoke test
+
+```bash
+curl -s http://10.0.50.80:8001/v1/embeddings -H 'content-type: application/json' \
+  -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"hello"}' | jq '.data[0].embedding | length'   # 1024
+curl -s http://10.0.50.80:8013/rerank -H 'content-type: application/json' \
+  -d '{"model":"BAAI/bge-reranker-v2-m3","query":"cat","documents":["a cat","a car"]}' | jq '.results[0]'
+```
+
+Parity against fv-ml1 is recorded in the host README; re-measure after any
+version bump on either side.
@@ -0,0 +1,130 @@
+# embed-rerank — the fleet's embedding + reranking models, served locally at ESH
+# on esh-ml1 (CT 110 on esh-pve, RTX 2000E Ada, 16 GB).
+#
+# WHY: Prime's decision 2026-09-24 — the Ada card offloads embedding and
+# reranking so ESH consumers (Open WebUI RAG, Paperless) are not tied to one
+# seat on fv-ml1. It serves the SAME models as fv-ml1's `vllm` stack, with the
+# SAME vLLM version and flags, because embedding vectors are model-specific:
+# a different embedding model here would silently poison every index built
+# against fv-ml1's. Mirror stacks/vllm/compose.yaml when that changes.
+#
+#   vllm-embed        Qwen/Qwen3-Embedding-0.6B  → /v1/embeddings   :8001
+#   vllm-rerank-bge   BAAI/bge-reranker-v2-m3    → /rerank, /score  :8013
+#
+# Ports match fv-ml1 on purpose, so a LiteLLM deployment for either site
+# differs only in the host part of api_base.
+#
+# `vllm-rerank-bge`, not fv-ml1's `vllm-rerank-a3`: that name carries R43
+# bake-off provenance for THAT container; and plain `vllm-rerank` was the
+# retired Qwen3-Reranker that measured harmful. Name the model instead.
+#
+# NO `tnet`/traefik-net: esh-ml1 runs no traefik, and an external network
+# that does not exist would stop the stack from starting. Consumers reach the
+# published ports directly.
+#
+# Host setup (driver, LXC, docker, toolkit): playbooks/esh-pve-nvidia-host.yaml
+# then playbooks/esh-ml1-lxc.yaml. Tunables live in .env — edit that, not this.
+
+name: embed-rerank
+
+services:
+  vllm-embed:
+    image: vllm/vllm-openai:${VLLM_VERSION}
+    container_name: vllm-embed
+    restart: unless-stopped
+    ipc: host
+    ports:
+      - "${EMBED_PORT}:8000"
+    volumes:
+      - /opt/aimodels/huggingface:/hfcache
+    environment:
+      - HF_HOME=/hfcache
+      - HF_HUB_CACHE=/hfcache/hub
+      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
+      - VLLM_API_KEY=${API_KEY:-}
+    command:
+      - ${EMBED_MODEL}
+      - --served-model-name
+      - ${EMBED_MODEL}
+      - --runner
+      - pooling
+      - --host
+      - 0.0.0.0
+      - --port
+      - "8000"
+      - --gpu-memory-utilization
+      - ${EMBED_GPU_MEM_UTIL}
+      - --max-model-len
+      - ${EMBED_MAX_MODEL_LEN}
+      - --dtype
+      - auto
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids: ["0"]
+              capabilities: [gpu]
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 180s
+    labels:
+      - homepage.group=AI - Eval & Retrieval
+      - homepage.name=vLLM Embed (esh-ml1)
+      - homepage.icon=mdi-vector-arrange-below
+      - homepage.description=Qwen3 Embedding 0.6B via vLLM (esh-ml1, RTX 2000E Ada)
+      - homepage.href=http://10.0.50.80:${EMBED_PORT}/docs
+
+  vllm-rerank-bge:
+    image: vllm/vllm-openai:${VLLM_VERSION}
+    container_name: vllm-rerank-bge
+    restart: unless-stopped
+    ipc: host
+    ports:
+      - "${RERANK_PORT}:8000"
+    volumes:
+      - /opt/aimodels/huggingface:/hfcache
+    environment:
+      - HF_HOME=/hfcache
+      - HF_HUB_CACHE=/hfcache/hub
+      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
+      - VLLM_API_KEY=${API_KEY:-}
+    command:
+      - ${RERANK_MODEL}
+      - --served-model-name
+      - ${RERANK_MODEL}
+      - --runner
+      - pooling
+      # bge-reranker-v2-m3 is natively a cross-encoder; no --hf-overrides.
+      - --host
+      - 0.0.0.0
+      - --port
+      - "8000"
+      - --gpu-memory-utilization
+      - ${RERANK_GPU_MEM_UTIL}
+      - --max-model-len
+      - ${RERANK_MAX_MODEL_LEN}
+      - --dtype
+      - auto
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids: ["0"]
+              capabilities: [gpu]
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 180s
+    labels:
+      - homepage.group=AI - Eval & Retrieval
+      - homepage.name=vLLM Rerank bge-v2-m3 (esh-ml1)
+      - homepage.icon=mdi-sort-variant
+      - homepage.description=bge-reranker-v2-m3 via vLLM (esh-ml1, RTX 2000E Ada)
+      - homepage.href=http://10.0.50.80:${RERANK_PORT}/docs