feat(esh-ml1): RTX 2000E Ada on esh-pve serves embed + rerank as a LiteLLM failover
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia` retired. playbooks/esh-pve-nvidia-host.yaml. esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN, NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce + nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the vzdump job on purpose. DNS esh-ml1.esh.internal. stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking. litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2 behind fv-ml1 (order 1). Order fallback proven with throwaway groups: refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500. Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the fv-ml1 relocation (still named 10.250.50.54); documented the third unkillable homepage wedge on esh-docker-vm.
This commit is contained in:
@@ -0,0 +1,29 @@
|
||||
# embed-rerank tunables (esh-ml1). Copy to `.env` on the server.
|
||||
#
|
||||
# Everything model-shaped here MUST match fv-ml1's stacks/vllm .env: the same
|
||||
# models, the same vLLM version, the same max-model-len. A drift in the
|
||||
# embedding model or its version makes this seat's vectors incompatible with
|
||||
# every index built against fv-ml1's.
|
||||
|
||||
# PINNED to fv-ml1's version. Bump both sites together.
|
||||
VLLM_VERSION=v0.24.0
|
||||
|
||||
# Same host ports as fv-ml1 (container listens on 8000).
|
||||
EMBED_PORT=8001
|
||||
RERANK_PORT=8013
|
||||
|
||||
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
|
||||
RERANK_MODEL=BAAI/bge-reranker-v2-m3
|
||||
|
||||
# Fractions of the RTX 2000E Ada's 16,380 MiB. fv-ml1 runs 0.03 of a 96 GB
|
||||
# card (~2.9 GB each); 0.20 here is ~3.2 GB each — the same budget plus a
|
||||
# little, leaving ~9.5 GB free.
|
||||
EMBED_GPU_MEM_UTIL=0.20
|
||||
RERANK_GPU_MEM_UTIL=0.20
|
||||
EMBED_MAX_MODEL_LEN=8192
|
||||
RERANK_MAX_MODEL_LEN=8192
|
||||
|
||||
# fv-ml1's seats run with no API key (LiteLLM fronts them); match that.
|
||||
API_KEY=
|
||||
# Both models are public; no token needed.
|
||||
HF_TOKEN=
|
||||
Reference in New Issue
Block a user