vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(esh-ml1): RTX 2000E Ada on esh-pve serves embed + rerank as a LiteLLM failover

Browse Source 
esh-pve: NVIDIA 580.178.04 (open modules, DKMS) installed on the host from
NVIDIA's .run and loaded live, no reboot. nvidia-persistenced unit creates the
device nodes before pve-guests; the T400's vfio-pci ids and `blacklist nvidia`
retired. playbooks/esh-pve-nvidia-host.yaml.

esh-ml1: CT 110, unprivileged Debian 12, 10.0.50.80, GPU nodes via devN,
NVIDIA userspace from the same .run (--no-kernel-modules), docker-ce +
nvidia-container-toolkit (no-cgroups). playbooks/esh-ml1-lxc.yaml. Not in the
vzdump job on purpose. DNS esh-ml1.esh.internal.

stacks/embed-rerank: Qwen3-Embedding-0.6B :8001 + bge-reranker-v2-m3 :8013 on
the same vLLM v0.24.0 digest and flags as fv-ml1. Measured parity: embed
cosine FV-vs-ESH median 0.999908 (min 0.999772), inside both self-noise
floors; rerank max |delta| 0.000145 vs floor 0.000181, identical ranking.

litellm: qwen3-embedding and reranker gain an esh-ml1 deployment at order 2
behind fv-ml1 (order 1). Order fallback proven with throwaway groups:
refused primary +0.15 s, host-down primary ~18.7 s per call, dead-only 500.

Also: repaired the DB-only alias reranker-a3-bge-v2-m3, dead since the
fv-ml1 relocation (still named 10.250.50.54); documented the third
unkillable homepage wedge on esh-docker-vm.
This commit is contained in:
vh
2026-09-24 22:38:55 -07:00
parent eb46973051
commit 5402568b76
16 changed files with 1053 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/litellm/conf/config.yaml
+33 -1
View File
@@ -443,11 +443,31 @@ model_list:
# explicitly. Operator ruling 2026-08-23. ---
# --- Qwen3 embeddings ---
# TWO deployments, one name, ordered: fv-ml1 serves (order 1); esh-ml1 takes
# over only when fv-ml1 fails (order 2 — LiteLLM's order-based fallback, v1.97
# router.py). Same model, same vLLM version and flags at both sites, so the
# vectors are interchangeable: measured 2026-09-24, cosine FV-vs-ESH median
# 0.999908 (min 0.999772) over 11 texts, inside the FV-vs-FV self-noise floor
# (median 0.999927, min 0.999791); different-text negative control 0.07–0.38.
# ⚠ Failover cost, measured n=3 each: primary REFUSING → +0.15 s; primary
# HOST DOWN (no ARP) → ~18.7 s on EVERY call (no cooldown kicked in). Slow,
# not broken. ⚠ Never add a deployment here that serves a DIFFERENT embedding
# model — vectors are model-specific and a mixed group corrupts indexes.
# esh-ml1: stacks/embed-rerank, playbooks/esh-ml1-lxc.yaml.
- model_name: qwen3-embedding
litellm_params:
model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
api_base: http://10.251.50.54:8001/v1
api_key: os.environ/VLLM_API_KEY
order: 1
model_info:
mode: embedding
- model_name: qwen3-embedding
litellm_params:
model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
api_base: http://10.0.50.80:8001/v1
api_key: os.environ/VLLM_API_KEY
order: 2
model_info:
mode: embedding
@@ -487,12 +507,24 @@ model_list:
enable_thinking: false
model_info:
mode: chat
# reranker → generic capability name for rerank (currently qwen3-reranker).
# reranker → generic capability name for rerank (bge-reranker-v2-m3 since the
# R43 cutover). fv-ml1 order 1, esh-ml1 order 2 — same failover shape as
# qwen3-embedding above. Parity measured 2026-09-24: |score| FV-vs-ESH max
# 0.000145 vs FV-vs-FV floor 0.000181, identical ranking over 11 documents.
- model_name: reranker
litellm_params:
model: hosted_vllm/BAAI/bge-reranker-v2-m3
api_base: http://10.251.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
order: 1
model_info:
mode: rerank
- model_name: reranker
litellm_params:
model: hosted_vllm/BAAI/bge-reranker-v2-m3
api_base: http://10.0.50.80:8013/v1
api_key: os.environ/VLLM_API_KEY
order: 2
model_info:
mode: rerank