feat(embed-rerank): TEI is the fleet embed/rerank engine; esh-ml1 sole backend; retire fv-ml1 seats

Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet
from now on; fv-ml1 retires both once esh-ml1 is up.

- stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports
  8001/8013, fail-closed truncation (--auto-truncate false; embed
  --max-batch-tokens 32768).
- litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address);
  reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's
  `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target.
- Verified via the gateway against the retiring fv-ml1 seats: embed cosine
  median 0.999927 (n=203); rerank top-1/top-3 29/30.
- stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on
  fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged.
- Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md;
  CLAUDE.md gains the TEI convention.

stacks/litellm/conf/config.yaml

@@ -7,8 +7,8 @@
 #
 # Model-name → upstream mapping:
 #   phi4-mini        → vLLM :8004  (generative chat)
-#   qwen3-embedding  → vLLM :8001  (/v1/embeddings)
-#   qwen3-reranker   → vLLM :8002  (/rerank)
+#   qwen3-embedding  → TEI on esh-ml1 :8001  (/v1/embeddings)
+#   reranker         → TEI on esh-ml1 :8013  (/rerank, huggingface/ provider)
 #   *  (wildcard)    → llama-swap :9292  (the swappable generative zoo)
 #
 # The wildcard fronts llama-swap so its whole model zoo logs through the
@@ -443,31 +443,21 @@ model_list:
   #     explicitly. Operator ruling 2026-08-23. ---
 
   # --- Qwen3 embeddings ---
-  # TWO deployments, one name, ordered: fv-ml1 serves (order 1); esh-ml1 takes
-  # over only when fv-ml1 fails (order 2 — LiteLLM's order-based fallback, v1.97
-  # router.py). Same model, same vLLM version and flags at both sites, so the
-  # vectors are interchangeable: measured 2026-09-24, cosine FV-vs-ESH median
-  # 0.999908 (min 0.999772) over 11 texts, inside the FV-vs-FV self-noise floor
-  # (median 0.999927, min 0.999791); different-text negative control 0.07–0.38.
-  # ⚠ Failover cost, measured n=3 each: primary REFUSING → +0.15 s; primary
-  # HOST DOWN (no ARP) → ~18.7 s on EVERY call (no cooldown kicked in). Slow,
-  # not broken. ⚠ Never add a deployment here that serves a DIFFERENT embedding
-  # model — vectors are model-specific and a mixed group corrupts indexes.
-  # esh-ml1: stacks/embed-rerank, playbooks/esh-ml1-lxc.yaml.
-  - model_name: qwen3-embedding
-    litellm_params:
-      model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
-      api_base: http://10.251.50.54:8001/v1
-      api_key: os.environ/VLLM_API_KEY
-      order: 1
-    model_info:
-      mode: embedding
+  # ONE backend: esh-ml1 (RTX 2000E Ada), served by Hugging Face TEI — the fleet's
+  # embed/rerank engine from 2026-09-25 (Prime). fv-ml1's vLLM seat was retired the
+  # same day. TEI's /v1/embeddings is OpenAI-compatible, so the provider stays
+  # `hosted_vllm/`. Same model as before; TEI vectors match the old vLLM ones
+  # (cosine median 0.999925, inside vLLM's own noise; old-index retrieval overlap
+  # 0.988 vs 0.986 self-noise), so no index needed re-embedding.
+  # ⚠ Single backend until the second RTX 2000 lands (Prime, 2026-09-25): if
+  # esh-ml1, esh-pve or the ESH mesh route is down, fleet embeddings are down.
+  # ⚠ Never point this name at a DIFFERENT embedding model — vectors are
+  # model-specific. Engine bake-off: docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md.
   - model_name: qwen3-embedding
     litellm_params:
       model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
       api_base: http://10.0.50.80:8001/v1
       api_key: os.environ/VLLM_API_KEY
-      order: 2
     model_info:
       mode: embedding
 
@@ -508,23 +498,17 @@ model_list:
     model_info:
       mode: chat
   # reranker → generic capability name for rerank (bge-reranker-v2-m3 since the
-  # R43 cutover). fv-ml1 order 1, esh-ml1 order 2 — same failover shape as
-  # qwen3-embedding above. Parity measured 2026-09-24: |score| FV-vs-ESH max
-  # 0.000145 vs FV-vs-FV floor 0.000181, identical ranking over 11 documents.
+  # R43 cutover). ONE backend: TEI on esh-ml1 (fv-ml1's vLLM seat retired
+  # 2026-09-25). ⚠ TEI's /rerank takes `texts`, not `documents`: this MUST use the
+  # `huggingface/` provider. `hosted_vllm/` gets a 422 "missing field texts".
+  # api_base has NO /v1. api_key is a placeholder; TEI runs without auth.
+  # Parity vs the retired vLLM seat: top-1/top-3 identical (129/130 top-1 over
+  # two runs); scores move up to 0.019, reshuffling only near-zero tail docs.
   - model_name: reranker
     litellm_params:
-      model: hosted_vllm/BAAI/bge-reranker-v2-m3
-      api_base: http://10.251.50.54:8013/v1
-      api_key: os.environ/VLLM_API_KEY
-      order: 1
-    model_info:
-      mode: rerank
-  - model_name: reranker
-    litellm_params:
-      model: hosted_vllm/BAAI/bge-reranker-v2-m3
-      api_base: http://10.0.50.80:8013/v1
-      api_key: os.environ/VLLM_API_KEY
-      order: 2
+      model: huggingface/BAAI/bge-reranker-v2-m3
+      api_base: http://10.0.50.80:8013
+      api_key: "none"
     model_info:
       mode: rerank