feat(embed-rerank): TEI is the fleet embed/rerank engine; esh-ml1 sole backend; retire fv-ml1 seats

Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet from now on; fv-ml1 retires both once esh-ml1 is up. - stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports 8001/8013, fail-closed truncation (--auto-truncate false; embed --max-batch-tokens 32768). - litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address); reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target. - Verified via the gateway against the retiring fv-ml1 seats: embed cosine median 0.999927 (n=203); rerank top-1/top-3 29/30. - stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged. - Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md; CLAUDE.md gains the TEI convention.
2026-09-25 08:30:53 -07:00
parent 49b4bf0177
commit 7bdac80878
17 changed files with 271 additions and 418 deletions
@@ -12,23 +12,15 @@
 VLLM_VERSION=v0.24.0

 # Host ports (container always listens on 8000 internally)
-EMBED_PORT=8001
-# 8013 — the reranker moved here 2026-08-20 when bge-v2-m3 (the R43 winner, which
-# had been running as a throwaway `docker run` on this port) was promoted into
-# this stack and the Qwen incumbent on :8002 was retired.
-RERANK_PORT=8013
+# (EMBED_PORT 8001 / RERANK_PORT 8013 retired 2026-09-25 with their seats —
+# embedding + reranking moved to TEI on esh-ml1, stacks/embed-rerank.)
 REWARD_PORT=8003

 # GPU assignment — all services share this GPU
 # (ana-ml2 has 0 and 1; default 1 keeps 0 free for heavy LLM work in llama-swap)
 GPU_ID=1

-# Models — reference by full repo name in API requests
-EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
-# bge-reranker-v2-m3 — the R43 bake-off winner, replacing Qwen3-Reranker-0.6B
-# (measured HARMING 80/90 fleet queries). Multilingual cross-encoder; needs no
-# --hf-overrides, unlike the Qwen reranker it displaced.
-RERANK_MODEL=BAAI/bge-reranker-v2-m3
+# Models
 # Skywork is a local-path AWQ output, not from HF Hub. Bind-mounted into the
 # reward container at /local-models — see compose.yaml. No env var here for
 # the model path itself since it's hard-coded in the compose command.
@@ -62,16 +54,12 @@ RERANK_MODEL=BAAI/bge-reranker-v2-m3
 # each at 0.05 (mostly util-reservation waste); 0.03 (~3.6 GB) fits weights +
 # CUDA context with room, freeing ~4 GB back to granite. Recreate them ONE AT A
 # TIME — concurrent recreate races the memory-profiling assertion.
-EMBED_GPU_MEM_UTIL=0.03
-RERANK_GPU_MEM_UTIL=0.03
 REWARD_GPU_MEM_UTIL=0.10

 # Context length caps — lower these if VRAM is tight.
 # Qwen3-Embedding supports up to 32k; reranker up to 32k.
 # Skywork capped at 16k server-side as defense-in-depth; JudgeClient also
 # enforces the cap at dispatch time per spec.
-EMBED_MAX_MODEL_LEN=8192
-RERANK_MAX_MODEL_LEN=8192
 REWARD_MAX_MODEL_LEN=16384

 # Optional API key — leave blank for no auth (fine on the internal network).
@@ -1,5 +1,11 @@
 # vllm

+> ⚠ **2026-09-25: embedding + reranking LEFT this stack.** `vllm-embed` and
+> `vllm-rerank-a3` were retired; the fleet's embed/rerank now runs on **TEI on
+> esh-ml1** (`stacks/embed-rerank`), and TEI is the fleet engine for those from
+> now on (Prime). What remains here is `vllm-reward` and `vllm-coder` on fv-ml1.
+> The history below predates that.
+
 Multi-service vLLM stack on ana-ml2. Started as Qwen3 embedding + rerank
 (replacing the unmaintained Infinity stack); generalized to host any vLLM
 model on the box, currently three services:
@@ -1,162 +1,27 @@
-# vLLM — Qwen3 Embedding + Reranker + Skywork Reward-V2 classifier.
+# vLLM — utility seats on fv-ml1: Skywork Reward-V2 classifier + the coder FIM seat.
 #
 # Originally created to replace the unmaintained Infinity stack (embed +
 # rerank); generalized 2026-05-13 to host any vLLM-served model on fv-ml1,
 # starting with the Skywork-Reward-V2-Llama-3.1-8B reward classifier
 # (AWQ-quantized locally, served from /tank/aimodels/llm/).
 #
-# vLLM runs one model per process, so this stack brings up three containers
-# sharing a single GPU:
-#
-#   vllm-embed   — Qwen3-Embedding served as an OpenAI /v1/embeddings server
-#   vllm-rerank  — Qwen3-Reranker served as a /rerank + /score server
-#   vllm-reward  — Skywork-Reward-V2-Llama-3.1-8B-AWQ served as a /classify scorer
-#
-# The reranker is a causal-LM checkpoint; --hf-overrides re-maps it to
-# Qwen3ForSequenceClassification so vLLM's reranking endpoints work and the
-# model only emits two class logits (no/yes) instead of the full 151k vocab.
+# ⚠ Embedding + reranking LEFT this stack 2026-09-25 (Prime): they now run on TEI
+# on esh-ml1 (stacks/embed-rerank), and TEI is the fleet's embed/rerank engine
+# from now on. Do not re-add them here.
 #
 # All tunables live in .env — edit that, not this file.
 #
-# Pre-download models to avoid first-run delay:
-#   scripts/elway fv-ml1 --playbook playbooks/pull-hf-repo.yaml \
-#       --var hf_repo=Qwen/Qwen3-Embedding-0.6B
-#   scripts/elway fv-ml1 --playbook playbooks/pull-hf-repo.yaml \
-#       --var hf_repo=Qwen/Qwen3-Reranker-0.6B
-#
 # Skywork-Reward-V2-Llama-3.1-8B-AWQ is a locally-quantized model — lives at
 # /tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ on fv-ml1 and is
 # bind-mounted into the reward service at /local-models. Not from HF Hub.

 services:
-  vllm-embed:
-    image: vllm/vllm-openai:${VLLM_VERSION}
-    container_name: vllm-embed
-    restart: unless-stopped
-    ipc: host
-    ports:
-      - "${EMBED_PORT}:8000"
-    volumes:
-      - /tank/aimodels/huggingface:/hfcache
-    environment:
-      - HF_HOME=/hfcache
-      - HF_HUB_CACHE=/hfcache/hub
-      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
-      - VLLM_API_KEY=${API_KEY:-}
-    command:
-      - ${EMBED_MODEL}
-      - --served-model-name
-      - ${EMBED_MODEL}
-      - --runner
-      - pooling
-      - --host
-      - 0.0.0.0
-      - --port
-      - "8000"
-      - --gpu-memory-utilization
-      - ${EMBED_GPU_MEM_UTIL}
-      - --max-model-len
-      - ${EMBED_MAX_MODEL_LEN}
-      - --dtype
-      - auto
-    deploy:
-      resources:
-        reservations:
-          devices:
-            - driver: nvidia
-              device_ids:
-                - "${GPU_ID}"
-              capabilities:
-                - gpu
-    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-      start_period: 180s
-    networks:
-      - tnet
-    labels:
-      - homepage.group=AI - Eval & Retrieval
-      - homepage.name=vLLM Embed (Qwen3)
-      - homepage.icon=mdi-vector-arrange-below
-      - homepage.description=Qwen3 Embedding via vLLM (fv-ml1)
-      - homepage.href=http://10.251.50.54:${EMBED_PORT}/docs
-
-  # THE fleet reranker. Backs the LiteLLM `reranker` alias, which is what every
-  # consumer should name — never the model, never a bake-off arm name.
-  #
-  # It won the Brokkr R43 bake-off (docs/pfi/reranker-selection-ledger.md) and
-  # replaced Qwen3-Reranker-0.6B, which was measured HARMING 80/90 fleet queries
-  # (no-reranker beat it 89/90 vs 56/90). The R42 v13 acceptance gate went
-  # 56/90 -> 90/90 on the cutover, its first-ever PASS.
-  #
-  # Promoted from a throwaway `docker run` to this service 2026-08-20 (the
-  # ledger's own open follow-up). It carried the bake-off's arm name from the
-  # start and KEEPS it: the ledger, persistent-memory and the R43 record all say
-  # `vllm-rerank-a3`, and renaming for tidiness would orphan every one of those
-  # references. The name carries its provenance.
-  #
-  # Retired alongside this promotion: `vllm-rerank` (Qwen3-Reranker-0.6B, :8002 —
-  # the rollback path, kept warm 13 days) and `vllm-rerank-a4`
-  # (gte-reranker-modernbert, :8014 — a documented throughput fallback that was
-  # never given a gateway alias, so it was unreachable the whole time).
-  vllm-rerank-a3:
-    image: vllm/vllm-openai:${VLLM_VERSION}
-    container_name: vllm-rerank-a3
-    restart: unless-stopped
-    ipc: host
-    ports:
-      - "${RERANK_PORT}:8000"
-    volumes:
-      - /tank/aimodels/huggingface:/hfcache
-    environment:
-      - HF_HOME=/hfcache
-      - HF_HUB_CACHE=/hfcache/hub
-      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
-      - VLLM_API_KEY=${API_KEY:-}
-    command:
-      - ${RERANK_MODEL}
-      - --served-model-name
-      - ${RERANK_MODEL}
-      - --runner
-      - pooling
-      # No --hf-overrides here. The Qwen reranker needed one to be coerced into a
-      # sequence-classification head; bge-reranker-v2-m3 is natively a
-      # cross-encoder and vLLM resolves it directly.
-      - --host
-      - 0.0.0.0
-      - --port
-      - "8000"
-      - --gpu-memory-utilization
-      - ${RERANK_GPU_MEM_UTIL}
-      - --max-model-len
-      - ${RERANK_MAX_MODEL_LEN}
-      - --dtype
-      - auto
-    deploy:
-      resources:
-        reservations:
-          devices:
-            - driver: nvidia
-              device_ids:
-                - "${GPU_ID}"
-              capabilities:
-                - gpu
-    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-      start_period: 180s
-    networks:
-      - tnet
-    labels:
-      - homepage.group=AI - Eval & Retrieval
-      - homepage.name=vLLM Rerank (bge-v2-m3)
-      - homepage.icon=mdi-sort-variant
-      - homepage.description=BAAI bge-reranker-v2-m3 — the fleet reranker, backs the `reranker` alias (fv-ml1)
-      - homepage.href=http://10.251.50.54:${RERANK_PORT}/docs
+  # vllm-embed (Qwen3-Embedding-0.6B, :8001) and vllm-rerank-a3
+  # (bge-reranker-v2-m3, :8013) were RETIRED 2026-09-25 (Prime): embedding and
+  # reranking moved to TEI on esh-ml1 (stacks/embed-rerank), and TEI is now the
+  # fleet's embed/rerank engine. The gateway names `qwen3-embedding`, `reranker`
+  # and `reranker-a3-bge-v2-m3` did not change. Their definitions are in git
+  # history before this commit if a rollback is ever needed.

  vllm-reward:
    image: vllm/vllm-openai:${VLLM_VERSION}