feat(embed-rerank): TEI is the fleet embed/rerank engine; esh-ml1 sole backend; retire fv-ml1 seats
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet from now on; fv-ml1 retires both once esh-ml1 is up. - stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports 8001/8013, fail-closed truncation (--auto-truncate false; embed --max-batch-tokens 32768). - litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address); reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target. - Verified via the gateway against the retiring fv-ml1 seats: embed cosine median 0.999927 (n=203); rerank top-1/top-3 29/30. - stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged. - Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md; CLAUDE.md gains the TEI convention.
This commit is contained in:
+10
-145
@@ -1,162 +1,27 @@
|
||||
# vLLM — Qwen3 Embedding + Reranker + Skywork Reward-V2 classifier.
|
||||
# vLLM — utility seats on fv-ml1: Skywork Reward-V2 classifier + the coder FIM seat.
|
||||
#
|
||||
# Originally created to replace the unmaintained Infinity stack (embed +
|
||||
# rerank); generalized 2026-05-13 to host any vLLM-served model on fv-ml1,
|
||||
# starting with the Skywork-Reward-V2-Llama-3.1-8B reward classifier
|
||||
# (AWQ-quantized locally, served from /tank/aimodels/llm/).
|
||||
#
|
||||
# vLLM runs one model per process, so this stack brings up three containers
|
||||
# sharing a single GPU:
|
||||
#
|
||||
# vllm-embed — Qwen3-Embedding served as an OpenAI /v1/embeddings server
|
||||
# vllm-rerank — Qwen3-Reranker served as a /rerank + /score server
|
||||
# vllm-reward — Skywork-Reward-V2-Llama-3.1-8B-AWQ served as a /classify scorer
|
||||
#
|
||||
# The reranker is a causal-LM checkpoint; --hf-overrides re-maps it to
|
||||
# Qwen3ForSequenceClassification so vLLM's reranking endpoints work and the
|
||||
# model only emits two class logits (no/yes) instead of the full 151k vocab.
|
||||
# ⚠ Embedding + reranking LEFT this stack 2026-09-25 (Prime): they now run on TEI
|
||||
# on esh-ml1 (stacks/embed-rerank), and TEI is the fleet's embed/rerank engine
|
||||
# from now on. Do not re-add them here.
|
||||
#
|
||||
# All tunables live in .env — edit that, not this file.
|
||||
#
|
||||
# Pre-download models to avoid first-run delay:
|
||||
# scripts/elway fv-ml1 --playbook playbooks/pull-hf-repo.yaml \
|
||||
# --var hf_repo=Qwen/Qwen3-Embedding-0.6B
|
||||
# scripts/elway fv-ml1 --playbook playbooks/pull-hf-repo.yaml \
|
||||
# --var hf_repo=Qwen/Qwen3-Reranker-0.6B
|
||||
#
|
||||
# Skywork-Reward-V2-Llama-3.1-8B-AWQ is a locally-quantized model — lives at
|
||||
# /tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ on fv-ml1 and is
|
||||
# bind-mounted into the reward service at /local-models. Not from HF Hub.
|
||||
|
||||
services:
|
||||
vllm-embed:
|
||||
image: vllm/vllm-openai:${VLLM_VERSION}
|
||||
container_name: vllm-embed
|
||||
restart: unless-stopped
|
||||
ipc: host
|
||||
ports:
|
||||
- "${EMBED_PORT}:8000"
|
||||
volumes:
|
||||
- /tank/aimodels/huggingface:/hfcache
|
||||
environment:
|
||||
- HF_HOME=/hfcache
|
||||
- HF_HUB_CACHE=/hfcache/hub
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||||
- VLLM_API_KEY=${API_KEY:-}
|
||||
command:
|
||||
- ${EMBED_MODEL}
|
||||
- --served-model-name
|
||||
- ${EMBED_MODEL}
|
||||
- --runner
|
||||
- pooling
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- "8000"
|
||||
- --gpu-memory-utilization
|
||||
- ${EMBED_GPU_MEM_UTIL}
|
||||
- --max-model-len
|
||||
- ${EMBED_MAX_MODEL_LEN}
|
||||
- --dtype
|
||||
- auto
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids:
|
||||
- "${GPU_ID}"
|
||||
capabilities:
|
||||
- gpu
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 180s
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI - Eval & Retrieval
|
||||
- homepage.name=vLLM Embed (Qwen3)
|
||||
- homepage.icon=mdi-vector-arrange-below
|
||||
- homepage.description=Qwen3 Embedding via vLLM (fv-ml1)
|
||||
- homepage.href=http://10.251.50.54:${EMBED_PORT}/docs
|
||||
|
||||
# THE fleet reranker. Backs the LiteLLM `reranker` alias, which is what every
|
||||
# consumer should name — never the model, never a bake-off arm name.
|
||||
#
|
||||
# It won the Brokkr R43 bake-off (docs/pfi/reranker-selection-ledger.md) and
|
||||
# replaced Qwen3-Reranker-0.6B, which was measured HARMING 80/90 fleet queries
|
||||
# (no-reranker beat it 89/90 vs 56/90). The R42 v13 acceptance gate went
|
||||
# 56/90 -> 90/90 on the cutover, its first-ever PASS.
|
||||
#
|
||||
# Promoted from a throwaway `docker run` to this service 2026-08-20 (the
|
||||
# ledger's own open follow-up). It carried the bake-off's arm name from the
|
||||
# start and KEEPS it: the ledger, persistent-memory and the R43 record all say
|
||||
# `vllm-rerank-a3`, and renaming for tidiness would orphan every one of those
|
||||
# references. The name carries its provenance.
|
||||
#
|
||||
# Retired alongside this promotion: `vllm-rerank` (Qwen3-Reranker-0.6B, :8002 —
|
||||
# the rollback path, kept warm 13 days) and `vllm-rerank-a4`
|
||||
# (gte-reranker-modernbert, :8014 — a documented throughput fallback that was
|
||||
# never given a gateway alias, so it was unreachable the whole time).
|
||||
vllm-rerank-a3:
|
||||
image: vllm/vllm-openai:${VLLM_VERSION}
|
||||
container_name: vllm-rerank-a3
|
||||
restart: unless-stopped
|
||||
ipc: host
|
||||
ports:
|
||||
- "${RERANK_PORT}:8000"
|
||||
volumes:
|
||||
- /tank/aimodels/huggingface:/hfcache
|
||||
environment:
|
||||
- HF_HOME=/hfcache
|
||||
- HF_HUB_CACHE=/hfcache/hub
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||||
- VLLM_API_KEY=${API_KEY:-}
|
||||
command:
|
||||
- ${RERANK_MODEL}
|
||||
- --served-model-name
|
||||
- ${RERANK_MODEL}
|
||||
- --runner
|
||||
- pooling
|
||||
# No --hf-overrides here. The Qwen reranker needed one to be coerced into a
|
||||
# sequence-classification head; bge-reranker-v2-m3 is natively a
|
||||
# cross-encoder and vLLM resolves it directly.
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- "8000"
|
||||
- --gpu-memory-utilization
|
||||
- ${RERANK_GPU_MEM_UTIL}
|
||||
- --max-model-len
|
||||
- ${RERANK_MAX_MODEL_LEN}
|
||||
- --dtype
|
||||
- auto
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids:
|
||||
- "${GPU_ID}"
|
||||
capabilities:
|
||||
- gpu
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 180s
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI - Eval & Retrieval
|
||||
- homepage.name=vLLM Rerank (bge-v2-m3)
|
||||
- homepage.icon=mdi-sort-variant
|
||||
- homepage.description=BAAI bge-reranker-v2-m3 — the fleet reranker, backs the `reranker` alias (fv-ml1)
|
||||
- homepage.href=http://10.251.50.54:${RERANK_PORT}/docs
|
||||
# vllm-embed (Qwen3-Embedding-0.6B, :8001) and vllm-rerank-a3
|
||||
# (bge-reranker-v2-m3, :8013) were RETIRED 2026-09-25 (Prime): embedding and
|
||||
# reranking moved to TEI on esh-ml1 (stacks/embed-rerank), and TEI is now the
|
||||
# fleet's embed/rerank engine. The gateway names `qwen3-embedding`, `reranker`
|
||||
# and `reranker-a3-bge-v2-m3` did not change. Their definitions are in git
|
||||
# history before this commit if a rollback is ever needed.
|
||||
|
||||
vllm-reward:
|
||||
image: vllm/vllm-openai:${VLLM_VERSION}
|
||||
|
||||
Reference in New Issue
Block a user