Files

T

vh 7bdac80878 feat(embed-rerank): TEI is the fleet embed/rerank engine; esh-ml1 sole backend; retire fv-ml1 seats

Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet
from now on; fv-ml1 retires both once esh-ml1 is up.

- stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports
  8001/8013, fail-closed truncation (--auto-truncate false; embed
  --max-batch-tokens 32768).
- litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address);
  reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's
  `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target.
- Verified via the gateway against the retiring fv-ml1 seats: embed cosine
  median 0.999927 (n=203); rerank top-1/top-3 29/30.
- stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on
  fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged.
- Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md;
  CLAUDE.md gains the TEI convention.

2026-09-25 08:30:53 -07:00

2.8 KiB

Raw Blame History

LiteLLM gateway — the fleet's OpenAI-compatible front door

Base URL: http://10.250.50.70:4000/v1 (ana-docker) · Logs UI: :4000/ui Fronts the ana-ml2 / fv-ml1 / irv-ml1 vLLM seats plus paid vendor passthroughs.

⚠ Never hardcode the model roster — query it

curl -s -H "Authorization: Bearer $KEY" http://10.250.50.70:4000/v1/models \
  | python3 -c "import sys,json;[print(m['id']) for m in json.load(sys.stdin)['data']]"

The roster drifts constantly (40 entries as of 2026-09-18; a doc that names them goes stale within weeks). Any list written down here would be a second copy that rots.

Keys

The gateway 401s without a virtual API key. Three tiers:

key	where	reach
shared all-agents (`all-agents-local`)	`secret get litellm/all-agents-shared-key`, also inline in `~/.claude/CLAUDE.md` § Global tools	every model, paid passthroughs included
infra-ops admin	`~/.config/litellm/infra-ops-key` (0600)	admin operations
project-scoped	request from infra-ops via althing	isolated, auditable spend

⚠ The shared key is NOT free-local-only. It reaches paid GLM/z.ai and Kimi passthroughs, and a paid call on it spends real vendor credits shared across every session and project holding it. There is no isolation. For anything where "shows up on someone else's tab" matters, get a project-scoped key.

Stable aliases worth knowing

summarizer / summarizer-large / classifier — the always-available local seat for summarization, naming/labeling, classification and triage. Essentially free compute; reach for it before spending premium API tokens on work a small local model handles. Embeddings and reranking (qwen3-embedding, reranker) are local and free too — served by TEI on esh-ml1 since 2026-09-25 (qwen3-reranker was retired 2026-08-20). Anything named *-frontier, glm-* or kimi-* is PAID.

Traps

LiteLLM mutates shared deployment config in-place. Per-request sampler params bleed into the SHARED litellm_params for that deployment. Fix: give each variant a distinct served-model-name rather than reusing one with different params.
Retiring a model orphans scoped keys. A stale allowlist produces a silent per-endpoint 403. Audit /key/list + /key/info on every repoint.
The :4000/ui playground cannot test vLLM backends — it sends tools: [], which vLLM rejects with a 400. Use a chat frontend (Open WebUI, Gateway Chat) instead.
"Random model calls" in the logs are usually AUTH FAILURES — empty api_key or a missing Bearer prefix, showing as zero-spend 401s.
Kimi models 400 on any temperature other than 1 — omit the field entirely rather than sending 0. They answer in content with chain-of-thought in reasoning_content.

2.8 KiB Raw Blame History