vh
7bdac80878
Prime, 2026-09-25: TEI serves embedding + reranking for esh-ml1 and the fleet from now on; fv-ml1 retires both once esh-ml1 is up. - stacks/embed-rerank: vLLM -> TEI 1.9.4 (89- Ada build), same ports 8001/8013, fail-closed truncation (--auto-truncate false; embed --max-batch-tokens 32768). - litellm: qwen3-embedding -> esh-ml1 only (hosted_vllm/, unchanged address); reranker -> huggingface/ provider at :8013 (hosted_vllm/ 422s on TEI's `texts` body). DB alias reranker-a3-bge-v2-m3 patched to the same target. - Verified via the gateway against the retiring fv-ml1 seats: embed cosine median 0.999927 (n=203); rerank top-1/top-3 29/30. - stacks/vllm: vllm-embed and vllm-rerank-a3 removed (containers retired on fv-ml1, GPU 1 freed ~6.1 GB); reward + coder unchanged. - Bake-off record moved to docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md; CLAUDE.md gains the TEI convention.
53 lines
2.8 KiB
Markdown
53 lines
2.8 KiB
Markdown
# LiteLLM gateway — the fleet's OpenAI-compatible front door
|
|
|
|
**Base URL:** `http://10.250.50.70:4000/v1` (ana-docker) · **Logs UI:** `:4000/ui`
|
|
Fronts the ana-ml2 / fv-ml1 / irv-ml1 vLLM seats plus paid vendor passthroughs.
|
|
|
|
## ⚠ Never hardcode the model roster — query it
|
|
|
|
```sh
|
|
curl -s -H "Authorization: Bearer $KEY" http://10.250.50.70:4000/v1/models \
|
|
| python3 -c "import sys,json;[print(m['id']) for m in json.load(sys.stdin)['data']]"
|
|
```
|
|
|
|
The roster drifts constantly (40 entries as of 2026-09-18; a doc that names them goes
|
|
stale within weeks). Any list written down here would be a second copy that rots.
|
|
|
|
## Keys
|
|
|
|
The gateway **401s without a virtual API key**. Three tiers:
|
|
|
|
| key | where | reach |
|
|
|---|---|---|
|
|
| shared all-agents (`all-agents-local`) | `secret get litellm/all-agents-shared-key`, also inline in `~/.claude/CLAUDE.md` § Global tools | **every** model, paid passthroughs included |
|
|
| infra-ops admin | `~/.config/litellm/infra-ops-key` (0600) | admin operations |
|
|
| project-scoped | request from infra-ops via althing | isolated, auditable spend |
|
|
|
|
⚠ **The shared key is NOT free-local-only.** It reaches paid GLM/z.ai and Kimi
|
|
passthroughs, and **a paid call on it spends real vendor credits shared across every
|
|
session and project holding it.** There is no isolation. For anything where "shows up
|
|
on someone else's tab" matters, get a project-scoped key.
|
|
|
|
## Stable aliases worth knowing
|
|
|
|
`summarizer` / `summarizer-large` / `classifier` — the always-available local seat for
|
|
summarization, naming/labeling, classification and triage. Essentially free compute;
|
|
reach for it before spending premium API tokens on work a small local model handles.
|
|
Embeddings and reranking (`qwen3-embedding`, `reranker`) are local and free too —
|
|
served by TEI on esh-ml1 since 2026-09-25 (`qwen3-reranker` was retired 2026-08-20). Anything named `*-frontier`, `glm-*` or `kimi-*` is **PAID**.
|
|
|
|
## Traps
|
|
|
|
- **LiteLLM mutates shared deployment config in-place.** Per-request sampler params
|
|
bleed into the SHARED `litellm_params` for that deployment. Fix: give each variant a
|
|
distinct served-model-name rather than reusing one with different params.
|
|
- **Retiring a model orphans scoped keys.** A stale allowlist produces a silent
|
|
per-endpoint 403. Audit `/key/list` + `/key/info` on every repoint.
|
|
- **The `:4000/ui` playground cannot test vLLM backends** — it sends `tools: []`, which
|
|
vLLM rejects with a 400. Use a chat frontend (Open WebUI, Gateway Chat) instead.
|
|
- **"Random model calls" in the logs are usually AUTH FAILURES** — empty `api_key` or a
|
|
missing `Bearer ` prefix, showing as zero-spend 401s.
|
|
- **Kimi models 400 on any `temperature` other than 1** — omit the field entirely
|
|
rather than sending 0. They answer in `content` with chain-of-thought in
|
|
`reasoning_content`.
|