# LiteLLM gateway — the fleet's OpenAI-compatible front door

**Base URL:** `http://10.250.50.70:4000/v1` (ana-docker) · **Logs UI:** `:4000/ui`
Fronts the ana-ml2 / fv-ml1 / irv-ml1 vLLM seats plus paid vendor passthroughs.

## ⚠ Never hardcode the model roster — query it

```sh
curl -s -H "Authorization: Bearer $KEY" http://10.250.50.70:4000/v1/models \
  | python3 -c "import sys,json;[print(m['id']) for m in json.load(sys.stdin)['data']]"
```

The roster drifts constantly (40 entries as of 2026-09-18; a doc that names them goes
stale within weeks). Any list written down here would be a second copy that rots.

## Keys

The gateway **401s without a virtual API key**. Three tiers:

| key | where | reach |
|---|---|---|
| shared all-agents (`all-agents-local`) | `~/.claude/CLAUDE.md` § Global tools | **every** model, paid passthroughs included |
| infra-ops admin | `~/.config/litellm/infra-ops-key` (0600) | admin operations |
| project-scoped | request from infra-ops via althing | isolated, auditable spend |

⚠ **The shared key is NOT free-local-only.** It reaches paid GLM/z.ai and Kimi
passthroughs, and **a paid call on it spends real vendor credits shared across every
session and project holding it.** There is no isolation. For anything where "shows up
on someone else's tab" matters, get a project-scoped key.

## Stable aliases worth knowing

`summarizer` / `summarizer-large` / `classifier` — the always-available local seat for
summarization, naming/labeling, classification and triage. Essentially free compute;
reach for it before spending premium API tokens on work a small local model handles.
Embeddings and rerankers (`qwen3-embedding`, `qwen3-reranker`, `reranker`) are local
and free too. Anything named `*-frontier`, `glm-*` or `kimi-*` is **PAID**.

## Traps

- **LiteLLM mutates shared deployment config in-place.** Per-request sampler params
  bleed into the SHARED `litellm_params` for that deployment. Fix: give each variant a
  distinct served-model-name rather than reusing one with different params.
- **Retiring a model orphans scoped keys.** A stale allowlist produces a silent
  per-endpoint 403. Audit `/key/list` + `/key/info` on every repoint.
- **The `:4000/ui` playground cannot test vLLM backends** — it sends `tools: []`, which
  vLLM rejects with a 400. Use a chat frontend (Open WebUI, Gateway Chat) instead.
- **"Random model calls" in the logs are usually AUTH FAILURES** — empty `api_key` or a
  missing `Bearer ` prefix, showing as zero-spend 401s.
- **Kimi models 400 on any `temperature` other than 1** — omit the field entirely
  rather than sending 0. They answer in `content` with chain-of-thought in
  `reasoning_content`.