vh
21d24c50b4
Codex reads a global AGENTS.md from CODEX_HOME; Grok always scans ~/.grok/rules/ and loads every *.md in it regardless of name. Both were empty, so AGENT-BOOTSTRAP.md is symlinked into each rather than copied — one file, three agent families, no drift surface. The bootstrap is a pointer, not a second index: it names ~/FLEETTOOLS.md, gives the three live-inventory endpoints, and inlines only the rules that must hold even if the agent never opens anything else — attribution to Vuong Hoang, no committed secrets, the operator owns architectural calls, n=1 is not a measurement, and absence of a signal is not a safe reading of it. The shared all-agents LiteLLM key was single-copy in ~/.claude/CLAUDE.md and is now also in the vault at litellm/all-agents-shared-key, per the standing directive that durable credentials never live in one place. It stays inline in CLAUDE.md too, since every session needs it and a vault round-trip measured over two minutes. Namespace is service-scoped rather than host-prefixed because the key is fleet-wide, matching the existing att/fortigate/headscale/unifi/worldtree entries.
53 lines
2.7 KiB
Markdown
53 lines
2.7 KiB
Markdown
# LiteLLM gateway — the fleet's OpenAI-compatible front door
|
|
|
|
**Base URL:** `http://10.250.50.70:4000/v1` (ana-docker) · **Logs UI:** `:4000/ui`
|
|
Fronts the ana-ml2 / fv-ml1 / irv-ml1 vLLM seats plus paid vendor passthroughs.
|
|
|
|
## ⚠ Never hardcode the model roster — query it
|
|
|
|
```sh
|
|
curl -s -H "Authorization: Bearer $KEY" http://10.250.50.70:4000/v1/models \
|
|
| python3 -c "import sys,json;[print(m['id']) for m in json.load(sys.stdin)['data']]"
|
|
```
|
|
|
|
The roster drifts constantly (40 entries as of 2026-09-18; a doc that names them goes
|
|
stale within weeks). Any list written down here would be a second copy that rots.
|
|
|
|
## Keys
|
|
|
|
The gateway **401s without a virtual API key**. Three tiers:
|
|
|
|
| key | where | reach |
|
|
|---|---|---|
|
|
| shared all-agents (`all-agents-local`) | `secret get litellm/all-agents-shared-key`, also inline in `~/.claude/CLAUDE.md` § Global tools | **every** model, paid passthroughs included |
|
|
| infra-ops admin | `~/.config/litellm/infra-ops-key` (0600) | admin operations |
|
|
| project-scoped | request from infra-ops via althing | isolated, auditable spend |
|
|
|
|
⚠ **The shared key is NOT free-local-only.** It reaches paid GLM/z.ai and Kimi
|
|
passthroughs, and **a paid call on it spends real vendor credits shared across every
|
|
session and project holding it.** There is no isolation. For anything where "shows up
|
|
on someone else's tab" matters, get a project-scoped key.
|
|
|
|
## Stable aliases worth knowing
|
|
|
|
`summarizer` / `summarizer-large` / `classifier` — the always-available local seat for
|
|
summarization, naming/labeling, classification and triage. Essentially free compute;
|
|
reach for it before spending premium API tokens on work a small local model handles.
|
|
Embeddings and rerankers (`qwen3-embedding`, `qwen3-reranker`, `reranker`) are local
|
|
and free too. Anything named `*-frontier`, `glm-*` or `kimi-*` is **PAID**.
|
|
|
|
## Traps
|
|
|
|
- **LiteLLM mutates shared deployment config in-place.** Per-request sampler params
|
|
bleed into the SHARED `litellm_params` for that deployment. Fix: give each variant a
|
|
distinct served-model-name rather than reusing one with different params.
|
|
- **Retiring a model orphans scoped keys.** A stale allowlist produces a silent
|
|
per-endpoint 403. Audit `/key/list` + `/key/info` on every repoint.
|
|
- **The `:4000/ui` playground cannot test vLLM backends** — it sends `tools: []`, which
|
|
vLLM rejects with a 400. Use a chat frontend (Open WebUI, Gateway Chat) instead.
|
|
- **"Random model calls" in the logs are usually AUTH FAILURES** — empty `api_key` or a
|
|
missing `Bearer ` prefix, showing as zero-spend 401s.
|
|
- **Kimi models 400 on any `temperature` other than 1** — omit the field entirely
|
|
rather than sending 0. They answer in `content` with chain-of-thought in
|
|
`reasoning_content`.
|