Files

T

vh 21d24c50b4 docs(fleettools): autoload for Codex and Grok, and a vaulted gateway key

Codex reads a global AGENTS.md from CODEX_HOME; Grok always scans
~/.grok/rules/ and loads every *.md in it regardless of name. Both were
empty, so AGENT-BOOTSTRAP.md is symlinked into each rather than copied
— one file, three agent families, no drift surface.

The bootstrap is a pointer, not a second index: it names
~/FLEETTOOLS.md, gives the three live-inventory endpoints, and inlines
only the rules that must hold even if the agent never opens anything
else — attribution to Vuong Hoang, no committed secrets, the operator
owns architectural calls, n=1 is not a measurement, and absence of a
signal is not a safe reading of it.

The shared all-agents LiteLLM key was single-copy in ~/.claude/CLAUDE.md
and is now also in the vault at litellm/all-agents-shared-key, per the
standing directive that durable credentials never live in one place.
It stays inline in CLAUDE.md too, since every session needs it and a
vault round-trip measured over two minutes. Namespace is service-scoped
rather than host-prefixed because the key is fleet-wide, matching the
existing att/fortigate/headscale/unifi/worldtree entries.

2026-09-18 07:44:37 -07:00

2.7 KiB

Raw Blame History

LiteLLM gateway — the fleet's OpenAI-compatible front door

Base URL: http://10.250.50.70:4000/v1 (ana-docker) · Logs UI: :4000/ui Fronts the ana-ml2 / fv-ml1 / irv-ml1 vLLM seats plus paid vendor passthroughs.

⚠ Never hardcode the model roster — query it

curl -s -H "Authorization: Bearer $KEY" http://10.250.50.70:4000/v1/models \
  | python3 -c "import sys,json;[print(m['id']) for m in json.load(sys.stdin)['data']]"

The roster drifts constantly (40 entries as of 2026-09-18; a doc that names them goes stale within weeks). Any list written down here would be a second copy that rots.

Keys

The gateway 401s without a virtual API key. Three tiers:

key	where	reach
shared all-agents (`all-agents-local`)	`secret get litellm/all-agents-shared-key`, also inline in `~/.claude/CLAUDE.md` § Global tools	every model, paid passthroughs included
infra-ops admin	`~/.config/litellm/infra-ops-key` (0600)	admin operations
project-scoped	request from infra-ops via althing	isolated, auditable spend

⚠ The shared key is NOT free-local-only. It reaches paid GLM/z.ai and Kimi passthroughs, and a paid call on it spends real vendor credits shared across every session and project holding it. There is no isolation. For anything where "shows up on someone else's tab" matters, get a project-scoped key.

Stable aliases worth knowing

summarizer / summarizer-large / classifier — the always-available local seat for summarization, naming/labeling, classification and triage. Essentially free compute; reach for it before spending premium API tokens on work a small local model handles. Embeddings and rerankers (qwen3-embedding, qwen3-reranker, reranker) are local and free too. Anything named *-frontier, glm-* or kimi-* is PAID.

Traps

LiteLLM mutates shared deployment config in-place. Per-request sampler params bleed into the SHARED litellm_params for that deployment. Fix: give each variant a distinct served-model-name rather than reusing one with different params.
Retiring a model orphans scoped keys. A stale allowlist produces a silent per-endpoint 403. Audit /key/list + /key/info on every repoint.
The :4000/ui playground cannot test vLLM backends — it sends tools: [], which vLLM rejects with a 400. Use a chat frontend (Open WebUI, Gateway Chat) instead.
"Random model calls" in the logs are usually AUTH FAILURES — empty api_key or a missing Bearer prefix, showing as zero-spend 401s.
Kimi models 400 on any temperature other than 1 — omit the field entirely rather than sending 0. They answer in content with chain-of-thought in reasoning_content.

2.7 KiB Raw Blame History