Operator: update LiteLLM to latest and repull; get rid of the spend-log
DB and cap its growth.
Upgrade: pinned v1.97.0 (latest stable point release; v1.98.0-rc.1 skipped
as a pre-release on the fleet gateway, v1.97.0-stable not yet cut). Image
pre-pulled, DB pg_dump'd (1.8GB gz, keys+config+schema) and .env backed up
before the Prisma migration, which applied cleanly.
DB was 6.08 GB, 6.02 GB of it LiteLLM_SpendLogs storing full prompt+
completion bodies (store_prompts_in_spend_logs: true). Purged via TRUNCATE
on the running 1.91 BEFORE the upgrade so the schema migration ran against
an empty table -- 6081 MB -> 16 MB, keys (32) and models (3) intact.
'Get rid of the db' read as the spend-log DATA, not the database: dropping
it would have destroyed every virtual key (incl. the Lobe key) and the
model config in the same DB.
Cap: store_prompts_in_spend_logs -> false (bodies no longer persisted;
lightweight cost/usage rows and cross-project spend tracking survive) plus
maximum_spend_logs_retention_period 7d / interval 1d as a hard age bound.
Verified post-upgrade: v1.97.0 running, liveliness 200, 31-model roster,
chat round-trip on master + scoped Lobe key, key scoping still enforced
(glm-5.2 blocked), ext-tts 200, and store_prompts confirmed off (a marked
prompt persisted 0 bodies; 4 lightweight rows). Rollback: .env
LITELLM_TAG=v1.91.0 + the 1.8GB dump, both on the host.
The Heid panel plan uses Kimi's coding endpoint, not the general Moonshot API.
kimi-k3 now → openai/k3 @ https://api.kimi.com/coding/v1 (KIMI_CODE_API_KEY,
Vivace); the original general-endpoint entry is kept as kimi-k3-gen-api
(api.moonshot.ai, MOONSHOT_API_KEY). Both verified live through the gateway.
Same k3 constraints on both: temperature MUST be 1 (else 400), reasoning model
(reasoning_content vs content, needs adequate max_tokens).
Adds model_name kimi-k3 → openai/kimi-k3 @ https://api.moonshot.ai/v1
(OpenAI-compatible), keyed by MOONSHOT_API_KEY (compose env + .env.example
placeholder; real key on server only). Verified live through the gateway.
Two Moonshot constraints captured in the config comment + pinned: K3 accepts
ONLY temperature=1 (else 400), and it is a reasoning model (CoT in
reasoning_content, answer in content — needs adequate max_tokens or content
returns empty). Model id confirmed via /v1/models.
Adds explicit gateway entries for the four z.ai GLM models (glm-5.1,
glm-5-turbo, glm-4.7, glm-4.5-air) routed to api.z.ai with Z_AI_API_KEY,
plus the compose env passthrough + .env.example doc. Explicit entries
win over the llama-swap wildcard (distinct IDs, no collision). Extends
the gateway's unified logging to cloud inference, not just local
vLLM/llama-swap.
Cost note: paid API — only gateway-keyed callers reach these, but calls
spend z.ai credits (documented in config + compose comments).
LiteLLM proxy fronting the vLLM services on ana-ml2 so every request +
response is captured and inspectable in a browser Logs UI — the
visibility vLLM itself lacks (Dozzle shows only connection metadata).
- compose: litellm (proxy + /ui Logs) + litellm-db (Postgres store)
- conf/config.yaml: routes phi4-mini (chat, :8004), qwen3-embedding
(:8001), qwen3-reranker (:8002); store_prompts_in_spend_logs persists
full prompt/completion text. reward classifier (:8003) stays direct
(no first-class LiteLLM route).
- Langfuse-ready: lean first cut intentionally skips Langfuse's heavy v3
stack; graduating is one env-var + callback step, no re-architecture.
- roadmap: mark the vLLM-observability item's first cut as shipped.
Lean first cut of docs/roadmap.md "Observability for the vLLM stack".
vh