feat(litellm): add vLLM request/response logging gateway on ana-docker

LiteLLM proxy fronting the vLLM services on ana-ml2 so every request +
response is captured and inspectable in a browser Logs UI — the
visibility vLLM itself lacks (Dozzle shows only connection metadata).

- compose: litellm (proxy + /ui Logs) + litellm-db (Postgres store)
- conf/config.yaml: routes phi4-mini (chat, :8004), qwen3-embedding
  (:8001), qwen3-reranker (:8002); store_prompts_in_spend_logs persists
  full prompt/completion text. reward classifier (:8003) stays direct
  (no first-class LiteLLM route).
- Langfuse-ready: lean first cut intentionally skips Langfuse's heavy v3
  stack; graduating is one env-var + callback step, no re-architecture.
- roadmap: mark the vLLM-observability item's first cut as shipped.

Lean first cut of docs/roadmap.md "Observability for the vLLM stack".
This commit is contained in:
vh
2026-06-04 01:17:50 -07:00
parent 1dba05d8c8
commit 83b2ec1a8a
5 changed files with 318 additions and 2 deletions
+38
View File
@@ -0,0 +1,38 @@
# LiteLLM gateway — copy to .env on ana-docker and fill the secrets.
# Real .env lives on the server only (gitignored); never commit it.
# Image tag. main-stable is the rolling stable; pin to a dated/SHA tag
# (e.g. main-v1.74.0-stable) once a known-good build is confirmed.
LITELLM_TAG=main-stable
# Publish. Bind to all interfaces on the LAN; 4000 is the LiteLLM default
# (proxy API + admin/Logs UI at /ui).
LITELLM_BIND=0.0.0.0
LITELLM_PORT=4000
# Proxy + admin-UI master key. MUST start with "sk-". Generate one:
# openssl rand -hex 24 | sed 's/^/sk-/'
LITELLM_MASTER_KEY=
# Salt for encrypting any virtual/model keys stored in Postgres. Generate:
# openssl rand -hex 32
# Changing this after keys are stored makes them undecryptable — set once.
LITELLM_SALT_KEY=
# Postgres (backing store for spend logs + keys). Password is required;
# user/db default to "litellm".
POSTGRES_USER=litellm
POSTGRES_DB=litellm
POSTGRES_PASSWORD=
# Upstream vLLM API key. The vllm stack on ana-ml2 ships API_KEY= (empty)
# by default → leave this blank. Set it ONLY if you set API_KEY in the
# vllm stack's .env.
VLLM_API_KEY=
# --- Langfuse-ready (leave blank for the lean first cut) ---
# Fill these AND uncomment success/failure_callback in conf/config.yaml
# to ship full traces to a Langfuse instance. That is the whole upgrade.
LANGFUSE_PUBLIC_KEY=
LANGFUSE_SECRET_KEY=
LANGFUSE_HOST=