feat(litellm): add vLLM request/response logging gateway on ana-docker
LiteLLM proxy fronting the vLLM services on ana-ml2 so every request + response is captured and inspectable in a browser Logs UI — the visibility vLLM itself lacks (Dozzle shows only connection metadata). - compose: litellm (proxy + /ui Logs) + litellm-db (Postgres store) - conf/config.yaml: routes phi4-mini (chat, :8004), qwen3-embedding (:8001), qwen3-reranker (:8002); store_prompts_in_spend_logs persists full prompt/completion text. reward classifier (:8003) stays direct (no first-class LiteLLM route). - Langfuse-ready: lean first cut intentionally skips Langfuse's heavy v3 stack; graduating is one env-var + callback step, no re-architecture. - roadmap: mark the vLLM-observability item's first cut as shipped. Lean first cut of docs/roadmap.md "Observability for the vLLM stack".
This commit is contained in:
@@ -0,0 +1,38 @@
|
||||
# LiteLLM gateway — copy to .env on ana-docker and fill the secrets.
|
||||
# Real .env lives on the server only (gitignored); never commit it.
|
||||
|
||||
# Image tag. main-stable is the rolling stable; pin to a dated/SHA tag
|
||||
# (e.g. main-v1.74.0-stable) once a known-good build is confirmed.
|
||||
LITELLM_TAG=main-stable
|
||||
|
||||
# Publish. Bind to all interfaces on the LAN; 4000 is the LiteLLM default
|
||||
# (proxy API + admin/Logs UI at /ui).
|
||||
LITELLM_BIND=0.0.0.0
|
||||
LITELLM_PORT=4000
|
||||
|
||||
# Proxy + admin-UI master key. MUST start with "sk-". Generate one:
|
||||
# openssl rand -hex 24 | sed 's/^/sk-/'
|
||||
LITELLM_MASTER_KEY=
|
||||
|
||||
# Salt for encrypting any virtual/model keys stored in Postgres. Generate:
|
||||
# openssl rand -hex 32
|
||||
# Changing this after keys are stored makes them undecryptable — set once.
|
||||
LITELLM_SALT_KEY=
|
||||
|
||||
# Postgres (backing store for spend logs + keys). Password is required;
|
||||
# user/db default to "litellm".
|
||||
POSTGRES_USER=litellm
|
||||
POSTGRES_DB=litellm
|
||||
POSTGRES_PASSWORD=
|
||||
|
||||
# Upstream vLLM API key. The vllm stack on ana-ml2 ships API_KEY= (empty)
|
||||
# by default → leave this blank. Set it ONLY if you set API_KEY in the
|
||||
# vllm stack's .env.
|
||||
VLLM_API_KEY=
|
||||
|
||||
# --- Langfuse-ready (leave blank for the lean first cut) ---
|
||||
# Fill these AND uncomment success/failure_callback in conf/config.yaml
|
||||
# to ship full traces to a Langfuse instance. That is the whole upgrade.
|
||||
LANGFUSE_PUBLIC_KEY=
|
||||
LANGFUSE_SECRET_KEY=
|
||||
LANGFUSE_HOST=
|
||||
Reference in New Issue
Block a user