feat(litellm): add vLLM request/response logging gateway on ana-docker

LiteLLM proxy fronting the vLLM services on ana-ml2 so every request + response is captured and inspectable in a browser Logs UI — the visibility vLLM itself lacks (Dozzle shows only connection metadata). - compose: litellm (proxy + /ui Logs) + litellm-db (Postgres store) - conf/config.yaml: routes phi4-mini (chat, :8004), qwen3-embedding (:8001), qwen3-reranker (:8002); store_prompts_in_spend_logs persists full prompt/completion text. reward classifier (:8003) stays direct (no first-class LiteLLM route). - Langfuse-ready: lean first cut intentionally skips Langfuse's heavy v3 stack; graduating is one env-var + callback step, no re-architecture. - roadmap: mark the vLLM-observability item's first cut as shipped. Lean first cut of docs/roadmap.md "Observability for the vLLM stack".
2026-06-04 01:17:50 -07:00
parent 1dba05d8c8
commit 83b2ec1a8a
5 changed files with 318 additions and 2 deletions
@@ -0,0 +1,65 @@
+# LiteLLM gateway config — fronts the vLLM services on ana-ml2
+# (10.250.50.54) and logs every request + response so they're
+# inspectable in the Logs UI at http://10.250.50.70:4000/ui.
+#
+# Deploys to /opt/docker/conf/litellm/config.yaml (mounted read-only
+# into the container at /app/config.yaml).
+#
+# Model-name → upstream vLLM port mapping:
+#   phi4-mini        → :8004  (generative chat — the priority target)
+#   qwen3-embedding  → :8001  (/v1/embeddings)
+#   qwen3-reranker   → :8002  (/rerank)
+#
+# The vllm-reward classifier (:8003) is a pooling /classify endpoint
+# with no first-class LiteLLM route — left direct; see README. The
+# generative model is the one whose prompts/completions we most need
+# to read, and it routes cleanly here.
+
+model_list:
+  # --- Phi-4-mini (generative chat) — production summarizer + dreaming
+  #     agent. Full prompt + completion captured per call. ---
+  - model_name: phi4-mini
+    litellm_params:
+      model: hosted_vllm/phi4-mini
+      api_base: http://10.250.50.54:8004/v1
+      api_key: os.environ/VLLM_API_KEY
+    model_info:
+      mode: chat
+
+  # --- Qwen3 embeddings ---
+  - model_name: qwen3-embedding
+    litellm_params:
+      model: hosted_vllm/Qwen/Qwen3-Embedding-0.6B
+      api_base: http://10.250.50.54:8001/v1
+      api_key: os.environ/VLLM_API_KEY
+    model_info:
+      mode: embedding
+
+  # --- Qwen3 reranker (proxy /rerank route) ---
+  - model_name: qwen3-reranker
+    litellm_params:
+      model: hosted_vllm/Qwen/Qwen3-Reranker-0.6B
+      api_base: http://10.250.50.54:8002/v1
+      api_key: os.environ/VLLM_API_KEY
+    model_info:
+      mode: rerank
+
+general_settings:
+  master_key: os.environ/LITELLM_MASTER_KEY
+  database_url: os.environ/DATABASE_URL
+  store_model_in_db: true
+  # THE log switch: persists full request messages + response bodies into
+  # SpendLogs so they render in the Logs UI. Without this you get metadata
+  # (tokens, latency, model) but not the prompt/completion text.
+  store_prompts_in_spend_logs: true
+
+litellm_settings:
+  # vLLM rejects some OpenAI params other backends accept; drop silently
+  # rather than 400 the caller.
+  drop_params: true
+  # --- Langfuse-ready: uncomment to ship full traces to a Langfuse
+  #     instance (set LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY /
+  #     LANGFUSE_HOST in .env). This is the entire upgrade — the gateway
+  #     and every consumer stay pointed here. ---
+  # success_callback: ["langfuse"]
+  # failure_callback: ["langfuse"]