feat(litellm): add vLLM request/response logging gateway on ana-docker
LiteLLM proxy fronting the vLLM services on ana-ml2 so every request + response is captured and inspectable in a browser Logs UI — the visibility vLLM itself lacks (Dozzle shows only connection metadata). - compose: litellm (proxy + /ui Logs) + litellm-db (Postgres store) - conf/config.yaml: routes phi4-mini (chat, :8004), qwen3-embedding (:8001), qwen3-reranker (:8002); store_prompts_in_spend_logs persists full prompt/completion text. reward classifier (:8003) stays direct (no first-class LiteLLM route). - Langfuse-ready: lean first cut intentionally skips Langfuse's heavy v3 stack; graduating is one env-var + callback step, no re-architecture. - roadmap: mark the vLLM-observability item's first cut as shipped. Lean first cut of docs/roadmap.md "Observability for the vLLM stack".
This commit is contained in:
+9
-2
@@ -41,5 +41,12 @@ coarse host/GPU stats but no app-layer inference metrics.
|
||||
KV-cache-bound, queueing, or has latency regressions — the operational
|
||||
view llama-swap's UI only hinted at.
|
||||
|
||||
**Status:** deferred during the phi4-mini deploy; do after phi4 is live.
|
||||
Langfuse first (the operator specifically wants full req/resp visibility).
|
||||
**Status:** phi4 is live. **Lean first cut shipped 2026-06-04** — the
|
||||
`litellm` stack (`stacks/litellm/`, ana-docker) is the req/resp logging
|
||||
gateway: full prompt/completion captured in a browser Logs UI, fronting
|
||||
the vLLM services on ana-ml2. It is **Langfuse-ready** (one env-var +
|
||||
callback step graduates it to full Langfuse traces, no re-architecture).
|
||||
Remaining: (a) re-point consumers (nevermore, dreaming agent,
|
||||
asset-engine) at the gateway; (b) stand up Langfuse + flip the callback
|
||||
when the polished trace UI is wanted; (c) the Prometheus + Grafana
|
||||
operational-metrics layer above.
|
||||
|
||||
Reference in New Issue
Block a user