feat(litellm): add vLLM request/response logging gateway on ana-docker

LiteLLM proxy fronting the vLLM services on ana-ml2 so every request +
response is captured and inspectable in a browser Logs UI — the
visibility vLLM itself lacks (Dozzle shows only connection metadata).

- compose: litellm (proxy + /ui Logs) + litellm-db (Postgres store)
- conf/config.yaml: routes phi4-mini (chat, :8004), qwen3-embedding
  (:8001), qwen3-reranker (:8002); store_prompts_in_spend_logs persists
  full prompt/completion text. reward classifier (:8003) stays direct
  (no first-class LiteLLM route).
- Langfuse-ready: lean first cut intentionally skips Langfuse's heavy v3
  stack; graduating is one env-var + callback step, no re-architecture.
- roadmap: mark the vLLM-observability item's first cut as shipped.

Lean first cut of docs/roadmap.md "Observability for the vLLM stack".
This commit is contained in:
vh
2026-06-04 01:17:50 -07:00
parent 1dba05d8c8
commit 83b2ec1a8a
5 changed files with 318 additions and 2 deletions
+9 -2
View File
@@ -41,5 +41,12 @@ coarse host/GPU stats but no app-layer inference metrics.
KV-cache-bound, queueing, or has latency regressions — the operational
view llama-swap's UI only hinted at.
**Status:** deferred during the phi4-mini deploy; do after phi4 is live.
Langfuse first (the operator specifically wants full req/resp visibility).
**Status:** phi4 is live. **Lean first cut shipped 2026-06-04** — the
`litellm` stack (`stacks/litellm/`, ana-docker) is the req/resp logging
gateway: full prompt/completion captured in a browser Logs UI, fronting
the vLLM services on ana-ml2. It is **Langfuse-ready** (one env-var +
callback step graduates it to full Langfuse traces, no re-architecture).
Remaining: (a) re-point consumers (nevermore, dreaming agent,
asset-engine) at the gateway; (b) stand up Langfuse + flip the callback
when the polished trace UI is wanted; (c) the Prometheus + Grafana
operational-metrics layer above.