vh
83b2ec1a8a
LiteLLM proxy fronting the vLLM services on ana-ml2 so every request + response is captured and inspectable in a browser Logs UI — the visibility vLLM itself lacks (Dozzle shows only connection metadata). - compose: litellm (proxy + /ui Logs) + litellm-db (Postgres store) - conf/config.yaml: routes phi4-mini (chat, :8004), qwen3-embedding (:8001), qwen3-reranker (:8002); store_prompts_in_spend_logs persists full prompt/completion text. reward classifier (:8003) stays direct (no first-class LiteLLM route). - Langfuse-ready: lean first cut intentionally skips Langfuse's heavy v3 stack; graduating is one env-var + callback step, no re-architecture. - roadmap: mark the vLLM-observability item's first cut as shipped. Lean first cut of docs/roadmap.md "Observability for the vLLM stack".
2.6 KiB
PFI infra roadmap
Deferred / planned infrastructure work. Not a ticket tracker — a durable list of "we decided to do this, later" items so they don't get lost.
Observability for the vLLM stack (ana-ml2)
Context: surfaced 2026-06-04 during the phi4-mini (summarizer/dreaming agent) deploy. vLLM has no built-in web UI for logs or model state the way llama-swap does. Dozzle (already running) shows vLLM's stdout = connection/request-metadata only — not full request/response bodies. Two complementary layers fill the gap:
1. Langfuse — request/response tracing (PRIORITY)
The thing llama-swap's UI gave us and vLLM doesn't: see the entire request and response per call, in a browser. Langfuse (open-source, self-hostable) captures every call's full prompt + full completion + tokens + latency + cost, with a polished trace UI — a strict upgrade over llama-swap's raw log dump.
- Where: a stack on ana-docker (alongside the other hubs — Dozzle, Beszel, gitea, etc.).
- How: vLLM services instrumented or fronted by it. Either point the consuming agents (nevermore, the dreaming agent, etc.) at a LiteLLM proxy that logs to Langfuse, or instrument the clients directly.
- Why it matters: phi4-mini is becoming a production summarizer + dreaming agent; being able to inspect exactly what it was asked and what it answered is the difference between debuggable and opaque.
2. Prometheus + Grafana — operational metrics
vLLM natively exposes a Prometheus /metrics endpoint (throughput,
time-to-first-token, KV-cache utilization, queue depth, running/waiting
requests) and ships official Grafana dashboards. We have Beszel for
coarse host/GPU stats but no app-layer inference metrics.
- Where: Prometheus + Grafana stack on ana-docker, scraping ana-ml2's
vLLM
:metricsports (and reusable for any future vLLM service). - Why: tells us if phi4 (or the embed/rerank/reward trio) is KV-cache-bound, queueing, or has latency regressions — the operational view llama-swap's UI only hinted at.
Status: phi4 is live. Lean first cut shipped 2026-06-04 — the
litellm stack (stacks/litellm/, ana-docker) is the req/resp logging
gateway: full prompt/completion captured in a browser Logs UI, fronting
the vLLM services on ana-ml2. It is Langfuse-ready (one env-var +
callback step graduates it to full Langfuse traces, no re-architecture).
Remaining: (a) re-point consumers (nevermore, dreaming agent,
asset-engine) at the gateway; (b) stand up Langfuse + flip the callback
when the polished trace UI is wanted; (c) the Prometheus + Grafana
operational-metrics layer above.