vh
83b2ec1a8a
LiteLLM proxy fronting the vLLM services on ana-ml2 so every request + response is captured and inspectable in a browser Logs UI — the visibility vLLM itself lacks (Dozzle shows only connection metadata). - compose: litellm (proxy + /ui Logs) + litellm-db (Postgres store) - conf/config.yaml: routes phi4-mini (chat, :8004), qwen3-embedding (:8001), qwen3-reranker (:8002); store_prompts_in_spend_logs persists full prompt/completion text. reward classifier (:8003) stays direct (no first-class LiteLLM route). - Langfuse-ready: lean first cut intentionally skips Langfuse's heavy v3 stack; graduating is one env-var + callback step, no re-architecture. - roadmap: mark the vLLM-observability item's first cut as shipped. Lean first cut of docs/roadmap.md "Observability for the vLLM stack".
53 lines
2.6 KiB
Markdown
53 lines
2.6 KiB
Markdown
# PFI infra roadmap
|
|
|
|
Deferred / planned infrastructure work. Not a ticket tracker — a durable
|
|
list of "we decided to do this, later" items so they don't get lost.
|
|
|
|
## Observability for the vLLM stack (ana-ml2)
|
|
|
|
Context: surfaced 2026-06-04 during the phi4-mini (summarizer/dreaming
|
|
agent) deploy. vLLM has **no built-in web UI** for logs or model state the
|
|
way llama-swap does. Dozzle (already running) shows vLLM's stdout =
|
|
connection/request-metadata only — **not** full request/response bodies.
|
|
Two complementary layers fill the gap:
|
|
|
|
### 1. Langfuse — request/response tracing (PRIORITY)
|
|
|
|
The thing llama-swap's UI gave us and vLLM doesn't: **see the entire
|
|
request and response** per call, in a browser. Langfuse (open-source,
|
|
self-hostable) captures every call's full prompt + full completion +
|
|
tokens + latency + cost, with a polished trace UI — a strict upgrade over
|
|
llama-swap's raw log dump.
|
|
|
|
- **Where:** a stack on ana-docker (alongside the other hubs — Dozzle,
|
|
Beszel, gitea, etc.).
|
|
- **How:** vLLM services instrumented or fronted by it. Either point the
|
|
consuming agents (nevermore, the dreaming agent, etc.) at a **LiteLLM
|
|
proxy** that logs to Langfuse, or instrument the clients directly.
|
|
- **Why it matters:** phi4-mini is becoming a production summarizer +
|
|
dreaming agent; being able to inspect exactly what it was asked and what
|
|
it answered is the difference between debuggable and opaque.
|
|
|
|
### 2. Prometheus + Grafana — operational metrics
|
|
|
|
vLLM natively exposes a Prometheus `/metrics` endpoint (throughput,
|
|
time-to-first-token, KV-cache utilization, queue depth, running/waiting
|
|
requests) and ships **official Grafana dashboards**. We have Beszel for
|
|
coarse host/GPU stats but no app-layer inference metrics.
|
|
|
|
- **Where:** Prometheus + Grafana stack on ana-docker, scraping ana-ml2's
|
|
vLLM `:metrics` ports (and reusable for any future vLLM service).
|
|
- **Why:** tells us if phi4 (or the embed/rerank/reward trio) is
|
|
KV-cache-bound, queueing, or has latency regressions — the operational
|
|
view llama-swap's UI only hinted at.
|
|
|
|
**Status:** phi4 is live. **Lean first cut shipped 2026-06-04** — the
|
|
`litellm` stack (`stacks/litellm/`, ana-docker) is the req/resp logging
|
|
gateway: full prompt/completion captured in a browser Logs UI, fronting
|
|
the vLLM services on ana-ml2. It is **Langfuse-ready** (one env-var +
|
|
callback step graduates it to full Langfuse traces, no re-architecture).
|
|
Remaining: (a) re-point consumers (nevermore, dreaming agent,
|
|
asset-engine) at the gateway; (b) stand up Langfuse + flip the callback
|
|
when the polished trace UI is wanted; (c) the Prometheus + Grafana
|
|
operational-metrics layer above.
|