# PFI infra roadmap

Deferred / planned infrastructure work. Not a ticket tracker — a durable
list of "we decided to do this, later" items so they don't get lost.

## Observability for the vLLM stack (ana-ml2)

Context: surfaced 2026-06-04 during the phi4-mini (summarizer/dreaming
agent) deploy. vLLM has **no built-in web UI** for logs or model state the
way llama-swap does. Dozzle (already running) shows vLLM's stdout =
connection/request-metadata only — **not** full request/response bodies.
Two complementary layers fill the gap:

### 1. Langfuse — request/response tracing (PRIORITY)

The thing llama-swap's UI gave us and vLLM doesn't: **see the entire
request and response** per call, in a browser. Langfuse (open-source,
self-hostable) captures every call's full prompt + full completion +
tokens + latency + cost, with a polished trace UI — a strict upgrade over
llama-swap's raw log dump.

- **Where:** a stack on ana-docker (alongside the other hubs — Dozzle,
  Beszel, gitea, etc.).
- **How:** vLLM services instrumented or fronted by it. Either point the
  consuming agents (nevermore, the dreaming agent, etc.) at a **LiteLLM
  proxy** that logs to Langfuse, or instrument the clients directly.
- **Why it matters:** phi4-mini is becoming a production summarizer +
  dreaming agent; being able to inspect exactly what it was asked and what
  it answered is the difference between debuggable and opaque.

### 2. Prometheus + Grafana — operational metrics

vLLM natively exposes a Prometheus `/metrics` endpoint (throughput,
time-to-first-token, KV-cache utilization, queue depth, running/waiting
requests) and ships **official Grafana dashboards**. We have Beszel for
coarse host/GPU stats but no app-layer inference metrics.

- **Where:** Prometheus + Grafana stack on ana-docker, scraping ana-ml2's
  vLLM `:metrics` ports (and reusable for any future vLLM service).
- **Why:** tells us if phi4 (or the embed/rerank/reward trio) is
  KV-cache-bound, queueing, or has latency regressions — the operational
  view llama-swap's UI only hinted at.

**Status:** phi4 is live. **Lean first cut shipped 2026-06-04** — the
`litellm` stack (`stacks/litellm/`, ana-docker) is the req/resp logging
gateway: full prompt/completion captured in a browser Logs UI, fronting
the vLLM services on ana-ml2. It is **Langfuse-ready** (one env-var +
callback step graduates it to full Langfuse traces, no re-architecture).
Remaining: (a) re-point consumers (nevermore, dreaming agent,
asset-engine) at the gateway; (b) stand up Langfuse + flip the callback
when the polished trace UI is wanted; (c) the Prometheus + Grafana
operational-metrics layer above.