vh
40a374b809
phi4-mini supersedes the granite-4-small llama-swap pin as the summarizer + dreaming agent. New vllm-phi4 service: Phi-4-mini-instruct, vLLM-native FP8 (near-lossless on RTX 6000 Ada cc 8.9), 50K ctx, FP8 KV cache, GPU 1, :8004. llama-swap: removed granite-4-small (depinned) + granite-4-micro config — both superseded. CONFIG ONLY; the GGUFs stay on disk. Frees granite-4-small's ~24 GB (it was pinned at 120K ctx). Placement: phi4 on GPU 1 with the embed/rerank/reward trio (~1.2 GB margin at 50K); keeps GPU 0 clear for llama-swap heavy models. docs/roadmap.md captures the deferred vLLM observability (Langfuse req/resp tracing + Prometheus/Grafana). Deploy order: llama-swap config (free granite) -> vllm-phi4 -> repoint nevermore.
46 lines
2.2 KiB
Markdown
46 lines
2.2 KiB
Markdown
# PFI infra roadmap
|
|
|
|
Deferred / planned infrastructure work. Not a ticket tracker — a durable
|
|
list of "we decided to do this, later" items so they don't get lost.
|
|
|
|
## Observability for the vLLM stack (ana-ml2)
|
|
|
|
Context: surfaced 2026-06-04 during the phi4-mini (summarizer/dreaming
|
|
agent) deploy. vLLM has **no built-in web UI** for logs or model state the
|
|
way llama-swap does. Dozzle (already running) shows vLLM's stdout =
|
|
connection/request-metadata only — **not** full request/response bodies.
|
|
Two complementary layers fill the gap:
|
|
|
|
### 1. Langfuse — request/response tracing (PRIORITY)
|
|
|
|
The thing llama-swap's UI gave us and vLLM doesn't: **see the entire
|
|
request and response** per call, in a browser. Langfuse (open-source,
|
|
self-hostable) captures every call's full prompt + full completion +
|
|
tokens + latency + cost, with a polished trace UI — a strict upgrade over
|
|
llama-swap's raw log dump.
|
|
|
|
- **Where:** a stack on ana-docker (alongside the other hubs — Dozzle,
|
|
Beszel, gitea, etc.).
|
|
- **How:** vLLM services instrumented or fronted by it. Either point the
|
|
consuming agents (nevermore, the dreaming agent, etc.) at a **LiteLLM
|
|
proxy** that logs to Langfuse, or instrument the clients directly.
|
|
- **Why it matters:** phi4-mini is becoming a production summarizer +
|
|
dreaming agent; being able to inspect exactly what it was asked and what
|
|
it answered is the difference between debuggable and opaque.
|
|
|
|
### 2. Prometheus + Grafana — operational metrics
|
|
|
|
vLLM natively exposes a Prometheus `/metrics` endpoint (throughput,
|
|
time-to-first-token, KV-cache utilization, queue depth, running/waiting
|
|
requests) and ships **official Grafana dashboards**. We have Beszel for
|
|
coarse host/GPU stats but no app-layer inference metrics.
|
|
|
|
- **Where:** Prometheus + Grafana stack on ana-docker, scraping ana-ml2's
|
|
vLLM `:metrics` ports (and reusable for any future vLLM service).
|
|
- **Why:** tells us if phi4 (or the embed/rerank/reward trio) is
|
|
KV-cache-bound, queueing, or has latency regressions — the operational
|
|
view llama-swap's UI only hinted at.
|
|
|
|
**Status:** deferred during the phi4-mini deploy; do after phi4 is live.
|
|
Langfuse first (the operator specifically wants full req/resp visibility).
|