vh
40a374b809
phi4-mini supersedes the granite-4-small llama-swap pin as the summarizer + dreaming agent. New vllm-phi4 service: Phi-4-mini-instruct, vLLM-native FP8 (near-lossless on RTX 6000 Ada cc 8.9), 50K ctx, FP8 KV cache, GPU 1, :8004. llama-swap: removed granite-4-small (depinned) + granite-4-micro config — both superseded. CONFIG ONLY; the GGUFs stay on disk. Frees granite-4-small's ~24 GB (it was pinned at 120K ctx). Placement: phi4 on GPU 1 with the embed/rerank/reward trio (~1.2 GB margin at 50K); keeps GPU 0 clear for llama-swap heavy models. docs/roadmap.md captures the deferred vLLM observability (Langfuse req/resp tracing + Prometheus/Grafana). Deploy order: llama-swap config (free granite) -> vllm-phi4 -> repoint nevermore.
2.2 KiB
PFI infra roadmap
Deferred / planned infrastructure work. Not a ticket tracker — a durable list of "we decided to do this, later" items so they don't get lost.
Observability for the vLLM stack (ana-ml2)
Context: surfaced 2026-06-04 during the phi4-mini (summarizer/dreaming agent) deploy. vLLM has no built-in web UI for logs or model state the way llama-swap does. Dozzle (already running) shows vLLM's stdout = connection/request-metadata only — not full request/response bodies. Two complementary layers fill the gap:
1. Langfuse — request/response tracing (PRIORITY)
The thing llama-swap's UI gave us and vLLM doesn't: see the entire request and response per call, in a browser. Langfuse (open-source, self-hostable) captures every call's full prompt + full completion + tokens + latency + cost, with a polished trace UI — a strict upgrade over llama-swap's raw log dump.
- Where: a stack on ana-docker (alongside the other hubs — Dozzle, Beszel, gitea, etc.).
- How: vLLM services instrumented or fronted by it. Either point the consuming agents (nevermore, the dreaming agent, etc.) at a LiteLLM proxy that logs to Langfuse, or instrument the clients directly.
- Why it matters: phi4-mini is becoming a production summarizer + dreaming agent; being able to inspect exactly what it was asked and what it answered is the difference between debuggable and opaque.
2. Prometheus + Grafana — operational metrics
vLLM natively exposes a Prometheus /metrics endpoint (throughput,
time-to-first-token, KV-cache utilization, queue depth, running/waiting
requests) and ships official Grafana dashboards. We have Beszel for
coarse host/GPU stats but no app-layer inference metrics.
- Where: Prometheus + Grafana stack on ana-docker, scraping ana-ml2's
vLLM
:metricsports (and reusable for any future vLLM service). - Why: tells us if phi4 (or the embed/rerank/reward trio) is KV-cache-bound, queueing, or has latency regressions — the operational view llama-swap's UI only hinted at.
Status: deferred during the phi4-mini deploy; do after phi4 is live. Langfuse first (the operator specifically wants full req/resp visibility).