2 Commits

Author	SHA1	Message	Date
vh	83b2ec1a8a	feat(litellm): add vLLM request/response logging gateway on ana-docker ... LiteLLM proxy fronting the vLLM services on ana-ml2 so every request + response is captured and inspectable in a browser Logs UI — the visibility vLLM itself lacks (Dozzle shows only connection metadata). - compose: litellm (proxy + /ui Logs) + litellm-db (Postgres store) - conf/config.yaml: routes phi4-mini (chat, :8004), qwen3-embedding (:8001), qwen3-reranker (:8002); store_prompts_in_spend_logs persists full prompt/completion text. reward classifier (:8003) stays direct (no first-class LiteLLM route). - Langfuse-ready: lean first cut intentionally skips Langfuse's heavy v3 stack; graduating is one env-var + callback step, no re-architecture. - roadmap: mark the vLLM-observability item's first cut as shipped. Lean first cut of docs/roadmap.md "Observability for the vLLM stack".	2026-06-04 01:17:50 -07:00
vh	40a374b809	feat(vllm): add phi4-mini FP8 summarizer/dreamer on ana-ml2; retire granite from llama-swap ... phi4-mini supersedes the granite-4-small llama-swap pin as the summarizer + dreaming agent. New vllm-phi4 service: Phi-4-mini-instruct, vLLM-native FP8 (near-lossless on RTX 6000 Ada cc 8.9), 50K ctx, FP8 KV cache, GPU 1, :8004. llama-swap: removed granite-4-small (depinned) + granite-4-micro config — both superseded. CONFIG ONLY; the GGUFs stay on disk. Frees granite-4-small's ~24 GB (it was pinned at 120K ctx). Placement: phi4 on GPU 1 with the embed/rerank/reward trio (~1.2 GB margin at 50K); keeps GPU 0 clear for llama-swap heavy models. docs/roadmap.md captures the deferred vLLM observability (Langfuse req/resp tracing + Prometheus/Grafana). Deploy order: llama-swap config (free granite) -> vllm-phi4 -> repoint nevermore.	2026-06-03 23:21:45 -07:00