 vh
|
40a374b809
|
feat(vllm): add phi4-mini FP8 summarizer/dreamer on ana-ml2; retire granite from llama-swap
phi4-mini supersedes the granite-4-small llama-swap pin as the summarizer +
dreaming agent. New vllm-phi4 service: Phi-4-mini-instruct, vLLM-native FP8
(near-lossless on RTX 6000 Ada cc 8.9), 50K ctx, FP8 KV cache, GPU 1, :8004.
llama-swap: removed granite-4-small (depinned) + granite-4-micro config —
both superseded. CONFIG ONLY; the GGUFs stay on disk. Frees granite-4-small's
~24 GB (it was pinned at 120K ctx).
Placement: phi4 on GPU 1 with the embed/rerank/reward trio (~1.2 GB margin at
50K); keeps GPU 0 clear for llama-swap heavy models. docs/roadmap.md captures
the deferred vLLM observability (Langfuse req/resp tracing + Prometheus/Grafana).
Deploy order: llama-swap config (free granite) -> vllm-phi4 -> repoint nevermore.
|
2026-06-03 23:21:45 -07:00 |
|