glm-5.1 now disables GLM thinking by default via extra_body (LiteLLM strips
top-level thinking under drop_params but forwards extra_body verbatim to z.ai).
New glm-5.1-reasoning alias = same upstream with thinking enabled, so reasoning
is opt-in. Operator call 2026-06-11; primary driver is the pi coding harness.
Verified live: glm-5.1 reasoning_tokens=0, glm-5.1-reasoning reasoning_tokens>0.
LLM observability for the fleet — pretty trace UI over the gateway: prompts,
completions, reasoning, latency, token counts. The pretty layer LiteLLM's
spend_logs lacked.
- stacks/langfuse: v3 self-host stack (web/worker/postgres/clickhouse/redis/
minio) on ana-docker, adapted from upstream. UI on :3001 (gitea owns :3000).
Project + API keys auto-provisioned via LANGFUSE_INIT_*. HOSTNAME=0.0.0.0 on
langfuse-web so it's reachable via the published port while also on tnet.
- litellm: enabled success_callback/failure_callback: ["langfuse"] (the
passthrough env was already wired); keys + host go in the litellm .env.
Verified: stack healthy, project keys authenticate, and a real gateway call
landed a litellm-acompletion trace in Langfuse within ~6s. Secrets live only in
the server .env (never committed).
Granite 4.1 8B beat phi4-mini on precision in brokkr's R15 P03 eval, so it's
the new production summarizer/dreamer for nevermore.
- vllm-phi4 -> vllm-granite: official IBM FP8 (ibm-granite/granite-4.1-8b-fp8,
compressed-tensors), GPU 1, 50K ctx, FP8-KV, CUDA graphs. Same :8004 slot.
- GPU 1 retune: the embed/rerank/reward trio was over-provisioned (embed ran a
5.89x KV pool, reward 3.90x). Trimmed utils 0.20/0.20/0.30 -> 0.07/0.07/0.18,
freeing ~10 GB so granite runs with CUDA graphs (not --enforce-eager) and
keeps ~10 GB free as a hedge for future Granite text-LoRAs (--enable-lora).
- LiteLLM: phi4-mini model_list entry -> granite-4.1-8b (hosted_vllm @ :8004);
explicit entry shadows the '*' wildcard's llama-swap route.
- nevermore repointed (LLAMA_SWAP_MODEL=granite-4.1-8b via the gateway) live.
Verified end-to-end: vLLM :8004 generates, gateway routes (gateway-granite-ok),
KV 86,768 tokens/1.69x at 50K, 0 restarts, GPU 1 10.3 GB free.
Adds explicit gateway entries for the four z.ai GLM models (glm-5.1,
glm-5-turbo, glm-4.7, glm-4.5-air) routed to api.z.ai with Z_AI_API_KEY,
plus the compose env passthrough + .env.example doc. Explicit entries
win over the llama-swap wildcard (distinct IDs, no collision). Extends
the gateway's unified logging to cloud inference, not just local
vLLM/llama-swap.
Cost note: paid API — only gateway-keyed callers reach these, but calls
spend z.ai credits (documented in config + compose comments).
Adds a `model_name: "*"` entry routing any unmatched model to llama-swap
(ana-ml2:9292) so its whole swappable LLM zoo logs through the gateway
without per-model registration — add/swap models in llama-swap freely,
litellm logs them all. Exact entries (phi4-mini/qwen3-embedding/
qwen3-reranker → vLLM) still win; the wildcard only catches the rest.
litellm does no inference; llama-swap keeps loading + serving. Enables
routing worldtree-personal's generative chat through the gateway for
full req/resp logging while preserving llama-swap's on-demand swapping.
LiteLLM proxy fronting the vLLM services on ana-ml2 so every request +
response is captured and inspectable in a browser Logs UI — the
visibility vLLM itself lacks (Dozzle shows only connection metadata).
- compose: litellm (proxy + /ui Logs) + litellm-db (Postgres store)
- conf/config.yaml: routes phi4-mini (chat, :8004), qwen3-embedding
(:8001), qwen3-reranker (:8002); store_prompts_in_spend_logs persists
full prompt/completion text. reward classifier (:8003) stays direct
(no first-class LiteLLM route).
- Langfuse-ready: lean first cut intentionally skips Langfuse's heavy v3
stack; graduating is one env-var + callback step, no re-architecture.
- roadmap: mark the vLLM-observability item's first cut as shipped.
Lean first cut of docs/roadmap.md "Observability for the vLLM stack".
vh