Duplicate-entry aliases (NOT router_settings.model_group_alias — that's hidden from
/v1/models and can be silently ignored in config per litellm #15020/#5524):
- summarizer -> granite-4.1-8b
- gen -> mistral-small-4
- gen-reasoning -> mistral-small-4-reasoning (reasoning_effort:high preserved)
Each alias is a real model_name co-located with its target (keep api_base in sync).
Verified live: all 3 in /v1/models + route end-to-end; gen-reasoning returns
reasoning_content.
GLM 5.2 released ~2026-06; confirmed reachable with our existing
Z_AI_API_KEY (z.ai /models lists glm-5.2; a live completion returned
clean). Added two model_list entries mirroring the glm-5.1 pattern:
glm-5.2 (thinking DISABLED by default, per the 2026-06-11 operator call)
and glm-5.2-reasoning (thinking ENABLED, opt-in). Deployed to
ana-docker /opt/docker/conf/litellm/config.yaml, litellm restarted,
both verified through the gateway (disabled -> reasoning_tokens 0;
reasoning -> 234).
vLLM's OpenAI server 400s on an empty tools array ("tools must not be an
empty array"), which broke every gateway call carrying tools:[] (clients
that send it to mean "no tools" -- OpenAI tolerates it, vLLM does not).
drop_params doesn't help: it drops unsupported PARAMS, not empty VALUES.
Add a CustomLogger async_pre_call_hook (conf/strip_empty_tools.py) that
pops an empty/None tools field (+ orphaned tool_choice) before forwarding,
registered globally via litellm_settings.callbacks so it covers every
vLLM-backed model, not just mistral-small-4. Mounted at
/app/strip_empty_tools.py beside config.yaml (LiteLLM resolves callbacks
relative to the config dir). Surgical: only fires when tools is present
and empty; real tools pass through untouched.
Verified on live gateway (1.87.0): mistral-small-4 and granite-4.1-8b
with tools:[] now 200 (were 400); no-tools baseline unchanged; a real
tool still passes through.
selene: AtlaAI Selene-1-Mini-Llama-3.1-8B judge restored on vLLM after the
llama-swap teardown took its Q6_K GGUF offline. FP8 (dynamic --quantization
fp8; FP8 >= the validated Q6_K fidelity, and text-only Llama so no vision-
tower-noise risk; NVFP4's W4A4 too aggressive for a precision judge). GPU1
util 0.13 (8.51 GiB weights + 2.53 GiB KV, 32K ctx, 1.27x concurrency),
~11 GB GPU1 buffer left. Gateway selene-1-mini-8b → :8011 (shadows the *
wildcard that used to reach it via llama-swap). Judge smoke: scored an
unfaithful claim 1/5 correctly.
mistral-small-4: max-model-len 131072 → 262144 (full native 256K) for
novel-length consistency-checking. KV pool is util-bound (~862K tokens), so
256K costs no extra VRAM — max concurrency just drops to 3.29x at full length.
max-num-seqs 64 → 32 keeps the warmup transient flat (scales with seqs × len),
so it fits the tight GPU0 (free unchanged at 5.2 GB). Verified loaded + healthy.
Operator needs a verified-working vision tower as the abliteration/tuning
baseline. vLLM 0.23.0 crashes Mistral multimodal at startup (#44911
fetch_images regression, ~0.22.1+). Pinned the Mistral container to
v0.22.0 — the last pre-regression release — which loads the NVFP4
(compressed-tensors) AND serves vision: verified a half-blue/half-red
image read correctly ('left blue, right red'). Dropped --limit-mm
(vision re-enabled). qwen36 stays on 0.23.0 (separate container; needs it
for its ModelOpt NVFP4).
- gateway: add mistral-small-4-reasoning. Operator asked for effort=medium
but Mistral's reasoning_effort is BINARY (none/high only — medium 400s);
set to 'high' (sole reasoning-ON level). NOTE: reasoning fires but
reasoning_content-splitting is unreliable on v0.22.0 (lands in content);
clean split would need 0.23.0, which breaks vision — vision prioritized.
- mistral-small-4 (instant) + mistral-small-4-reasoning both gateway-live.
Mistral-Small-4-119B-2603-NVFP4 (119B/6.5B-active MoE, 65.3 GiB) on the
freed GPU 0 (dedicated 96 GB Blackwell), vLLM 0.23.0, :8010. NVFP4 is the
only variant that fits one card (FP8 ~119 GB / bf16 ~238 GB need 2 GPUs).
- TEXT-ONLY: vLLM 0.23.0's Mistral multimodal processor crashes at startup
(fetch_images bug); loaded with --limit-mm-per-prompt image/video=0.
Remove the flag to restore vision once vLLM patches it.
- MLA attn (TRITON_MLA), mistral tool-call + reasoning parsers, util 0.93,
max-len 131072 (capped from native 256K), image pinned by 0.23.0 digest.
- litellm: mistral-small-4 → :8010, shadows the * wildcard.
- GPU 0 reassigned from the (now-offline) llama-swap zoo per operator.
The qwen3.6-35b-a3b VL checkpoint is a single hybrid model with a per-
request enable_thinking switch (Qwen3-style), defaulting thinking ON.
Make the default non-thinking and add an opt-in reasoning variant,
mirroring the existing glm-5.1 / glm-5.1-reasoning gateway split.
- qwen36-vl compose: add --reasoning-parser qwen3 (model-matched) so the
single :8007 endpoint splits <think> into reasoning_content when on and
routes all output to content when off — serving both modes cleanly.
- litellm gateway: base qwen3.6-35b-a3b pins chat_template_kwargs
enable_thinking=false (non-thinking default); new qwen3.6-35b-a3b-thinking
pins enable_thinking=true (opt-in reasoning). Same upstream checkpoint,
no extra VRAM/container.
Deployed + verified on ana-ml2 (vLLM recreated, healthy) and ana-docker
(litellm reloaded): default returns a direct answer with no reasoning_content;
-thinking returns cleanly-separated reasoning_content, no raw tag leak.
Retire qwen35-vl (Qwen3.5-9B); add qwen36-vl serving the official FP8
Qwen3.6-35B-A3B vision MoE on :8007 under its TRUE name only — no alias.
qwen3.5-9b-fp8 is killed at vLLM AND the litellm gateway (404/400); a model is
never served under a prior model's name. Consumer (comfy-dev/arbo) notified +
migrated; arbo vkeys flipped to all-proxy-models; shared all-agents-local key
repointed to qwen3.6-35b-a3b.
GPU-1 rebalance for the heavier FP8 weights (~34 GB): granite 0.35->0.24 /
131K->64K, embed/rerank 0.05->0.03 (reclaimed util-reservation waste). Verified:
vision correct, 20-concurrent/endpoint load test = no OOM (~7.5 GB headroom).
Drop the llama-swap qwen3.5-9b GPU-0 pin (GPU 0 freed for the creative-writing
hot-swap card). NVFP4 was the lighter fit (~21 GB) but its vLLM ModelOpt-MoE
loader is broken (KeyError w2_input_scale / lm_head.input_scale, vllm #44081);
revisit when fixed.
Qwen3.5-9B vision-language served FP8 on ana-ml2 GPU1 (co-located with
granite + the embed/rerank/reward trio; GPU0 kept free for hot-loading
large models), :8007, fronted by LiteLLM as qwen3.5-9b-fp8.
Pinned to vllm/vllm-openai nightly@sha256:49211ab2 — :latest (v0.19.1)
quantizes the VL vision tower under fp8 and garbles vision; the nightly
correctly excludes it (LM stays FP8, vision tower BF16). util 0.40
(~38GB) on the shared card (vLLM needs free>=util*total here). Vision
verified end-to-end through the gateway.
glm-5.1 now disables GLM thinking by default via extra_body (LiteLLM strips
top-level thinking under drop_params but forwards extra_body verbatim to z.ai).
New glm-5.1-reasoning alias = same upstream with thinking enabled, so reasoning
is opt-in. Operator call 2026-06-11; primary driver is the pi coding harness.
Verified live: glm-5.1 reasoning_tokens=0, glm-5.1-reasoning reasoning_tokens>0.
LLM observability for the fleet — pretty trace UI over the gateway: prompts,
completions, reasoning, latency, token counts. The pretty layer LiteLLM's
spend_logs lacked.
- stacks/langfuse: v3 self-host stack (web/worker/postgres/clickhouse/redis/
minio) on ana-docker, adapted from upstream. UI on :3001 (gitea owns :3000).
Project + API keys auto-provisioned via LANGFUSE_INIT_*. HOSTNAME=0.0.0.0 on
langfuse-web so it's reachable via the published port while also on tnet.
- litellm: enabled success_callback/failure_callback: ["langfuse"] (the
passthrough env was already wired); keys + host go in the litellm .env.
Verified: stack healthy, project keys authenticate, and a real gateway call
landed a litellm-acompletion trace in Langfuse within ~6s. Secrets live only in
the server .env (never committed).
Granite 4.1 8B beat phi4-mini on precision in brokkr's R15 P03 eval, so it's
the new production summarizer/dreamer for nevermore.
- vllm-phi4 -> vllm-granite: official IBM FP8 (ibm-granite/granite-4.1-8b-fp8,
compressed-tensors), GPU 1, 50K ctx, FP8-KV, CUDA graphs. Same :8004 slot.
- GPU 1 retune: the embed/rerank/reward trio was over-provisioned (embed ran a
5.89x KV pool, reward 3.90x). Trimmed utils 0.20/0.20/0.30 -> 0.07/0.07/0.18,
freeing ~10 GB so granite runs with CUDA graphs (not --enforce-eager) and
keeps ~10 GB free as a hedge for future Granite text-LoRAs (--enable-lora).
- LiteLLM: phi4-mini model_list entry -> granite-4.1-8b (hosted_vllm @ :8004);
explicit entry shadows the '*' wildcard's llama-swap route.
- nevermore repointed (LLAMA_SWAP_MODEL=granite-4.1-8b via the gateway) live.
Verified end-to-end: vLLM :8004 generates, gateway routes (gateway-granite-ok),
KV 86,768 tokens/1.69x at 50K, 0 restarts, GPU 1 10.3 GB free.
Adds explicit gateway entries for the four z.ai GLM models (glm-5.1,
glm-5-turbo, glm-4.7, glm-4.5-air) routed to api.z.ai with Z_AI_API_KEY,
plus the compose env passthrough + .env.example doc. Explicit entries
win over the llama-swap wildcard (distinct IDs, no collision). Extends
the gateway's unified logging to cloud inference, not just local
vLLM/llama-swap.
Cost note: paid API — only gateway-keyed callers reach these, but calls
spend z.ai credits (documented in config + compose comments).
Adds a `model_name: "*"` entry routing any unmatched model to llama-swap
(ana-ml2:9292) so its whole swappable LLM zoo logs through the gateway
without per-model registration — add/swap models in llama-swap freely,
litellm logs them all. Exact entries (phi4-mini/qwen3-embedding/
qwen3-reranker → vLLM) still win; the wildcard only catches the rest.
litellm does no inference; llama-swap keeps loading + serving. Enables
routing worldtree-personal's generative chat through the gateway for
full req/resp logging while preserving llama-swap's on-demand swapping.
LiteLLM proxy fronting the vLLM services on ana-ml2 so every request +
response is captured and inspectable in a browser Logs UI — the
visibility vLLM itself lacks (Dozzle shows only connection metadata).
- compose: litellm (proxy + /ui Logs) + litellm-db (Postgres store)
- conf/config.yaml: routes phi4-mini (chat, :8004), qwen3-embedding
(:8001), qwen3-reranker (:8002); store_prompts_in_spend_logs persists
full prompt/completion text. reward classifier (:8003) stays direct
(no first-class LiteLLM route).
- Langfuse-ready: lean first cut intentionally skips Langfuse's heavy v3
stack; graduating is one env-var + callback step, no re-architecture.
- roadmap: mark the vLLM-observability item's first cut as shipped.
Lean first cut of docs/roadmap.md "Observability for the vLLM stack".
vh