fix(litellm): strip empty tools:[] before forwarding to vLLM

vLLM's OpenAI server 400s on an empty tools array ("tools must not be an
empty array"), which broke every gateway call carrying tools:[] (clients
that send it to mean "no tools" -- OpenAI tolerates it, vLLM does not).
drop_params doesn't help: it drops unsupported PARAMS, not empty VALUES.

Add a CustomLogger async_pre_call_hook (conf/strip_empty_tools.py) that
pops an empty/None tools field (+ orphaned tool_choice) before forwarding,
registered globally via litellm_settings.callbacks so it covers every
vLLM-backed model, not just mistral-small-4. Mounted at
/app/strip_empty_tools.py beside config.yaml (LiteLLM resolves callbacks
relative to the config dir). Surgical: only fires when tools is present
and empty; real tools pass through untouched.

Verified on live gateway (1.87.0): mistral-small-4 and granite-4.1-8b
with tools:[] now 200 (were 400); no-tools baseline unchanged; a real
tool still passes through.
This commit is contained in:
vh
2026-06-16 00:43:57 -07:00
parent f9277f5440
commit d1bea13994
4 changed files with 65 additions and 0 deletions
+7
View File
@@ -197,6 +197,13 @@ litellm_settings:
# vLLM rejects some OpenAI params other backends accept; drop silently
# rather than 400 the caller.
drop_params: true
# Custom pre-call hook: strip an empty `tools: []` (+ orphaned tool_choice)
# before forwarding upstream. vLLM 400s on empty tools arrays ("tools must
# not be an empty array"); drop_params doesn't catch empty VALUES, only
# unsupported params. Runs on every request → fixes it for all vLLM models.
# File mounted at /app/strip_empty_tools.py; reference is module.instance,
# resolved relative to this config's directory.
callbacks: ["strip_empty_tools.strip_empty_tools_instance"]
# --- Langfuse trace export (live 2026-06-05). Full prompt/completion +
# reasoning + tok-derivable latency traces ship to the Langfuse stack on
# ana-docker (project "gateway"). Keys + host in .env. The gateway and