fix(litellm): strip empty tools:[] before forwarding to vLLM
vLLM's OpenAI server 400s on an empty tools array ("tools must not be an
empty array"), which broke every gateway call carrying tools:[] (clients
that send it to mean "no tools" -- OpenAI tolerates it, vLLM does not).
drop_params doesn't help: it drops unsupported PARAMS, not empty VALUES.
Add a CustomLogger async_pre_call_hook (conf/strip_empty_tools.py) that
pops an empty/None tools field (+ orphaned tool_choice) before forwarding,
registered globally via litellm_settings.callbacks so it covers every
vLLM-backed model, not just mistral-small-4. Mounted at
/app/strip_empty_tools.py beside config.yaml (LiteLLM resolves callbacks
relative to the config dir). Surgical: only fires when tools is present
and empty; real tools pass through untouched.
Verified on live gateway (1.87.0): mistral-small-4 and granite-4.1-8b
with tools:[] now 200 (were 400); no-tools baseline unchanged; a real
tool still passes through.
This commit is contained in:
@@ -197,6 +197,13 @@ litellm_settings:
|
||||
# vLLM rejects some OpenAI params other backends accept; drop silently
|
||||
# rather than 400 the caller.
|
||||
drop_params: true
|
||||
# Custom pre-call hook: strip an empty `tools: []` (+ orphaned tool_choice)
|
||||
# before forwarding upstream. vLLM 400s on empty tools arrays ("tools must
|
||||
# not be an empty array"); drop_params doesn't catch empty VALUES, only
|
||||
# unsupported params. Runs on every request → fixes it for all vLLM models.
|
||||
# File mounted at /app/strip_empty_tools.py; reference is module.instance,
|
||||
# resolved relative to this config's directory.
|
||||
callbacks: ["strip_empty_tools.strip_empty_tools_instance"]
|
||||
# --- Langfuse trace export (live 2026-06-05). Full prompt/completion +
|
||||
# reasoning + tok-derivable latency traces ship to the Langfuse stack on
|
||||
# ana-docker (project "gateway"). Keys + host in .env. The gateway and
|
||||
|
||||
Reference in New Issue
Block a user