vh
d1bea13994
vLLM's OpenAI server 400s on an empty tools array ("tools must not be an
empty array"), which broke every gateway call carrying tools:[] (clients
that send it to mean "no tools" -- OpenAI tolerates it, vLLM does not).
drop_params doesn't help: it drops unsupported PARAMS, not empty VALUES.
Add a CustomLogger async_pre_call_hook (conf/strip_empty_tools.py) that
pops an empty/None tools field (+ orphaned tool_choice) before forwarding,
registered globally via litellm_settings.callbacks so it covers every
vLLM-backed model, not just mistral-small-4. Mounted at
/app/strip_empty_tools.py beside config.yaml (LiteLLM resolves callbacks
relative to the config dir). Surgical: only fires when tools is present
and empty; real tools pass through untouched.
Verified on live gateway (1.87.0): mistral-small-4 and granite-4.1-8b
with tools:[] now 200 (were 400); no-tools baseline unchanged; a real
tool still passes through.
45 lines
2.0 KiB
Python
45 lines
2.0 KiB
Python
# strip_empty_tools.py
|
|
#
|
|
# LiteLLM proxy pre-call hook: drop an empty / falsy `tools` field (and any
|
|
# now-orphaned `tool_choice`) before the request is forwarded upstream.
|
|
#
|
|
# Why: vLLM's OpenAI server validates that `tools`, when present, is non-empty.
|
|
# It 400s on `tools: []`:
|
|
# "Value error, tools must not be an empty array. Either provide at least one
|
|
# tool or omit the field entirely."
|
|
# Several clients send `tools: []` to mean "no tools" — OpenAI and most backends
|
|
# tolerate it; vLLM does not. `drop_params: true` does NOT help (it drops
|
|
# provider-UNSUPPORTED params, not empty VALUES), so we strip it at the gateway.
|
|
#
|
|
# Registered globally via litellm_settings.callbacks, so it runs on EVERY request
|
|
# regardless of model — covers all vLLM-backed routes (granite / qwen3.6 / mistral
|
|
# / selene / the llama-swap wildcard), not just one model. Harmless on backends
|
|
# that accept empty tools (we only remove a no-op field), and it only fires when
|
|
# the `tools` key is actually present and empty, so normal tool-less calls are
|
|
# untouched.
|
|
#
|
|
# LiteLLM loads callbacks RELATIVE TO THE CONFIG FILE'S DIRECTORY, so this file
|
|
# must sit beside config.yaml — mounted into the container at
|
|
# /app/strip_empty_tools.py (config.yaml is at /app/config.yaml).
|
|
|
|
from litellm.integrations.custom_logger import CustomLogger
|
|
from litellm._logging import verbose_proxy_logger
|
|
|
|
|
|
class StripEmptyTools(CustomLogger):
|
|
async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
|
|
# Only act when `tools` is present AND falsy ([] or None). Leaving the
|
|
# key absent for normal requests means this is a no-op on that path.
|
|
if isinstance(data, dict) and "tools" in data and not data["tools"]:
|
|
data.pop("tools", None)
|
|
data.pop("tool_choice", None)
|
|
verbose_proxy_logger.debug(
|
|
"strip_empty_tools: dropped empty `tools` (model=%s, call_type=%s)",
|
|
data.get("model"),
|
|
call_type,
|
|
)
|
|
return data
|
|
|
|
|
|
strip_empty_tools_instance = StripEmptyTools()
|