vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(litellm): strip empty tools:[] before forwarding to vLLM

Browse Source 
vLLM's OpenAI server 400s on an empty tools array ("tools must not be an
empty array"), which broke every gateway call carrying tools:[] (clients
that send it to mean "no tools" -- OpenAI tolerates it, vLLM does not).
drop_params doesn't help: it drops unsupported PARAMS, not empty VALUES.

Add a CustomLogger async_pre_call_hook (conf/strip_empty_tools.py) that
pops an empty/None tools field (+ orphaned tool_choice) before forwarding,
registered globally via litellm_settings.callbacks so it covers every
vLLM-backed model, not just mistral-small-4. Mounted at
/app/strip_empty_tools.py beside config.yaml (LiteLLM resolves callbacks
relative to the config dir). Surgical: only fires when tools is present
and empty; real tools pass through untouched.

Verified on live gateway (1.87.0): mistral-small-4 and granite-4.1-8b
with tools:[] now 200 (were 400); no-tools baseline unchanged; a real
tool still passes through.
This commit is contained in:
vh
2026-06-16 00:43:57 -07:00
parent f9277f5440
commit d1bea13994
4 changed files with 65 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/litellm/README.md
+10
View File
@@ -107,3 +107,13 @@ Then open `http://10.250.50.70:4000/ui` (log in with the master key) →
`API_KEY=` empty. Set it here only if you set it there.
- `LITELLM_SALT_KEY` must be set **once** and never changed — rotating it
makes any keys stored in Postgres undecryptable.
- **Empty `tools: []` stripping** — `conf/strip_empty_tools.py` is a pre-call
hook (registered via `litellm_settings.callbacks`) that drops an empty/None
`tools` field (and any orphaned `tool_choice`) before forwarding. vLLM 400s on
`tools: []` ("tools must not be an empty array"); `drop_params` doesn't catch
empty *values*, only unsupported params. It runs on **every** request, so all
vLLM-backed models are covered, and only fires when `tools` is present-and-empty
(real tools pass through untouched). The file mounts at `/app/strip_empty_tools.py`
beside `config.yaml` because LiteLLM resolves callbacks relative to the config
dir. Note: real tool-calls additionally need the upstream vLLM server launched
with `--enable-auto-tool-choice` — a vLLM-side flag, separate from this gateway.