vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(qwen3.5-122b): enable tool-calling (--enable-auto-tool-choice --tool-call-parser qwen3_xml)

Browse Source 
gen/gen-reasoning tool-calling 400'd (operator + brokkr's capability battery both caught it):
the bjk110 serve command shipped --reasoning-parser qwen3 but no tool flags. Qwen3.5 emits XML
tool calls (<tool_call><function=NAME><parameter=K>V</parameter></function></tool_call>), NOT
Hermes JSON — so `hermes` mis-parsed to raw text; `qwen3_xml` is the correct parser. Reasoning +
tools coexist (gen-reasoning keeps its thinking split). Verified live: a get_weather request
returns tool_calls=[get_weather {"city":"Paris"}].
This commit is contained in:
vh
2026-06-19 01:56:56 -07:00
parent 5dfce049f4
commit a5b626b3d5
1 changed files with 5 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/qwen3.5-122b/compose.yaml
+5 -1
View File
@@ -40,7 +40,11 @@ services:
- MAX_NUM_BATCHED_TOKENS=${QWEN35_MAX_NUM_BATCHED_TOKENS:-32768} - MAX_NUM_BATCHED_TOKENS=${QWEN35_MAX_NUM_BATCHED_TOKENS:-32768}
# --reasoning-parser qwen3 surfaces <think>…</think> as reasoning_content; # --reasoning-parser qwen3 surfaces <think>…</think> as reasoning_content;
# the thinking on/off itself is per-request (litellm chat_template_kwargs). # the thinking on/off itself is per-request (litellm chat_template_kwargs).
- VLLM_EXTRA_ARGS=--reasoning-parser qwen3 --enable-chunked-prefill # --enable-auto-tool-choice + --tool-call-parser: Qwen3.5 emits XML tool calls
# <tool_call><function=NAME><parameter=K>V</parameter></function></tool_call>
# (NOT Hermes JSON), so the parser is qwen3_xml. Without these flags vLLM never
# parses tool calls (tool-calling is broken). The bjk110 repo command omitted them.
- VLLM_EXTRA_ARGS=--reasoning-parser qwen3 --enable-chunked-prefill --enable-auto-tool-choice --tool-call-parser qwen3_xml
- NVIDIA_VISIBLE_DEVICES=0 - NVIDIA_VISIBLE_DEVICES=0
- PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
- VLLM_API_KEY=${API_KEY:-} - VLLM_API_KEY=${API_KEY:-}