fix(erp-seat): tool_choice=none returned an empty turn — add --exclude-tools-when-tool-choice-none (vLLM kept the tools in the prompt, the model called one, parsing was off); 12-shape tool matrix green before/after

This commit is contained in:
vh
2026-09-08 23:01:26 -07:00
parent 32399d0da2
commit 7f6be8a56a
3 changed files with 18 additions and 0 deletions
+7
View File
@@ -12,6 +12,13 @@ checkpoint so the GX10 is free to train the next run. First occupant: **run 6**
and reasoning parsers, `enable_thinking` pinned false, the model's own stock template
(`ae53464b…`, the one it trained through). Without the reasoning parser the post-tool turn leaks
`<|channel>` markers; without the kwargs pin all prose lands in `reasoning_content`.
- **`tool_choice: "none"` trap (measured 2026-09-08, fixed with `--exclude-tools-when-tool-choice-none`).**
Without the flag vLLM still renders the tools into the prompt, the model emits a tool call
anyway, and because parsing is off for `none` the reply is `content: null, tool_calls: null`
— an empty turn, 3/3 reproductions. With the flag the tools are dropped from the prompt and
the model answers in prose (3/3). The rest of the matrix (auto / required / named / parallel /
nested schema / empty `tools: []` / streaming / tool-result round trip) was green before and
after. `stacks/gemma4-charrp` has the same exposure and does NOT carry the flag yet.
- **GPU1 is shared** — check real usage (`nvidia-smi --query-compute-apps=pid,used_memory`) before
raising `ERP_GPU_MEM_UTIL`; the flag sizes KV, not CUDA context.
- **Rollback / next run:** point `ERP_MODEL` + `ERP_SERVED_NAME` at the next quant dir, keep the
+5
View File
@@ -35,6 +35,11 @@ services:
- gemma4
- --default-chat-template-kwargs
- '{"enable_thinking": false}'
# tool_choice:"none" trap (measured 2026-09-08): without this flag vLLM still renders the
# tools into the prompt, the model emits a tool call anyway, and because tool parsing is
# off for tool_choice=none the reply comes back with content=null AND tool_calls=null —
# an empty turn. The flag drops the tools from the prompt so the model answers in prose.
- --exclude-tools-when-tool-choice-none
- --chat-template
- ${ERP_CHAT_TEMPLATE:-/tank/aimodels/erp-tune-v6-nvfp4a16/chat_template.jinja}
- --max-model-len