feat(qwen36-vl): split thinking — non-thinking default + qwen3.6-35b-a3b-thinking variant

The qwen3.6-35b-a3b VL checkpoint is a single hybrid model with a per-
request enable_thinking switch (Qwen3-style), defaulting thinking ON.
Make the default non-thinking and add an opt-in reasoning variant,
mirroring the existing glm-5.1 / glm-5.1-reasoning gateway split.

- qwen36-vl compose: add --reasoning-parser qwen3 (model-matched) so the
  single :8007 endpoint splits <think> into reasoning_content when on and
  routes all output to content when off — serving both modes cleanly.
- litellm gateway: base qwen3.6-35b-a3b pins chat_template_kwargs
  enable_thinking=false (non-thinking default); new qwen3.6-35b-a3b-thinking
  pins enable_thinking=true (opt-in reasoning). Same upstream checkpoint,
  no extra VRAM/container.

Deployed + verified on ana-ml2 (vLLM recreated, healthy) and ana-docker
(litellm reloaded): default returns a direct answer with no reasoning_content;
-thinking returns cleanly-separated reasoning_content, no raw tag leak.
This commit is contained in:
vh
2026-06-15 13:55:11 -07:00
parent 0943d145fb
commit 6de0844323
2 changed files with 45 additions and 1 deletions
+26 -1
View File
@@ -33,12 +33,37 @@ model_list:
# --- Qwen3.6-35B-A3B vision-language MoE (official FP8) — vision + chat. vLLM
# on ana-ml2 GPU 1, :8007. Explicit entry shadows the "*" wildcard llama-swap
# route. REPLACED qwen3.5-9b-fp8 2026-06-14 (the 9B is retired; this is a
# 35B-A3B MoE — served under its TRUE name, never aliased under the old one). ---
# 35B-A3B MoE — served under its TRUE name, never aliased under the old one).
#
# THINKING SPLIT (2026-06-15, operator call — mirrors the glm-5.1 pattern
# above). One hybrid checkpoint; the per-request `enable_thinking` switch
# picks the mode. LiteLLM forwards extra_body verbatim to vLLM, where
# chat_template_kwargs lands in the chat template. vLLM runs
# --reasoning-parser qwen3 so reasoning surfaces as reasoning_content. ---
# qwen3.6-35b-a3b: thinking DISABLED by default. The checkpoint defaults
# thinking ON; enable_thinking=false forces the empty <think></think> block.
# Reasoning is opt-in via qwen3.6-35b-a3b-thinking below.
- model_name: qwen3.6-35b-a3b
litellm_params:
model: hosted_vllm/qwen3.6-35b-a3b
api_base: http://10.250.50.54:8007/v1
api_key: os.environ/VLLM_API_KEY
extra_body:
chat_template_kwargs:
enable_thinking: false
model_info:
mode: chat
# qwen3.6-35b-a3b-thinking: identical upstream checkpoint, thinking ENABLED
# (opt-in reasoning). The qwen3 reasoning-parser splits <think>…</think> into
# reasoning_content; content holds just the answer.
- model_name: qwen3.6-35b-a3b-thinking
litellm_params:
model: hosted_vllm/qwen3.6-35b-a3b
api_base: http://10.250.50.54:8007/v1
api_key: os.environ/VLLM_API_KEY
extra_body:
chat_template_kwargs:
enable_thinking: true
model_info:
mode: chat