vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

litellm: canonical general-use sampling defaults across gateway models

Browse Source 
Per operator directive (overrideable defaults => sane optimal general-use
sampling on every served model, for quality not just repetition):
- qwen3.5-122-a10b: thinking-split canonicals (non-thinking temp 0.7/top_p 0.8;
  thinking temp 0.6/top_p 0.95; top_k 20 both), keep presence_penalty 1.0.
  Fixes the non-thinking routes previously running thinking-mode sampling.
- qwen-image-bench / image-judge: deterministic judge profile temp 0 / top_k 1 /
  top_p 1.0 / repetition_penalty 1.05.
- selene-1-mini-8b / chat-judge: temp 0.6 / top_p 0.9 (gen_config).
vLLM-only params (top_k, repetition_penalty) in extra_body to survive
drop_params. Values dvalin-researched + KB-corroborated (docs/pfi/
recommended-model-settings.md). granite + GLM pending dvalin deeper search.
This commit is contained in:
Vuong Hoang
2026-06-27 08:26:46 -07:00
parent 52d5f66216
commit 4f094fa653
1 changed files with 35 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/litellm/conf/config.yaml
+35
View File
@@ -60,7 +60,10 @@ model_list:
api_base: http://10.250.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.0
temperature: 0.7
top_p: 0.8
extra_body:
top_k: 20
chat_template_kwargs:
enable_thinking: false
model_info:
@@ -78,6 +81,11 @@ model_list:
model: hosted_vllm/qwen-image-bench
api_base: http://10.250.50.54:8014/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
top_p: 1.0
extra_body:
top_k: 1
repetition_penalty: 1.05
model_info:
mode: chat
- model_name: image-judge
@@ -85,6 +93,11 @@ model_list:
model: hosted_vllm/qwen-image-bench
api_base: http://10.250.50.54:8014/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0
top_p: 1.0
extra_body:
top_k: 1
repetition_penalty: 1.05
model_info:
mode: chat
@@ -110,7 +123,10 @@ model_list:
api_base: http://10.250.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.0
temperature: 0.7
top_p: 0.8
extra_body:
top_k: 20
chat_template_kwargs:
enable_thinking: false
model_info:
@@ -121,7 +137,10 @@ model_list:
api_base: http://10.250.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.0
temperature: 0.6
top_p: 0.95
extra_body:
top_k: 20
chat_template_kwargs:
enable_thinking: true
model_info:
@@ -132,7 +151,10 @@ model_list:
api_base: http://10.250.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.0
temperature: 0.7
top_p: 0.8
extra_body:
top_k: 20
chat_template_kwargs:
enable_thinking: false
model_info:
@@ -143,7 +165,10 @@ model_list:
api_base: http://10.250.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.0
temperature: 0.6
top_p: 0.95
extra_body:
top_k: 20
chat_template_kwargs:
enable_thinking: true
model_info:
@@ -154,7 +179,10 @@ model_list:
api_base: http://10.250.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.0
temperature: 0.7
top_p: 0.8
extra_body:
top_k: 20
chat_template_kwargs:
enable_thinking: false
model_info:
@@ -165,7 +193,10 @@ model_list:
api_base: http://10.250.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.0
temperature: 0.6
top_p: 0.95
extra_body:
top_k: 20
chat_template_kwargs:
enable_thinking: true
model_info:
@@ -180,6 +211,8 @@ model_list:
model: hosted_vllm/selene-1-mini-8b
api_base: http://10.250.50.54:8011/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0.6
top_p: 0.9
model_info:
mode: chat
@@ -212,6 +245,8 @@ model_list:
model: hosted_vllm/selene-1-mini-8b
api_base: http://10.250.50.54:8011/v1
api_key: os.environ/VLLM_API_KEY
temperature: 0.6
top_p: 0.9
model_info:
mode: chat
# reranker → generic capability name for rerank (currently qwen3-reranker).