vh
926fc2fb7a
The gen-reasoning seat accepts only xhigh/medium/low and 400s on anything else — including `high`, which is the default of several clients, so the seat presented as broken rather than as one enum value out of step. The DeepSeek Harness failed every request on its default setting, and the only working client value was `low`: the seat's WEAKEST reasoning tier, while its own default is xhigh. conf/reasoning_effort_map.py is a pre-call hook in the same shape as the existing strip_empty_tools hook. It is scoped to one model group, measured rather than assumed: gen-reasoning rejects `high`; gen, sec, char-rp-reasoning and summarizer all accept it and are left alone. Paid passthroughs were not probed, because probing them spends vendor credits, and are not mapped. Verified after deploy: high and max now succeed on gen-reasoning, low and xhigh still work, a request with no effort param still works, `gen` with `high` still passes through unmapped, and the harness completes a real file-edit task at full reasoning. Needed a compose change as well as a conf push — callbacks are bind-mounted per file, so the volume only attaches on container create. Recreated the litellm service by name so the DB was not bounced with it.
66 lines
2.8 KiB
Python
66 lines
2.8 KiB
Python
# reasoning_effort_map.py
|
|
#
|
|
# LiteLLM proxy pre-call hook: translate `reasoning_effort` values that a
|
|
# backend does not accept into the ones it does, per model group.
|
|
#
|
|
# Why: the OpenAI-shaped `reasoning_effort` vocabulary is not universal. Our
|
|
# `gen-reasoning` seat (qwen3.8-27b-uncensored-thinking on ana-ml2 :8015)
|
|
# accepts ONLY `xhigh` (its default), `medium`, and `low`, and 400s on anything
|
|
# else:
|
|
# "Unexpected reasoning effort high. Supported types are xhigh (default),
|
|
# medium, and low."
|
|
# Clients that emit the common OpenAI trio (low/medium/high) or their own
|
|
# vocabulary therefore fail on their DEFAULT setting, which reads as "the seat
|
|
# is broken" rather than "one enum value differs".
|
|
#
|
|
# Concrete case that motivated this (2026-09-02): the DeepSeek Harness
|
|
# (@deepseek-ai/dsh) emits off/low/high/max and defaults to `high`. Every
|
|
# request failed. Without this map the only working client setting was `low` —
|
|
# the seat's WEAKEST reasoning tier — while its own default is `xhigh`.
|
|
#
|
|
# ⚠ SCOPED DELIBERATELY. Measured 2026-09-02 against every local seat:
|
|
# gen-reasoning high -> REJECTED
|
|
# gen, sec, char-rp-reasoning, summarizer high -> accepted
|
|
# so only `gen-reasoning` is remapped. Paid passthroughs (gen-frontier*, glm*,
|
|
# kimi*) were NOT probed — they spend vendor credits — and are not mapped.
|
|
# Add a model here only after measuring that it actually rejects the value.
|
|
#
|
|
# Registered via litellm_settings.callbacks. LiteLLM loads callbacks RELATIVE
|
|
# TO THE CONFIG FILE'S DIRECTORY, so this file must sit beside config.yaml —
|
|
# mounted into the container at /app/reasoning_effort_map.py.
|
|
|
|
from litellm.integrations.custom_logger import CustomLogger
|
|
from litellm._logging import verbose_proxy_logger
|
|
|
|
# model group -> {value the client sent: value the backend accepts}
|
|
# Values already accepted by the backend are simply absent and pass through.
|
|
EFFORT_MAP = {
|
|
"gen-reasoning": {
|
|
# Map the two "as hard as you can think" spellings onto the seat's own
|
|
# maximum, which is also its default. `medium` and `low` are native.
|
|
"high": "xhigh",
|
|
"max": "xhigh",
|
|
},
|
|
}
|
|
|
|
|
|
class ReasoningEffortMap(CustomLogger):
|
|
async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
|
|
if not isinstance(data, dict):
|
|
return data
|
|
mapping = EFFORT_MAP.get(data.get("model"))
|
|
if not mapping:
|
|
return data
|
|
sent = data.get("reasoning_effort")
|
|
mapped = mapping.get(sent)
|
|
if mapped is not None:
|
|
data["reasoning_effort"] = mapped
|
|
verbose_proxy_logger.debug(
|
|
"reasoning_effort_map: %s %r -> %r (call_type=%s)",
|
|
data.get("model"), sent, mapped, call_type,
|
|
)
|
|
return data
|
|
|
|
|
|
reasoning_effort_map_instance = ReasoningEffortMap()
|