feat(litellm): Worldtree capability aliases (chat-judge, reranker, scalar-judge)
Stand up the gateway-side capability aliases for the role→capability model indirection (worldtree-dev's transparent-swap direction; operator: no wt- prefix, reuse the existing summarizer/classifier/gen alias convention). - chat-judge -> selene-1-mini-8b (mode chat) — WT selene-judgment role. - reranker -> qwen3-reranker (mode rerank) — generic name for the cap. - scalar-judge -> Skywork-Reward-V2 via a pass_through_endpoint to ana-ml2:8003 (LiteLLM has no reward/pooling MODE, so it's a passthrough, gateway-key-gated; consumers hit /scalar-judge/<route> e.g. /score|/pooling|/classify). Deliberately NO generic `embedding` alias: embedding vectors are model-specific (not swap-transparent), so that capability stays `qwen3-embedding` — the model- specific name is the guardrail against treating it as freely swappable. Verified all three live (chat-judge 200, reranker present, scalar-judge passthrough 200 returning a Skywork reward). Deployed + gateway health-gated.
This commit is contained in:
@@ -189,6 +189,28 @@ model_list:
|
|||||||
model_info:
|
model_info:
|
||||||
mode: rerank
|
mode: rerank
|
||||||
|
|
||||||
|
# --- Worldtree capability aliases (role→capability gateway swaps, ADR-0012).
|
||||||
|
# Stable role-named aliases so consumers bind the CAPABILITY, not a concrete
|
||||||
|
# model; swap the backing model here and callers are unaffected. NO generic
|
||||||
|
# `embedding` alias ON PURPOSE — embedding vectors are model-specific (not
|
||||||
|
# swap-transparent), so that capability stays `qwen3-embedding` above. ---
|
||||||
|
# chat-judge → generative LLM-as-judge (Selene); WT selene-judgment role.
|
||||||
|
- model_name: chat-judge
|
||||||
|
litellm_params:
|
||||||
|
model: hosted_vllm/selene-1-mini-8b
|
||||||
|
api_base: http://10.250.50.54:8011/v1
|
||||||
|
api_key: os.environ/VLLM_API_KEY
|
||||||
|
model_info:
|
||||||
|
mode: chat
|
||||||
|
# reranker → generic capability name for rerank (currently qwen3-reranker).
|
||||||
|
- model_name: reranker
|
||||||
|
litellm_params:
|
||||||
|
model: hosted_vllm/Qwen/Qwen3-Reranker-0.6B
|
||||||
|
api_base: http://10.250.50.54:8002/v1
|
||||||
|
api_key: os.environ/VLLM_API_KEY
|
||||||
|
model_info:
|
||||||
|
mode: rerank
|
||||||
|
|
||||||
# --- z.ai GLM (cloud API) — fronted for unified logging across local
|
# --- z.ai GLM (cloud API) — fronted for unified logging across local
|
||||||
# + cloud inference. Explicit entries, so they win over the "*"
|
# + cloud inference. Explicit entries, so they win over the "*"
|
||||||
# wildcard below (no collision with llama-swap's glm4.7-flash etc.
|
# wildcard below (no collision with llama-swap's glm4.7-flash etc.
|
||||||
@@ -276,6 +298,17 @@ general_settings:
|
|||||||
# SpendLogs so they render in the Logs UI. Without this you get metadata
|
# SpendLogs so they render in the Logs UI. Without this you get metadata
|
||||||
# (tokens, latency, model) but not the prompt/completion text.
|
# (tokens, latency, model) but not the prompt/completion text.
|
||||||
store_prompts_in_spend_logs: true
|
store_prompts_in_spend_logs: true
|
||||||
|
# scalar-judge → Skywork-Reward-V2 (scalar reward model; vLLM pooling on
|
||||||
|
# ana-ml2:8003). LiteLLM has no reward/pooling MODE, so this is a passthrough,
|
||||||
|
# not a model_list alias. Gateway-key-gated. Consumers POST the reward body to
|
||||||
|
# /scalar-judge/<route> (e.g. /pooling or /classify), forwarded to :8003.
|
||||||
|
# SWAP-SENSITIVE: a different reward model shifts the score scale, so consumers
|
||||||
|
# must recalibrate thresholds after a backing swap.
|
||||||
|
pass_through_endpoints:
|
||||||
|
- path: "/scalar-judge"
|
||||||
|
target: "http://10.250.50.54:8003"
|
||||||
|
forward_headers: true
|
||||||
|
include_subpath: true
|
||||||
|
|
||||||
litellm_settings:
|
litellm_settings:
|
||||||
# vLLM rejects some OpenAI params other backends accept; drop silently
|
# vLLM rejects some OpenAI params other backends accept; drop silently
|
||||||
|
|||||||
Reference in New Issue
Block a user