feat(litellm): Worldtree capability aliases (chat-judge, reranker, scalar-judge)
Stand up the gateway-side capability aliases for the role→capability model indirection (worldtree-dev's transparent-swap direction; operator: no wt- prefix, reuse the existing summarizer/classifier/gen alias convention). - chat-judge -> selene-1-mini-8b (mode chat) — WT selene-judgment role. - reranker -> qwen3-reranker (mode rerank) — generic name for the cap. - scalar-judge -> Skywork-Reward-V2 via a pass_through_endpoint to ana-ml2:8003 (LiteLLM has no reward/pooling MODE, so it's a passthrough, gateway-key-gated; consumers hit /scalar-judge/<route> e.g. /score|/pooling|/classify). Deliberately NO generic `embedding` alias: embedding vectors are model-specific (not swap-transparent), so that capability stays `qwen3-embedding` — the model- specific name is the guardrail against treating it as freely swappable. Verified all three live (chat-judge 200, reranker present, scalar-judge passthrough 200 returning a Skywork reward). Deployed + gateway health-gated.
This commit is contained in:
@@ -189,6 +189,28 @@ model_list:
|
||||
model_info:
|
||||
mode: rerank
|
||||
|
||||
# --- Worldtree capability aliases (role→capability gateway swaps, ADR-0012).
|
||||
# Stable role-named aliases so consumers bind the CAPABILITY, not a concrete
|
||||
# model; swap the backing model here and callers are unaffected. NO generic
|
||||
# `embedding` alias ON PURPOSE — embedding vectors are model-specific (not
|
||||
# swap-transparent), so that capability stays `qwen3-embedding` above. ---
|
||||
# chat-judge → generative LLM-as-judge (Selene); WT selene-judgment role.
|
||||
- model_name: chat-judge
|
||||
litellm_params:
|
||||
model: hosted_vllm/selene-1-mini-8b
|
||||
api_base: http://10.250.50.54:8011/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
model_info:
|
||||
mode: chat
|
||||
# reranker → generic capability name for rerank (currently qwen3-reranker).
|
||||
- model_name: reranker
|
||||
litellm_params:
|
||||
model: hosted_vllm/Qwen/Qwen3-Reranker-0.6B
|
||||
api_base: http://10.250.50.54:8002/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
model_info:
|
||||
mode: rerank
|
||||
|
||||
# --- z.ai GLM (cloud API) — fronted for unified logging across local
|
||||
# + cloud inference. Explicit entries, so they win over the "*"
|
||||
# wildcard below (no collision with llama-swap's glm4.7-flash etc.
|
||||
@@ -276,6 +298,17 @@ general_settings:
|
||||
# SpendLogs so they render in the Logs UI. Without this you get metadata
|
||||
# (tokens, latency, model) but not the prompt/completion text.
|
||||
store_prompts_in_spend_logs: true
|
||||
# scalar-judge → Skywork-Reward-V2 (scalar reward model; vLLM pooling on
|
||||
# ana-ml2:8003). LiteLLM has no reward/pooling MODE, so this is a passthrough,
|
||||
# not a model_list alias. Gateway-key-gated. Consumers POST the reward body to
|
||||
# /scalar-judge/<route> (e.g. /pooling or /classify), forwarded to :8003.
|
||||
# SWAP-SENSITIVE: a different reward model shifts the score scale, so consumers
|
||||
# must recalibrate thresholds after a backing swap.
|
||||
pass_through_endpoints:
|
||||
- path: "/scalar-judge"
|
||||
target: "http://10.250.50.54:8003"
|
||||
forward_headers: true
|
||||
include_subpath: true
|
||||
|
||||
litellm_settings:
|
||||
# vLLM rejects some OpenAI params other backends accept; drop silently
|
||||
|
||||
Reference in New Issue
Block a user