vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

config(litellm): name gen backend by real model (aeon → qwen3.6-35b-a3b-heretic)

Browse Source 
The gen seat's vLLM served-name was still qwen3.6-27b-aeon, a stale skin
left over from the AEON-27B → 35B-A3B-heretic swap — it named neither the
right family (aeon) nor size (27b vs 35B-A3B). Renamed the served-name to
qwen3.6-35b-a3b-heretic (+ -thinking) on ana-ml2 :8015 via the stack .env,
and repointed litellm's gen / gen-reasoning / summarizer-large model refs +
comments to match, so /v1/models, the gateway config, and spend-logs all
reveal the actual model in the request path.

Verified end-to-end: gen -> 'PIPELINE OK', gen-reasoning -> content + reasoning
surfaced, all three aliases healthy. char-rp / char-rp-reasoning untouched.
This commit is contained in:
Vuong Hoang
2026-07-08 18:40:08 -07:00
parent b889c55229
commit 99a4a1721f
1 changed files with 11 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/litellm/conf/config.yaml
+11 -11
View File
@@ -54,12 +54,12 @@ model_list:
model_info:
mode: chat
# alias: summarizer-large -> gen / qwen3.6-27b-aeon (operator 2026-07-05). For heavier
# summarization that wants the AEON 27B `gen` model instead of granite-8b. Thinking OFF
# alias: summarizer-large -> gen / qwen3.6-35b-a3b-heretic (operator 2026-07-05). For heavier
# summarization that wants the 35B-A3B heretic `gen` model instead of granite-8b. Thinking OFF
# (matches gen). Keep api_base (:8015) + enable_thinking in sync with the gen record below.
- model_name: summarizer-large
litellm_params:
model: hosted_vllm/qwen3.6-27b-aeon
model: hosted_vllm/qwen3.6-35b-a3b-heretic
api_base: http://10.250.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.5
@@ -104,12 +104,12 @@ model_list:
model_info:
mode: chat
# --- Qwen3.6-27B AEON (uncensored/abliterated, NVFP4 ModelOpt, VISION-INTACT image+video)
# — the general / `gen` model on ana-ml2 GPU 0. REPLACED qwopus3.5-122b 2026-07-05
# (operator: displace qwopus, this assumes gen/gen-reasoning). Dense 27B, qwen3_5
# GDN-hybrid (qwopus's little sibling), native MTP head. Served on :8015 via vLLM,
# served-name qwen3.6-27b-aeon, MTP OFF (spec-decode hurts concurrent aggregate — the
# MTP twin is char-rp below). Thinking split = chat_template_kwargs.enable_thinking +
# --- Qwen3.6-35B-A3B heretic (llmfan46, uncensored, NVFP4 ModelOpt Experts-Only, VISION-INTACT)
# — the general / `gen` model on ana-ml2 GPU 0, at /tank/aimodels/qwen36-35b-a3b-heretic-nvfp4.
# Displaced AEON-27B 2026-07-08 (which had displaced qwopus3.5-122b 2026-07-05). MoE 35B-A3B
# (256 experts / 8 active), qwen3_5_moe GDN-hybrid, native MTP preserved but served MTP-OFF
# (spec-decode hurts concurrent aggregate). Served on :8015 via vLLM,
# served-name qwen3.6-35b-a3b-heretic. Thinking split = chat_template_kwargs.enable_thinking +
# --reasoning-parser qwen3; tool-calling qwen3_coder. gen / gen-reasoning + summarizer-
# large route here; -reasoning enables thinking. Keep api_base (:8015) in sync.
# RETIRED with the displacement (→ 404, callers migrate to gen): qwen3.5-122-a10b
@@ -121,7 +121,7 @@ model_list:
# GENERAL thinking profile, not the 0.6 coding sub-profile). docs/pfi/model-sampler-defaults.md. ---
- model_name: gen
litellm_params:
model: hosted_vllm/qwen3.6-27b-aeon
model: hosted_vllm/qwen3.6-35b-a3b-heretic
api_base: http://10.250.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.5
@@ -137,7 +137,7 @@ model_list:
litellm_params:
# Distinct served-name so a thinking-off `gen` request can't mutate this deployment's
# enable_thinking (shared-config-mutation footgun). Same backend :8015, different model id.
model: hosted_vllm/qwen3.6-27b-aeon-thinking
model: hosted_vllm/qwen3.6-35b-a3b-heretic-thinking
api_base: http://10.250.50.54:8015/v1
api_key: os.environ/VLLM_API_KEY
presence_penalty: 1.5