feat(qwen3.5-122b): replace mistral-small-4 as gen (abliterated NVFP4, text-only)

bjk110/Qwen3.5-122B-A10B-abliterated-NVFP4 on ana-ml2 GPU 0 (heretic downed):
- stacks/qwen3.5-122b/ — vLLM serve via the repo's text-only patch (Qwen3.5 MoE is a
  multimodal arch but this checkpoint is text-only weights), --reasoning-parser qwen3,
  GPU 0 pin, :8013; entrypoint+patch mounted from the model dir.
- serve-qwen3.5-122b.yaml — displace heretic + serve + verify.
- litellm: REMOVED dead mistral-small-4 / -reasoning; added qwen3.5-122-a10b[-reasoning]
  + aliases qwen-large[-reasoning] + repointed gen[-reasoning] -> qwen (thinking split via
  chat_template_kwargs.enable_thinking + --reasoning-parser qwen3).

Verified live: qwen healthy on :8013; gen / qwen-large / qwen3.5-122-a10b route, and
gen-reasoning returns reasoning_content; mistral-small-4 removed.
NOTE: Worldtree character backend (was bound to mistral-small-4) is dark until repointed
(operator-acknowledged).
This commit is contained in:
vh
2026-06-19 00:49:04 -07:00
parent 67102b5b94
commit 89c83c4271
4 changed files with 229 additions and 31 deletions
+53 -31
View File
@@ -78,51 +78,73 @@ model_list:
model_info:
mode: chat
# --- Mistral Small 4 (official NVFP4) — creative-writing / general text. 119B
# MoE (6.5B active), vLLM on ana-ml2 GPU 0 (dedicated 96 GB Blackwell), :8010.
# Explicit entry shadows the "*" wildcard. TEXT-ONLY for now — vLLM 0.23.0's
# Mistral multimodal processor crashes at startup (loaded with image/video
# limit 0); vision returns when vLLM patches it. Deployed 2026-06-15. ---
- model_name: mistral-small-4
# --- Qwen3.5-122B-A10B (abliterated NVFP4, text-only) — the general / `gen` model.
# REPLACED mistral-small-4 on ana-ml2 GPU 0 (operator 2026-06-19; heretic downed,
# mistral entries REMOVED). bjk110 NVFP4 served on :8013 via vLLM + the text-only
# patch, served-name qwen3.5-122-a10b. Thinking split = chat_template_kwargs.
# enable_thinking + --reasoning-parser qwen3 (mirrors qwen3.6-35b-a3b). One upstream
# fanned out under qwen3.5-122-a10b[-reasoning] + aliases qwen-large[-reasoning] +
# gen[-reasoning]; the -reasoning variants enable thinking. Keep api_base in sync.
# ⚠️ Worldtree CHARACTER backend (was bound to mistral-small-4) is dark until
# repointed — operator-acknowledged. ---
- model_name: qwen3.5-122-a10b
litellm_params:
model: hosted_vllm/mistral-small-4
api_base: http://10.250.50.54:8010/v1
api_key: os.environ/VLLM_API_KEY
model_info:
mode: chat
# mistral-small-4-reasoning: same upstream checkpoint, reasoning ON (2026-06-15,
# operator wanted "medium" — but Mistral's reasoning_effort is BINARY, only
# 'none' or 'high' (a medium/low request 400s). 'high' is the sole reasoning-ON
# level, so it carries the -reasoning intent. LiteLLM forwards extra_body to
# vLLM; the mistral reasoning-parser splits [THINK]…[/THINK] into reasoning_content.
- model_name: mistral-small-4-reasoning
litellm_params:
model: hosted_vllm/mistral-small-4
api_base: http://10.250.50.54:8010/v1
model: hosted_vllm/qwen3.5-122-a10b
api_base: http://10.250.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
extra_body:
reasoning_effort: high
chat_template_kwargs:
enable_thinking: false
model_info:
mode: chat
- model_name: qwen3.5-122-a10b-reasoning
litellm_params:
model: hosted_vllm/qwen3.5-122-a10b
api_base: http://10.250.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
extra_body:
chat_template_kwargs:
enable_thinking: true
model_info:
mode: chat
- model_name: qwen-large
litellm_params:
model: hosted_vllm/qwen3.5-122-a10b
api_base: http://10.250.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
extra_body:
chat_template_kwargs:
enable_thinking: false
model_info:
mode: chat
- model_name: qwen-large-reasoning
litellm_params:
model: hosted_vllm/qwen3.5-122-a10b
api_base: http://10.250.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
extra_body:
chat_template_kwargs:
enable_thinking: true
model_info:
mode: chat
# aliases: gen -> mistral-small-4, gen-reasoning -> mistral-small-4-reasoning
# (operator 2026-06-19). Duplicate-entry aliases — keep api_base in sync with the
# two mistral-small-4 entries above.
- model_name: gen
litellm_params:
model: hosted_vllm/mistral-small-4
api_base: http://10.250.50.54:8010/v1
model: hosted_vllm/qwen3.5-122-a10b
api_base: http://10.250.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
extra_body:
chat_template_kwargs:
enable_thinking: false
model_info:
mode: chat
- model_name: gen-reasoning
litellm_params:
model: hosted_vllm/mistral-small-4
api_base: http://10.250.50.54:8010/v1
model: hosted_vllm/qwen3.5-122-a10b
api_base: http://10.250.50.54:8013/v1
api_key: os.environ/VLLM_API_KEY
extra_body:
reasoning_effort: high
chat_template_kwargs:
enable_thinking: true
model_info:
mode: chat