feat(char-rp-gguf): replace broken Angel NVFP4 with dual GGUF RP seat on ana-ml2 GPU0
char-rp -> TheDrummer Magidonia-24B-v4.3 Q6_K (Magistral prose, ~65 tok/s,
zero refusal, tight POV) via llama.cpp (:8016).
char-rp-reasoning -> ArliAI QwQ-32B-RpR-v4 Q5_K_M (abliterated managed reasoning,
~52 tok/s, reasoning surfaces in reasoning_content) via llama.cpp (:8018).
- New canonical stack stacks/char-rp-gguf/ (llama-server x2, GPU0-pinned, ~86/97G
co-resident with gen). GGUF sidesteps the vLLM-NVFP4 + Mistral-tokenizer traps that
killed the Angel serve. Never Ollama.
- Best-of-breed per seat: no single dense 24-32B is both an elite non-thinking prose
seat AND a clean managed-reasoning seat on llama.cpp (Magidonia [THINK] boundary is
loose; Cydonia-R1 <think> runs away; QwQ is template-managed). Pantheon-Reasoning-27B
stays rejected (re-censors in <think>; RpR-v4 abliterated reasoning is the fix).
- Gateway rewired: char-rp->:8016, char-rp-reasoning->:8018, Mistral/QwQ samplers,
dropped the Qwen enable_thinking kwarg. One-model Magidonia fallback documented.
- Retired the ms32-24b-angel stack.
This commit is contained in:
@@ -148,49 +148,42 @@ model_list:
|
||||
enable_thinking: true
|
||||
model_info:
|
||||
mode: chat
|
||||
# char-rp -> the native-MTP single-seat RP twin (:8016, served qwen3.6-27b-aeon-rp).
|
||||
# SAME weights as gen, MTP ON (qwen3_5_mtp n=3) for single-stream RP latency. RP-shaped
|
||||
# sampler defaults (Worldtree character-rp role / callers override); thinking OFF.
|
||||
# ⚠️ MTP silently drops min_p/logit_bias — don't rely on those through char-rp.
|
||||
# char-rp -> GGUF dark-romantasy PROSE seat (:8016, llama.cpp, char-rp-gguf stack on
|
||||
# ana-ml2 GPU 0). TheDrummer Magidonia-24B-v4.3 Q6_K — Magistral (Mistral) RP tune.
|
||||
# NON-thinking: elite literary prose, zero refusal on dark/explicit scenes, ~65 tok/s,
|
||||
# tight POV/instruction adherence (live-tested 2026-07-08). Replaced the broken Angel
|
||||
# NVFP4 serve AND the earlier AEON-rp MTP twin. Magistral is stable WITHOUT a repetition
|
||||
# penalty (dropped the old 1.05); min_p 0.03 is the anti-slop knob. No enable_thinking
|
||||
# kwarg — meaningless to the Mistral template. Alt prose model (swap via the stack .env):
|
||||
# MS3.2-PaintedFantasy-v4.1-24B. Callers may override the sampler.
|
||||
- model_name: char-rp
|
||||
litellm_params:
|
||||
model: hosted_vllm/qwen3.6-27b-aeon-rp
|
||||
model: hosted_vllm/magidonia-24b-v4.3
|
||||
api_base: http://10.250.50.54:8016/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
# Qwen3.x non-thinking RP recs (operator 2026-07-05, adapted from Qwen + community
|
||||
# RP testing). presence_penalty light (0.1) to reduce topic drift; repetition_penalty
|
||||
# 1.05. min_p skipped (Qwen rec + MTP drops it anyway). DRY off (Qwen3.x artifacts).
|
||||
temperature: 0.7
|
||||
top_p: 0.8
|
||||
presence_penalty: 0.1
|
||||
temperature: 1.0
|
||||
top_p: 0.95
|
||||
extra_body:
|
||||
top_k: 20
|
||||
repetition_penalty: 1.05
|
||||
chat_template_kwargs:
|
||||
enable_thinking: false
|
||||
min_p: 0.03
|
||||
model_info:
|
||||
mode: chat
|
||||
# char-rp-reasoning -> same RP seat (:8016, MTP), thinking ON. Reasoning-profile
|
||||
# sampler (lower temp than char-rp for coherent thought); tunable. ⚠️ The thinking
|
||||
# TRACE does not yet surface in reasoning_content (chat_template injects <think> in the
|
||||
# prompt → parser drops the span); pending a template fix, NOT a parser swap.
|
||||
# char-rp-reasoning -> GGUF managed-REASONING seat (:8018, llama.cpp, char-rp-gguf stack).
|
||||
# ArliAI QwQ-32B-ArliAI-RpR-v4 Q5_K_M — QwQ reasoning RP tune. Reasoning is ON server-side
|
||||
# (--reasoning on): the CoT SURFACES in reasoning_content and content stays clean prose
|
||||
# (fixes the old trace-not-surfacing gap), CoT budget-capped so it can't starve the prose.
|
||||
# Its reasoning data is QwQ-ABLITERATED → no re-censor inside <think> (the failure mode
|
||||
# that disqualified Pantheon-Reasoning-27B). ~52 tok/s @ Q5_K_M. RpR card: temp 1.0,
|
||||
# top_k 40, min_p 0.02, and NO repetition / DRY / XTC penalties. NOT the same model as
|
||||
# char-rp (best-of-breed per seat) — see stacks/char-rp-gguf/README.md.
|
||||
- model_name: char-rp-reasoning
|
||||
litellm_params:
|
||||
# Distinct served-name so a thinking-off char-rp request can't clobber this to
|
||||
# enable_thinking:false (the bug that broke it). Same backend :8016, different model id.
|
||||
model: hosted_vllm/qwen3.6-27b-aeon-rp-thinking
|
||||
api_base: http://10.250.50.54:8016/v1
|
||||
model: hosted_vllm/qwq-32b-rpr-v4
|
||||
api_base: http://10.250.50.54:8018/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
# Same RP profile as char-rp but thinking-mode top_p 0.95 (Qwen thinking rec) + focused
|
||||
# temp 0.6. (Reasoning-trace surfacing still pending the parser/template fix.)
|
||||
temperature: 0.6
|
||||
top_p: 0.95
|
||||
presence_penalty: 0.1
|
||||
temperature: 1.0
|
||||
extra_body:
|
||||
top_k: 20
|
||||
repetition_penalty: 1.05
|
||||
chat_template_kwargs:
|
||||
enable_thinking: true
|
||||
top_k: 40
|
||||
min_p: 0.02
|
||||
model_info:
|
||||
mode: chat
|
||||
|
||||
|
||||
Reference in New Issue
Block a user