feat(char-rp-gguf): replace broken Angel NVFP4 with dual GGUF RP seat on ana-ml2 GPU0

char-rp        -> TheDrummer Magidonia-24B-v4.3 Q6_K (Magistral prose, ~65 tok/s,
                  zero refusal, tight POV) via llama.cpp (:8016).
char-rp-reasoning -> ArliAI QwQ-32B-RpR-v4 Q5_K_M (abliterated managed reasoning,
                  ~52 tok/s, reasoning surfaces in reasoning_content) via llama.cpp (:8018).

- New canonical stack stacks/char-rp-gguf/ (llama-server x2, GPU0-pinned, ~86/97G
  co-resident with gen). GGUF sidesteps the vLLM-NVFP4 + Mistral-tokenizer traps that
  killed the Angel serve. Never Ollama.
- Best-of-breed per seat: no single dense 24-32B is both an elite non-thinking prose
  seat AND a clean managed-reasoning seat on llama.cpp (Magidonia [THINK] boundary is
  loose; Cydonia-R1 <think> runs away; QwQ is template-managed). Pantheon-Reasoning-27B
  stays rejected (re-censors in <think>; RpR-v4 abliterated reasoning is the fix).
- Gateway rewired: char-rp->:8016, char-rp-reasoning->:8018, Mistral/QwQ samplers,
  dropped the Qwen enable_thinking kwarg. One-model Magidonia fallback documented.
- Retired the ms32-24b-angel stack.
This commit is contained in:
vh
2026-07-08 02:43:00 -07:00
parent 75851c2837
commit b268f93035
5 changed files with 500 additions and 55 deletions
+25 -32
View File
@@ -148,49 +148,42 @@ model_list:
enable_thinking: true
model_info:
mode: chat
# char-rp -> the native-MTP single-seat RP twin (:8016, served qwen3.6-27b-aeon-rp).
# SAME weights as gen, MTP ON (qwen3_5_mtp n=3) for single-stream RP latency. RP-shaped
# sampler defaults (Worldtree character-rp role / callers override); thinking OFF.
# ⚠️ MTP silently drops min_p/logit_bias — don't rely on those through char-rp.
# char-rp -> GGUF dark-romantasy PROSE seat (:8016, llama.cpp, char-rp-gguf stack on
# ana-ml2 GPU 0). TheDrummer Magidonia-24B-v4.3 Q6_K — Magistral (Mistral) RP tune.
# NON-thinking: elite literary prose, zero refusal on dark/explicit scenes, ~65 tok/s,
# tight POV/instruction adherence (live-tested 2026-07-08). Replaced the broken Angel
# NVFP4 serve AND the earlier AEON-rp MTP twin. Magistral is stable WITHOUT a repetition
# penalty (dropped the old 1.05); min_p 0.03 is the anti-slop knob. No enable_thinking
# kwarg — meaningless to the Mistral template. Alt prose model (swap via the stack .env):
# MS3.2-PaintedFantasy-v4.1-24B. Callers may override the sampler.
- model_name: char-rp
litellm_params:
model: hosted_vllm/qwen3.6-27b-aeon-rp
model: hosted_vllm/magidonia-24b-v4.3
api_base: http://10.250.50.54:8016/v1
api_key: os.environ/VLLM_API_KEY
# Qwen3.x non-thinking RP recs (operator 2026-07-05, adapted from Qwen + community
# RP testing). presence_penalty light (0.1) to reduce topic drift; repetition_penalty
# 1.05. min_p skipped (Qwen rec + MTP drops it anyway). DRY off (Qwen3.x artifacts).
temperature: 0.7
top_p: 0.8
presence_penalty: 0.1
temperature: 1.0
top_p: 0.95
extra_body:
top_k: 20
repetition_penalty: 1.05
chat_template_kwargs:
enable_thinking: false
min_p: 0.03
model_info:
mode: chat
# char-rp-reasoning -> same RP seat (:8016, MTP), thinking ON. Reasoning-profile
# sampler (lower temp than char-rp for coherent thought); tunable. ⚠️ The thinking
# TRACE does not yet surface in reasoning_content (chat_template injects <think> in the
# prompt → parser drops the span); pending a template fix, NOT a parser swap.
# char-rp-reasoning -> GGUF managed-REASONING seat (:8018, llama.cpp, char-rp-gguf stack).
# ArliAI QwQ-32B-ArliAI-RpR-v4 Q5_K_M — QwQ reasoning RP tune. Reasoning is ON server-side
# (--reasoning on): the CoT SURFACES in reasoning_content and content stays clean prose
# (fixes the old trace-not-surfacing gap), CoT budget-capped so it can't starve the prose.
# Its reasoning data is QwQ-ABLITERATED → no re-censor inside <think> (the failure mode
# that disqualified Pantheon-Reasoning-27B). ~52 tok/s @ Q5_K_M. RpR card: temp 1.0,
# top_k 40, min_p 0.02, and NO repetition / DRY / XTC penalties. NOT the same model as
# char-rp (best-of-breed per seat) — see stacks/char-rp-gguf/README.md.
- model_name: char-rp-reasoning
litellm_params:
# Distinct served-name so a thinking-off char-rp request can't clobber this to
# enable_thinking:false (the bug that broke it). Same backend :8016, different model id.
model: hosted_vllm/qwen3.6-27b-aeon-rp-thinking
api_base: http://10.250.50.54:8016/v1
model: hosted_vllm/qwq-32b-rpr-v4
api_base: http://10.250.50.54:8018/v1
api_key: os.environ/VLLM_API_KEY
# Same RP profile as char-rp but thinking-mode top_p 0.95 (Qwen thinking rec) + focused
# temp 0.6. (Reasoning-trace surfacing still pending the parser/template fix.)
temperature: 0.6
top_p: 0.95
presence_penalty: 0.1
temperature: 1.0
extra_body:
top_k: 20
repetition_penalty: 1.05
chat_template_kwargs:
enable_thinking: true
top_k: 40
min_p: 0.02
model_info:
mode: chat