From 53096bffdc4002ac0735d3fbcd69ad6537fea72e Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 16 Aug 2026 09:57:48 -0700 Subject: [PATCH] docs(litellm): correct stale char-rp description -- it is MeroMero/Gemma-4, not the retired GGUF Magidonia seat The comment still described TheDrummer Magidonia-24B-v4.3 Q6_K on llama.cpp, which was replaced by the vLLM MeroMero-v2 NVFP4A16 seat on 2026-08-12. Routing was already correct (:8016 is MeroMero); only the prose was wrong, so anyone reading the config got the wrong model family entirely. Records why the seat exists: char-rp-reasoning is a Qwen3.x derivative and emits ~5-6k chars of CoT per turn regardless of which Qwen RP tune is loaded. Measured 2026-08-16 on identical prompts -- Dark-Scarlett 6036 ch vs Fable-Fusion 5323 ch -- so that is the base family, not the finetune, and no swap within it fixes it. Gemma-4 is the non-thinking seat. Also pins the mandatory --default-chat-template-kwargs '{"enable_thinking": false}' rationale from b8f0f4c, and flags that the temp 1.1 / min_p 0.10 samplers were tuned against the retired Mistral-family seat and never re-tuned for Gemma-4. Docs-only: no litellm_params touched, no routing change. --- stacks/litellm/conf/config.yaml | 35 ++++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index 7b2c6c4..95f6ad0 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -161,17 +161,30 @@ model_list: enable_thinking: true model_info: mode: chat - # char-rp -> GGUF dark-romantasy PROSE seat (:8016, llama.cpp, char-rp-gguf stack on - # ana-ml2 GPU 0). TheDrummer Magidonia-24B-v4.3 Q6_K — Magistral (Mistral) RP tune. - # NON-thinking: elite literary prose, zero refusal on dark/explicit scenes, ~65 tok/s, - # tight POV/instruction adherence (live-tested 2026-07-08). Replaced the broken Angel - # NVFP4 serve AND the earlier AEON-rp MTP twin. Sampler A/B-tuned 2026-07-08 vs the - # dvalin-smithy canonical: temp 1.1 / top_p 0.95 / min_p 0.10 / top_k 0, NO repetition - # penalty. min_p 0.10 richened imagery vs 0.03; rep 1.05 REJECTED (injected a stray - # markdown title in a grief scene — rep-style penalties hurt Drummer RP, matching the - # card). No enable_thinking kwarg — meaningless to the Mistral template. Alt prose model - # (swap via the stack .env): MS3.2-PaintedFantasy-v4.1-24B. Callers may override. - # docs/pfi/model-sampler-defaults.md. + # char-rp -> MeroMero-v2 NON-THINKING prose seat (:8016, vLLM, meromero-charrp stack on + # ana-ml2 GPU 0). G4-MeroMero-v2-31B NVFP4A16, **Gemma-4 base** (google/gemma-4-31B-it), + # 256K ctx, in-house quant. Replaced the GGUF/llama.cpp Magidonia-24B seat 2026-08-12. + # + # THIS SEAT EXISTS BECAUSE THE QWEN BASE THINKS INCESSANTLY. char-rp-reasoning is a + # Qwen3.x derivative and emits ~5-6k chars of CoT per turn no matter which Qwen RP tune + # is loaded — that is the base family, not the finetune, and no swap within it fixes it + # (measured 2026-08-16: Dark-Scarlett 6036 ch vs Fable-Fusion 5323 ch on the same + # prompts). Gemma-4 gives a genuinely non-thinking prose seat. Reach for THIS one when + # you want prose without a reasoning trace; reach for char-rp-reasoning when you want + # the deliberation. Best-of-breed per seat — deliberately NOT the same model. + # + # Serving flags are load-bearing (commit b8f0f4c): `--tool-call-parser gemma4 + # --enable-auto-tool-choice --reasoning-parser gemma4` AND + # `--default-chat-template-kwargs '{"enable_thinking": false}'`. That last flag is + # MANDATORY, not decorative — the gemma4 parser defaults enable_thinking to True, which + # pre-initialises the engine to REASONING and returns null `content` for all plain RP + # prose. Before it was set, every tools-bearing request also 400'd (no parser at all). + # Verified 0 chars reasoning / clean prose end-to-end 2026-08-16. + # + # Sampler note: the temp 1.1 / min_p 0.10 / top_k 0 values below were A/B-tuned + # 2026-07-08 against the retired Mistral-family Magidonia seat, NOT against MeroMero. + # They have not been re-tuned for Gemma-4 — treat as inherited, not canonical. + # Callers may override. docs/pfi/model-sampler-defaults.md; stacks/meromero-charrp/. - model_name: char-rp litellm_params: model: hosted_vllm/char-rp