Files

T

vh f960a73a79 feat(char-rp-gguf): swap reasoning seat Deckard-PKD → NEO-CODE (Heretic2-Thinking Qwen3.6-27B)

R36 gate (2026-07-14) validated NEO-CODE ships on all axes: tool-calling 0.967
(attach_tool 1.00, 0 runaways — #355 eliminated), prose genre-artifact-fine
(less clichéd than gen), refusal uncensored-as-spec + CSAM-clean.

#355 root cause was MODEL-level, not the reasoning-budget-forcing bug: Deckard
emitted Qwen's native qwen3_coder XML tool format malformed -> llama.cpp leaked
the closing tags into the arg value -> Bifrost attach_tool schema error -> retry
-> reasoning runaway to max_tokens. NEO-CODE emits the same native format cleanly
on the same seat/parser -> no schema error -> no runaway. The fix was the model
swap; there was never a wrong parser (the XML is Qwen3.5/3.6-native).

- reasoning seat: Deckard-PKD (Qwen3.5) -> NEO-CODE=Heretic2-Thinking (Qwen3.6-27B) Q5
- samplers: card defaults (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0), DRY dropped
- ctx: 256K max; custom llama.cpp kept (qwen3_coder parse + PR#25544 belt-and-suspenders)
- persistent-memory ACTIVE 1 marked resolved

2026-07-13 22:21:10 -07:00

llamacpp-custom

feat(char-rp-gguf): swap reasoning seat to custom llama.cpp (master 6eddde0 + PR #25544 )

2026-07-13 13:47:42 -07:00

.env.example

config(char-rp-gguf): Deckard reasoning seat to full 256K (GDN-hybrid KV cheap)

2026-07-08 15:41:02 -07:00

compose.yaml

feat(char-rp-gguf): swap reasoning seat Deckard-PKD → NEO-CODE (Heretic2-Thinking Qwen3.6-27B)

2026-07-13 22:21:10 -07:00

README.md

feat(char-rp-gguf): replace broken Angel NVFP4 with dual GGUF RP seat on ana-ml2 GPU0

2026-07-08 02:43:00 -07:00

README.md

char-rp-gguf

Dedicated GGUF character-RP seat on ana-ml2 GPU 0, served via llama.cpp (llama-server) — never Ollama. Replaces the broken ms32-24b-angel NVFP4 serve (self-quant produced garbage even at greedy).

Two co-located instances, alongside the 35B-A3B heretic gen (qwen36-27b-aeon, :8015):

service	port	gateway alias	model	role
`llama-charrp`	8016	`char-rp`	TheDrummer Magidonia-24B-v4.3 Q6_K	non-thinking prose seat
`llama-charrp-reasoning`	8018	`char-rp-reasoning`	ArliAI QwQ-32B-RpR-v4 Q5_K_M	managed-reasoning seat

Why these two models

Requirements (operator, 2026-07-08): highest prose quality; ≥50 tok/s (single-seat); very low refusal for dark explicit romantasy; dense; GGUF-via-llama.cpp; thinking strongly preferred; must fit alongside the 35B-A3B on GPU 0.

Magidonia-24B-v4.3 (char-rp) — Magistral (Mistral) dark-romantasy RP tune. Live-tested: elite literary prose, zero refusal on explicit scenes, precise POV/instruction adherence, ~65 tok/s at Q6_K. The prose star.
QwQ-32B-RpR-v4 (char-rp-reasoning) — QwQ reasoning RP tune whose reasoning data was generated with QwQ-abliterated, so it does not re-censor inside the think phase — the failure mode that disqualified Pantheon-Reasoning-27B (its DeepSeek-distilled traces reason themselves into refusals). llama.cpp manages QwQ reasoning: --reasoning on surfaces the trace in reasoning_content (content stays clean prose, no <think> leak) and --reasoning-budget caps the CoT. ~50 tok/s at Q5_K_M (46 at Q6).

Why not one model for both: no single dense 24–32B is both an elite non-thinking prose seat and a clean managed-reasoning seat on llama.cpp. Magidonia's Magistral [THINK] discipline is loose (won't reliably close [/THINK] on substantive reasoning → prose bleeds into reasoning_content, content empties). Cydonia-R1's <think> is emergent, so llama.cpp can't manage/cap it → runaway CoT that never reaches the prose. QwQ's template opens <think> natively → llama.cpp manages+caps it. Best-of-breed per seat. See the compose header for the one-model fallback.

Alternate prose model: MS3.2-PaintedFantasy-v4.1-24B (Magistral; more literary flair, looser POV adherence) — swap via CHARRP_MODEL in .env. All candidate GGUFs are pre-pulled to /tank/aimodels/llm/rp/.

Deploy

Canonical copies live here; authoritative copies live on the host under /opt/docker/compose/char-rp-gguf/.

# On ana-ml2 (models already pre-pulled to /tank/aimodels/llm/rp/):
scripts/deploy-stack.sh ana-ml2 char-rp-gguf     # from this workspace
# or, on the host:
cd /opt/docker/compose/char-rp-gguf && cp .env.example .env && docker compose up -d

GGUFs are pulled with a detached huggingface_hub container, e.g.:

docker run -d --entrypoint python3 -v /tank:/tank vllm/vllm-openai:latest -c \
  'from huggingface_hub import hf_hub_download as d; \
   d(repo_id="bartowski/TheDrummer_Magidonia-24B-v4.3-GGUF", \
     filename="TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf", local_dir="/tank/aimodels/llm/rp")'

Gateway wiring (ana-docker LiteLLM `:4000`)

char-rp → http://10.250.50.54:8016/v1, char-rp-reasoning → http://10.250.50.54:8018/v1. Sampler defaults (per the model cards):

char-rp (Magidonia / Mistral): temperature 1.0, top_p 0.95, min_p 0.03; no repetition penalty (Magistral is stable without one). Callers may override.
char-rp-reasoning (QwQ RpR-v4): temperature 1.0, top_k 40, min_p 0.02; no repetition / DRY / XTC penalties (RpR card is explicit about this). Reasoning surfaces in reasoning_content.

Do not carry the Qwen-specific chat_template_kwargs.enable_thinking on these aliases — it is meaningless to Mistral/QwQ templates. Reasoning is controlled server-side (--reasoning on on :8018).

Revert

docker compose -f /opt/docker/compose/char-rp-gguf/compose.yaml down

The retired ms32-24b-angel stack is left staged for reference (its NVFP4 quant is broken — do not re-serve it).

Verify

curl -s http://10.250.50.54:8016/v1/models        # char-rp up
curl -s http://10.250.50.54:8018/v1/models        # char-rp-reasoning up
nvidia-smi --query-gpu=index,memory.free --format=csv   # GPU 0 margin

README.md Unescape Escape

char-rp-gguf

Why these two models

Deploy

Gateway wiring (ana-docker LiteLLM :4000)

Revert

Verify

README.md

Gateway wiring (ana-docker LiteLLM `:4000`)