vh
a2b026d499
Operator-directed 2026-07-16. Moved the char-rp prose seat (Magidonia-24B,
llama-charrp) from GPU0 to GPU1 (CHARRP_GPU_ID 0->1; recreate llama-charrp
only -- the var is shared with the retired GGUF reasoning service), then
re-optimized every context-relevant seat on both cards to native/max context
with acceptable headroom:
GPU0 (both seats now 256K native, ~14 GB reserve):
- char-rp-reasoning 150K -> 256K (heretic2 stack, util 0.38->0.46, 1.56x)
- gen 256K, max-num-seqs 16 -> 32 (qwen36-27b-aeon, util 0.30->0.42, 5.43x)
GPU1 (~6.7 GB headroom):
- granite 64K -> 128K full-chapter (vllm stack, util 0.18->0.27, 1.50x)
- char-rp 128K native (4 slots), selene/reward/embed/rerank unchanged
All seats gateway-verified healthy. Live .env changes on ana-ml2 with per-stack
backups (*-20260716). Templates updated to match; the qwen36-27b-aeon template
carries a NOTE that its served-name/model still lag the 2026-07-08 gen model swap
(35B-A3B-heretic) -- separate reconciliation. persistent-memory records the full
layout + the util-floor / per-model-KV-cost lessons.
Note: the heretic2-charrp-reasoning stack (char-rp-reasoning's live config) is
still untracked in git -- standing open-loop, its .env change lives server-side only.
char-rp-gguf
Dedicated GGUF character-RP seat on ana-ml2 GPU 0, served via llama.cpp
(llama-server) — never Ollama. Replaces the broken ms32-24b-angel NVFP4
serve (self-quant produced garbage even at greedy).
Two co-located instances, alongside the 35B-A3B heretic gen (qwen36-27b-aeon,
:8015):
| service | port | gateway alias | model | role |
|---|---|---|---|---|
llama-charrp |
8016 | char-rp |
TheDrummer Magidonia-24B-v4.3 Q6_K | non-thinking prose seat |
llama-charrp-reasoning |
8018 | char-rp-reasoning |
ArliAI QwQ-32B-RpR-v4 Q5_K_M | managed-reasoning seat |
Why these two models
Requirements (operator, 2026-07-08): highest prose quality; ≥50 tok/s (single-seat); very low refusal for dark explicit romantasy; dense; GGUF-via-llama.cpp; thinking strongly preferred; must fit alongside the 35B-A3B on GPU 0.
- Magidonia-24B-v4.3 (
char-rp) — Magistral (Mistral) dark-romantasy RP tune. Live-tested: elite literary prose, zero refusal on explicit scenes, precise POV/instruction adherence, ~65 tok/s at Q6_K. The prose star. - QwQ-32B-RpR-v4 (
char-rp-reasoning) — QwQ reasoning RP tune whose reasoning data was generated with QwQ-abliterated, so it does not re-censor inside the think phase — the failure mode that disqualified Pantheon-Reasoning-27B (its DeepSeek-distilled traces reason themselves into refusals). llama.cpp manages QwQ reasoning:--reasoning onsurfaces the trace inreasoning_content(content stays clean prose, no<think>leak) and--reasoning-budgetcaps the CoT. ~50 tok/s at Q5_K_M (46 at Q6).
Why not one model for both: no single dense 24–32B is both an elite non-thinking
prose seat and a clean managed-reasoning seat on llama.cpp. Magidonia's Magistral
[THINK] discipline is loose (won't reliably close [/THINK] on substantive
reasoning → prose bleeds into reasoning_content, content empties). Cydonia-R1's
<think> is emergent, so llama.cpp can't manage/cap it → runaway CoT that never
reaches the prose. QwQ's template opens <think> natively → llama.cpp manages+caps
it. Best-of-breed per seat. See the compose header for the one-model fallback.
Alternate prose model: MS3.2-PaintedFantasy-v4.1-24B (Magistral; more literary
flair, looser POV adherence) — swap via CHARRP_MODEL in .env. All candidate GGUFs
are pre-pulled to /tank/aimodels/llm/rp/.
Deploy
Canonical copies live here; authoritative copies live on the host under
/opt/docker/compose/char-rp-gguf/.
# On ana-ml2 (models already pre-pulled to /tank/aimodels/llm/rp/):
scripts/deploy-stack.sh ana-ml2 char-rp-gguf # from this workspace
# or, on the host:
cd /opt/docker/compose/char-rp-gguf && cp .env.example .env && docker compose up -d
GGUFs are pulled with a detached huggingface_hub container, e.g.:
docker run -d --entrypoint python3 -v /tank:/tank vllm/vllm-openai:latest -c \
'from huggingface_hub import hf_hub_download as d; \
d(repo_id="bartowski/TheDrummer_Magidonia-24B-v4.3-GGUF", \
filename="TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf", local_dir="/tank/aimodels/llm/rp")'
Gateway wiring (ana-docker LiteLLM :4000)
char-rp → http://10.250.50.54:8016/v1, char-rp-reasoning →
http://10.250.50.54:8018/v1. Sampler defaults (per the model cards):
- char-rp (Magidonia / Mistral):
temperature 1.0, top_p 0.95, min_p 0.03; no repetition penalty (Magistral is stable without one). Callers may override. - char-rp-reasoning (QwQ RpR-v4):
temperature 1.0, top_k 40, min_p 0.02; no repetition / DRY / XTC penalties (RpR card is explicit about this). Reasoning surfaces inreasoning_content.
Do not carry the Qwen-specific chat_template_kwargs.enable_thinking on these
aliases — it is meaningless to Mistral/QwQ templates. Reasoning is controlled
server-side (--reasoning on on :8018).
Revert
docker compose -f /opt/docker/compose/char-rp-gguf/compose.yaml down
The retired ms32-24b-angel stack is left staged for reference (its NVFP4 quant is
broken — do not re-serve it).
Verify
curl -s http://10.250.50.54:8016/v1/models # char-rp up
curl -s http://10.250.50.54:8018/v1/models # char-rp-reasoning up
nvidia-smi --query-gpu=index,memory.free --format=csv # GPU 0 margin