vh
b268f93035
char-rp -> TheDrummer Magidonia-24B-v4.3 Q6_K (Magistral prose, ~65 tok/s,
zero refusal, tight POV) via llama.cpp (:8016).
char-rp-reasoning -> ArliAI QwQ-32B-RpR-v4 Q5_K_M (abliterated managed reasoning,
~52 tok/s, reasoning surfaces in reasoning_content) via llama.cpp (:8018).
- New canonical stack stacks/char-rp-gguf/ (llama-server x2, GPU0-pinned, ~86/97G
co-resident with gen). GGUF sidesteps the vLLM-NVFP4 + Mistral-tokenizer traps that
killed the Angel serve. Never Ollama.
- Best-of-breed per seat: no single dense 24-32B is both an elite non-thinking prose
seat AND a clean managed-reasoning seat on llama.cpp (Magidonia [THINK] boundary is
loose; Cydonia-R1 <think> runs away; QwQ is template-managed). Pantheon-Reasoning-27B
stays rejected (re-censors in <think>; RpR-v4 abliterated reasoning is the fix).
- Gateway rewired: char-rp->:8016, char-rp-reasoning->:8018, Mistral/QwQ samplers,
dropped the Qwen enable_thinking kwarg. One-model Magidonia fallback documented.
- Retired the ms32-24b-angel stack.
96 lines
4.3 KiB
Markdown
96 lines
4.3 KiB
Markdown
# char-rp-gguf
|
||
|
||
Dedicated GGUF **character-RP seat** on ana-ml2 GPU 0, served via llama.cpp
|
||
(`llama-server`) — **never Ollama**. Replaces the broken `ms32-24b-angel` NVFP4
|
||
serve (self-quant produced garbage even at greedy).
|
||
|
||
Two co-located instances, alongside the 35B-A3B heretic `gen` (`qwen36-27b-aeon`,
|
||
:8015):
|
||
|
||
| service | port | gateway alias | model | role |
|
||
|---|---|---|---|---|
|
||
| `llama-charrp` | 8016 | `char-rp` | TheDrummer **Magidonia-24B-v4.3** Q6_K | non-thinking **prose** seat |
|
||
| `llama-charrp-reasoning` | 8018 | `char-rp-reasoning` | ArliAI **QwQ-32B-RpR-v4** Q5_K_M | managed-**reasoning** seat |
|
||
|
||
## Why these two models
|
||
|
||
Requirements (operator, 2026-07-08): highest prose quality; ≥50 tok/s (single-seat);
|
||
very low refusal for dark explicit romantasy; dense; GGUF-via-llama.cpp; thinking
|
||
strongly preferred; must fit alongside the 35B-A3B on GPU 0.
|
||
|
||
- **Magidonia-24B-v4.3** (`char-rp`) — Magistral (Mistral) dark-romantasy RP tune.
|
||
Live-tested: elite literary prose, **zero refusal** on explicit scenes, precise
|
||
POV/instruction adherence, **~65 tok/s** at Q6_K. The prose star.
|
||
- **QwQ-32B-RpR-v4** (`char-rp-reasoning`) — QwQ reasoning RP tune whose reasoning
|
||
data was generated with **QwQ-abliterated**, so it does **not re-censor inside the
|
||
think phase** — the failure mode that disqualified Pantheon-Reasoning-27B (its
|
||
DeepSeek-distilled traces reason themselves into refusals). llama.cpp **manages**
|
||
QwQ reasoning: `--reasoning on` surfaces the trace in `reasoning_content` (content
|
||
stays clean prose, no `<think>` leak) and `--reasoning-budget` caps the CoT.
|
||
~50 tok/s at Q5_K_M (46 at Q6).
|
||
|
||
**Why not one model for both:** no single dense 24–32B is both an elite non-thinking
|
||
prose seat and a clean *managed*-reasoning seat on llama.cpp. Magidonia's Magistral
|
||
`[THINK]` discipline is loose (won't reliably close `[/THINK]` on substantive
|
||
reasoning → prose bleeds into `reasoning_content`, `content` empties). Cydonia-R1's
|
||
`<think>` is emergent, so llama.cpp can't manage/cap it → runaway CoT that never
|
||
reaches the prose. QwQ's template opens `<think>` natively → llama.cpp manages+caps
|
||
it. Best-of-breed per seat. See the compose header for the one-model fallback.
|
||
|
||
**Alternate prose model:** `MS3.2-PaintedFantasy-v4.1-24B` (Magistral; more literary
|
||
flair, looser POV adherence) — swap via `CHARRP_MODEL` in `.env`. All candidate GGUFs
|
||
are pre-pulled to `/tank/aimodels/llm/rp/`.
|
||
|
||
## Deploy
|
||
|
||
Canonical copies live here; authoritative copies live on the host under
|
||
`/opt/docker/compose/char-rp-gguf/`.
|
||
|
||
```bash
|
||
# On ana-ml2 (models already pre-pulled to /tank/aimodels/llm/rp/):
|
||
scripts/deploy-stack.sh ana-ml2 char-rp-gguf # from this workspace
|
||
# or, on the host:
|
||
cd /opt/docker/compose/char-rp-gguf && cp .env.example .env && docker compose up -d
|
||
```
|
||
|
||
GGUFs are pulled with a detached `huggingface_hub` container, e.g.:
|
||
|
||
```bash
|
||
docker run -d --entrypoint python3 -v /tank:/tank vllm/vllm-openai:latest -c \
|
||
'from huggingface_hub import hf_hub_download as d; \
|
||
d(repo_id="bartowski/TheDrummer_Magidonia-24B-v4.3-GGUF", \
|
||
filename="TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf", local_dir="/tank/aimodels/llm/rp")'
|
||
```
|
||
|
||
## Gateway wiring (ana-docker LiteLLM `:4000`)
|
||
|
||
`char-rp` → `http://10.250.50.54:8016/v1`, `char-rp-reasoning` →
|
||
`http://10.250.50.54:8018/v1`. Sampler defaults (per the model cards):
|
||
|
||
- **char-rp (Magidonia / Mistral):** `temperature 1.0, top_p 0.95, min_p 0.03`;
|
||
no repetition penalty (Magistral is stable without one). Callers may override.
|
||
- **char-rp-reasoning (QwQ RpR-v4):** `temperature 1.0, top_k 40, min_p 0.02`;
|
||
**no repetition / DRY / XTC penalties** (RpR card is explicit about this).
|
||
Reasoning surfaces in `reasoning_content`.
|
||
|
||
Do **not** carry the Qwen-specific `chat_template_kwargs.enable_thinking` on these
|
||
aliases — it is meaningless to Mistral/QwQ templates. Reasoning is controlled
|
||
server-side (`--reasoning on` on :8018).
|
||
|
||
## Revert
|
||
|
||
```bash
|
||
docker compose -f /opt/docker/compose/char-rp-gguf/compose.yaml down
|
||
```
|
||
|
||
The retired `ms32-24b-angel` stack is left staged for reference (its NVFP4 quant is
|
||
broken — do not re-serve it).
|
||
|
||
## Verify
|
||
|
||
```bash
|
||
curl -s http://10.250.50.54:8016/v1/models # char-rp up
|
||
curl -s http://10.250.50.54:8018/v1/models # char-rp-reasoning up
|
||
nvidia-smi --query-gpu=index,memory.free --format=csv # GPU 0 margin
|
||
```
|