vh
5f79b40982
Operator wanted a reasoning-RP model that tolerates DRY (RpR-v4 forbids rep/DRY -> a 1/30 loop tail). Ran the full A/B on brokkr's 30-prompt D1 suite (content-only, slop-scored): - Deckard-PKD (Qwen3.5-27B, DavidAU creative tune) WON: 0/30 loops, 0/30 refusals, clean managed reasoning (native Qwen3.5 <think>/enable_thinking), DRY-tolerant, ~57 tok/s, runs on the base llama-swap b8840 image. -> now the char-rp-reasoning seat (:8018). - RpR-v4: 0 refusals but 1/30 loop (no-DRY). Pantheon-27B: clean slop but 7/30 explicit refusals + needs the newer ggml-org/llama.cpp image (Qwen3.6 won't load on b8840). Snowdrop + Gembrain (Gemma-4): floored (llama.cpp can't manage their reasoning without the vetoed template hacks). Losers kept on disk as alternates. - char-rp (Magidonia) unchanged; gen unchanged. gateway char-rp-reasoning -> Deckard sampler (temp 1.0/top_p 0.95/top_k 40/min_p 0.05; DRY server-side).
60 lines
3.5 KiB
Bash
60 lines
3.5 KiB
Bash
# char-rp-gguf stack tunables. Copy to `.env` on ana-ml2 before deploying.
|
|
#
|
|
# cp .env.example .env
|
|
# docker compose up -d
|
|
#
|
|
# Keep compose.yaml constant; edit values here.
|
|
|
|
# llama.cpp server image (bundles /app/llama-server; already present on ana-ml2).
|
|
LLAMA_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda
|
|
|
|
# GPU 0 = the on-demand large-model card (co-resident with `gen`). GPU 1 is the
|
|
# always-on vLLM trio (granite/embed/rerank/reward) — do NOT put this stack there.
|
|
CHARRP_GPU_ID=0
|
|
|
|
# GGUF store on the host (mounted read-only at /models). Candidate GGUFs live in
|
|
# /tank/aimodels/llm/rp/ (pre-pulled).
|
|
MODELS_DIR=/tank/aimodels/llm
|
|
|
|
# ── PROSE seat (char-rp) ────────────────────────────────────────────────────
|
|
CHARRP_CONTAINER=llama-charrp
|
|
CHARRP_PORT=8016
|
|
# Model path relative to /models. Default = Magidonia-24B-v4.3 (Magistral RP tune):
|
|
# elite dark-romantasy prose, zero refusal, ~65 tok/s, tight POV adherence.
|
|
CHARRP_MODEL=rp/TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf
|
|
# Alternate prose model (more literary flair, looser POV adherence) — uncomment to swap:
|
|
# CHARRP_MODEL=rp/MS3.2-PaintedFantasy-v4.1-24B-Q6_K.gguf
|
|
# Context. Magidonia native = 131072 (128K) — now at FULL 128K. 128K @ q8_0 KV ~ 10.9G VRAM.
|
|
# Funded 2026-07-08 by dropping gen's util 0.40->0.37 (AEON_GEN_GPU_MEM_UTIL in the
|
|
# qwen36-27b-aeon stack), which freed ~2.9G of gen's IDLE KV headroom (gen KV usage runs
|
|
# 0-2%). GPU0 ~4.4G margin at: char-rp 128K + char-rp-reasoning 40K + gen@0.37, all q8_0 KV.
|
|
CHARRP_CTX=131072
|
|
# KV cache dtype: q8_0 (~half VRAM, near-lossless) or f16 (max fidelity, 2x VRAM).
|
|
CHARRP_KV_TYPE=q8_0
|
|
|
|
# ── REASONING seat (char-rp-reasoning) ──────────────────────────────────────
|
|
CHARRP_REASONING_CONTAINER=llama-charrp-reasoning
|
|
CHARRP_REASONING_PORT=8018
|
|
# Default = Qwen3.5-27B-Deckard-PKD-Heretic-Uncensored-Thinking i1-Q5_K_M (DavidAU creative
|
|
# tune, native Qwen3.5 managed reasoning, DRY-tolerant). A/B WINNER 2026-07-08: 0/30 loops,
|
|
# 0/30 refusals, clean slop; beat RpR-v4 (1/30 loop, no-DRY), Pantheon (7/30 refusals),
|
|
# Snowdrop + Gembrain (template-incompatible with llama.cpp managed reasoning).
|
|
CHARRP_REASONING_MODEL=rp/Qwen3.5-27B-Deckard-PKD-Heretic-Uncensored-Thinking.i1-Q5_K_M.gguf
|
|
# Qwen3.5-27B native ctx is large; 40960 = a sane reasoning-seat cap. ~5G VRAM @ q8_0 KV.
|
|
CHARRP_REASONING_CTX=40960
|
|
# KV cache dtype: q8_0 verified coherent; f16 if a future model gibbers.
|
|
CHARRP_REASONING_KV_TYPE=q8_0
|
|
# Thinking-token cap (concise scene-plan before the response). 300-500 is a good band.
|
|
CHARRP_REASONING_BUDGET=400
|
|
# DRY anti-repetition multiplier (0 disables). Deckard tolerates DRY; DRY is the correct
|
|
# anti-loop tool for the Qwen/QwQ family (a repetition PENALTY worsens their looping).
|
|
CHARRP_REASONING_DRY=0.8
|
|
|
|
# ── ALTERNATES / FALLBACKS (all pre-pulled to /tank/aimodels/llm/rp/) ────────
|
|
# Reasoning-seat A/B losers, kept on disk: QwQ-32B-ArliAI-RpR-v4-Q5_K_M (no-DRY → 1/30 loop
|
|
# tail); Gryphe_Pantheon-Reasoning-27B-Q5_K_M (7/30 explicit refusals; also needs the newer
|
|
# ggml-org/llama.cpp:server-cuda image — Qwen3.6 won't load on llama-swap b8840);
|
|
# Gemma-4-Gembrain-31B (Gemma-4 reasoning-parser broken on llama.cpp — floored).
|
|
# One-model fallback (collapse the reasoning seat onto Magidonia, lighter /think reasoning):
|
|
# CHARRP_REASONING_MODEL=rp/TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf + drop the DRY/--reasoning flags.
|