vh
b268f93035
char-rp -> TheDrummer Magidonia-24B-v4.3 Q6_K (Magistral prose, ~65 tok/s,
zero refusal, tight POV) via llama.cpp (:8016).
char-rp-reasoning -> ArliAI QwQ-32B-RpR-v4 Q5_K_M (abliterated managed reasoning,
~52 tok/s, reasoning surfaces in reasoning_content) via llama.cpp (:8018).
- New canonical stack stacks/char-rp-gguf/ (llama-server x2, GPU0-pinned, ~86/97G
co-resident with gen). GGUF sidesteps the vLLM-NVFP4 + Mistral-tokenizer traps that
killed the Angel serve. Never Ollama.
- Best-of-breed per seat: no single dense 24-32B is both an elite non-thinking prose
seat AND a clean managed-reasoning seat on llama.cpp (Magidonia [THINK] boundary is
loose; Cydonia-R1 <think> runs away; QwQ is template-managed). Pantheon-Reasoning-27B
stays rejected (re-censors in <think>; RpR-v4 abliterated reasoning is the fix).
- Gateway rewired: char-rp->:8016, char-rp-reasoning->:8018, Mistral/QwQ samplers,
dropped the Qwen enable_thinking kwarg. One-model Magidonia fallback documented.
- Retired the ms32-24b-angel stack.
47 lines
2.3 KiB
Bash
47 lines
2.3 KiB
Bash
# char-rp-gguf stack tunables. Copy to `.env` on ana-ml2 before deploying.
|
|
#
|
|
# cp .env.example .env
|
|
# docker compose up -d
|
|
#
|
|
# Keep compose.yaml constant; edit values here.
|
|
|
|
# llama.cpp server image (bundles /app/llama-server; already present on ana-ml2).
|
|
LLAMA_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda
|
|
|
|
# GPU 0 = the on-demand large-model card (co-resident with `gen`). GPU 1 is the
|
|
# always-on vLLM trio (granite/embed/rerank/reward) — do NOT put this stack there.
|
|
CHARRP_GPU_ID=0
|
|
|
|
# GGUF store on the host (mounted read-only at /models). Candidate GGUFs live in
|
|
# /tank/aimodels/llm/rp/ (pre-pulled).
|
|
MODELS_DIR=/tank/aimodels/llm
|
|
|
|
# ── PROSE seat (char-rp) ────────────────────────────────────────────────────
|
|
CHARRP_CONTAINER=llama-charrp
|
|
CHARRP_PORT=8016
|
|
# Model path relative to /models. Default = Magidonia-24B-v4.3 (Magistral RP tune):
|
|
# elite dark-romantasy prose, zero refusal, ~65 tok/s, tight POV adherence.
|
|
CHARRP_MODEL=rp/TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf
|
|
# Alternate prose model (more literary flair, looser POV adherence) — uncomment to swap:
|
|
# CHARRP_MODEL=rp/MS3.2-PaintedFantasy-v4.1-24B-Q6_K.gguf
|
|
CHARRP_CTX=16384
|
|
|
|
# ── REASONING seat (char-rp-reasoning) ──────────────────────────────────────
|
|
CHARRP_REASONING_CONTAINER=llama-charrp-reasoning
|
|
CHARRP_REASONING_PORT=8018
|
|
# Default = QwQ-32B-ArliAI-RpR-v4 Q5_K_M (abliterated reasoning → no re-censor;
|
|
# llama.cpp-managed CoT). ~50 tok/s @ Q5. Use Q6_K (~46 tok/s) for a touch more
|
|
# quality if speed is not binding.
|
|
CHARRP_REASONING_MODEL=rp/QwQ-32B-ArliAI-RpR-v4-Q5_K_M.gguf
|
|
CHARRP_REASONING_CTX=16384
|
|
# Thinking-token cap (QwQ over-thinks otherwise → starves the prose). 300-500 = a
|
|
# concise, useful scene-plan before the response.
|
|
CHARRP_REASONING_BUDGET=400
|
|
|
|
# ── ONE-MODEL FALLBACK (consistent Mistral style, lighter reasoning) ─────────
|
|
# To collapse both seats onto Magidonia (drop QwQ): set
|
|
# CHARRP_REASONING_MODEL=rp/TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf
|
|
# and remove the --reasoning* flags from the reasoning service in compose.yaml
|
|
# (Magistral reasons only when the caller's system prompt contains "/think";
|
|
# managed but LIGHT — see the compose header for why QwQ is the default).
|