vh
f5706046b1
Raise both RP seats to near-max context within the GPU0 budget using q8_0 KV cache (near-lossless 8-bit, ~2x context/GB, flash-attn-backed). Verified coherent on both (no Qwen KV-quant gibberish) at 64/50 tok/s. - char-rp (Magidonia): 16K -> 96K (native 128K; 128K would starve the reasoning seat). - char-rp-reasoning (QwQ): 16K -> 40960 (QwQ native max; beyond needs YaRN). - kv_unified=true -> a single conversation gets the full n_ctx (slots share the pool). - GPU0 ~93/97G, ~4.3G margin (gen fixed-util + static KV = stable, no OOM risk). - New .env knobs: CHARRP_CTX / CHARRP_REASONING_CTX / CHARRP_KV_TYPE / CHARRP_REASONING_KV_TYPE.
54 lines
2.8 KiB
Bash
54 lines
2.8 KiB
Bash
# char-rp-gguf stack tunables. Copy to `.env` on ana-ml2 before deploying.
|
|
#
|
|
# cp .env.example .env
|
|
# docker compose up -d
|
|
#
|
|
# Keep compose.yaml constant; edit values here.
|
|
|
|
# llama.cpp server image (bundles /app/llama-server; already present on ana-ml2).
|
|
LLAMA_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda
|
|
|
|
# GPU 0 = the on-demand large-model card (co-resident with `gen`). GPU 1 is the
|
|
# always-on vLLM trio (granite/embed/rerank/reward) — do NOT put this stack there.
|
|
CHARRP_GPU_ID=0
|
|
|
|
# GGUF store on the host (mounted read-only at /models). Candidate GGUFs live in
|
|
# /tank/aimodels/llm/rp/ (pre-pulled).
|
|
MODELS_DIR=/tank/aimodels/llm
|
|
|
|
# ── PROSE seat (char-rp) ────────────────────────────────────────────────────
|
|
CHARRP_CONTAINER=llama-charrp
|
|
CHARRP_PORT=8016
|
|
# Model path relative to /models. Default = Magidonia-24B-v4.3 (Magistral RP tune):
|
|
# elite dark-romantasy prose, zero refusal, ~65 tok/s, tight POV adherence.
|
|
CHARRP_MODEL=rp/TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf
|
|
# Alternate prose model (more literary flair, looser POV adherence) — uncomment to swap:
|
|
# CHARRP_MODEL=rp/MS3.2-PaintedFantasy-v4.1-24B-Q6_K.gguf
|
|
# Context. Magidonia native = 131072 (128K). 96K @ q8_0 KV ~ 8.4G VRAM; f16 KV doubles
|
|
# that. Raise toward 131072 only if you drop the reasoning seat's ctx (shared GPU-0 budget).
|
|
CHARRP_CTX=98304
|
|
# KV cache dtype: q8_0 (~half VRAM, near-lossless) or f16 (max fidelity, 2x VRAM).
|
|
CHARRP_KV_TYPE=q8_0
|
|
|
|
# ── REASONING seat (char-rp-reasoning) ──────────────────────────────────────
|
|
CHARRP_REASONING_CONTAINER=llama-charrp-reasoning
|
|
CHARRP_REASONING_PORT=8018
|
|
# Default = QwQ-32B-ArliAI-RpR-v4 Q5_K_M (abliterated reasoning → no re-censor;
|
|
# llama.cpp-managed CoT). ~50 tok/s @ Q5. Use Q6_K (~46 tok/s) for a touch more
|
|
# quality if speed is not binding.
|
|
CHARRP_REASONING_MODEL=rp/QwQ-32B-ArliAI-RpR-v4-Q5_K_M.gguf
|
|
# QwQ native ctx = 40960 (its max without YaRN). ~5.5G VRAM @ q8_0 KV.
|
|
CHARRP_REASONING_CTX=40960
|
|
# KV cache dtype (Qwen-arch): q8_0 verified coherent here; f16 if a future model gibbers.
|
|
CHARRP_REASONING_KV_TYPE=q8_0
|
|
# Thinking-token cap (QwQ over-thinks otherwise → starves the prose). 300-500 = a
|
|
# concise, useful scene-plan before the response.
|
|
CHARRP_REASONING_BUDGET=400
|
|
|
|
# ── ONE-MODEL FALLBACK (consistent Mistral style, lighter reasoning) ─────────
|
|
# To collapse both seats onto Magidonia (drop QwQ): set
|
|
# CHARRP_REASONING_MODEL=rp/TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf
|
|
# and remove the --reasoning* flags from the reasoning service in compose.yaml
|
|
# (Magistral reasons only when the caller's system prompt contains "/think";
|
|
# managed but LIGHT — see the compose header for why QwQ is the default).
|