vh
a2b026d499
Operator-directed 2026-07-16. Moved the char-rp prose seat (Magidonia-24B,
llama-charrp) from GPU0 to GPU1 (CHARRP_GPU_ID 0->1; recreate llama-charrp
only -- the var is shared with the retired GGUF reasoning service), then
re-optimized every context-relevant seat on both cards to native/max context
with acceptable headroom:
GPU0 (both seats now 256K native, ~14 GB reserve):
- char-rp-reasoning 150K -> 256K (heretic2 stack, util 0.38->0.46, 1.56x)
- gen 256K, max-num-seqs 16 -> 32 (qwen36-27b-aeon, util 0.30->0.42, 5.43x)
GPU1 (~6.7 GB headroom):
- granite 64K -> 128K full-chapter (vllm stack, util 0.18->0.27, 1.50x)
- char-rp 128K native (4 slots), selene/reward/embed/rerank unchanged
All seats gateway-verified healthy. Live .env changes on ana-ml2 with per-stack
backups (*-20260716). Templates updated to match; the qwen36-27b-aeon template
carries a NOTE that its served-name/model still lag the 2026-07-08 gen model swap
(35B-A3B-heretic) -- separate reconciliation. persistent-memory records the full
layout + the util-floor / per-model-KV-cost lessons.
Note: the heretic2-charrp-reasoning stack (char-rp-reasoning's live config) is
still untracked in git -- standing open-loop, its .env change lives server-side only.
63 lines
3.8 KiB
Bash
63 lines
3.8 KiB
Bash
# char-rp-gguf stack tunables. Copy to `.env` on ana-ml2 before deploying.
|
|
#
|
|
# cp .env.example .env
|
|
# docker compose up -d
|
|
#
|
|
# Keep compose.yaml constant; edit values here.
|
|
|
|
# llama.cpp server image (bundles /app/llama-server; already present on ana-ml2).
|
|
LLAMA_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda
|
|
|
|
# GPU pin for the char-rp PROSE seat (llama-charrp). MOVED to GPU 1 on 2026-07-16 to
|
|
# decongest GPU 0 (freed ~29 GB there for gen + char-rp-reasoning to grow to 256K).
|
|
# NOTE: this var is ALSO read by the (retired, stopped) llama-charrp-reasoning service
|
|
# below — recreate ONLY llama-charrp when changing it, or split into a dedicated var.
|
|
CHARRP_GPU_ID=1
|
|
|
|
# GGUF store on the host (mounted read-only at /models). Candidate GGUFs live in
|
|
# /tank/aimodels/llm/rp/ (pre-pulled).
|
|
MODELS_DIR=/tank/aimodels/llm
|
|
|
|
# ── PROSE seat (char-rp) ────────────────────────────────────────────────────
|
|
CHARRP_CONTAINER=llama-charrp
|
|
CHARRP_PORT=8016
|
|
# Model path relative to /models. Default = Magidonia-24B-v4.3 (Magistral RP tune):
|
|
# elite dark-romantasy prose, zero refusal, ~65 tok/s, tight POV adherence.
|
|
CHARRP_MODEL=rp/TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf
|
|
# Alternate prose model (more literary flair, looser POV adherence) — uncomment to swap:
|
|
# CHARRP_MODEL=rp/MS3.2-PaintedFantasy-v4.1-24B-Q6_K.gguf
|
|
# Context. Magidonia native = 131072 (128K) — now at FULL 128K. 128K @ q8_0 KV ~ 10.9G VRAM.
|
|
# Funded 2026-07-08 by dropping gen's util 0.40->0.37 (AEON_GEN_GPU_MEM_UTIL in the
|
|
# qwen36-27b-aeon stack), which freed ~2.9G of gen's IDLE KV headroom (gen KV usage runs
|
|
# 0-2%). GPU0 ~4.4G margin at: char-rp 128K + char-rp-reasoning 40K + gen@0.37, all q8_0 KV.
|
|
CHARRP_CTX=131072
|
|
# KV cache dtype: q8_0 (~half VRAM, near-lossless) or f16 (max fidelity, 2x VRAM).
|
|
CHARRP_KV_TYPE=q8_0
|
|
|
|
# ── REASONING seat (char-rp-reasoning) ──────────────────────────────────────
|
|
CHARRP_REASONING_CONTAINER=llama-charrp-reasoning
|
|
CHARRP_REASONING_PORT=8018
|
|
# Default = Qwen3.5-27B-Deckard-PKD-Heretic-Uncensored-Thinking i1-Q5_K_M (DavidAU creative
|
|
# tune, native Qwen3.5 managed reasoning, DRY-tolerant). A/B WINNER 2026-07-08: 0/30 loops,
|
|
# 0/30 refusals, clean slop; beat RpR-v4 (1/30 loop, no-DRY), Pantheon (7/30 refusals),
|
|
# Snowdrop + Gembrain (template-incompatible with llama.cpp managed reasoning).
|
|
CHARRP_REASONING_MODEL=rp/Qwen3.5-27B-Deckard-PKD-Heretic-Uncensored-Thinking.i1-Q5_K_M.gguf
|
|
# Deckard = Qwen3.5-27B GDN-hybrid (only 16 KV-caching layers → KV is CHEAP). Native ctx 262144
|
|
# (256K, no YaRN). Full 256K @ q8_0 KV ≈ 8.5G VRAM (~0.033G/1K-tok, vs Magidonia ~0.085). Fits GPU0.
|
|
CHARRP_REASONING_CTX=262144
|
|
# KV cache dtype: q8_0 verified coherent; f16 if a future model gibbers.
|
|
CHARRP_REASONING_KV_TYPE=q8_0
|
|
# Thinking-token cap (concise scene-plan before the response). 300-500 is a good band.
|
|
CHARRP_REASONING_BUDGET=400
|
|
# DRY anti-repetition multiplier (0 disables). Deckard tolerates DRY; DRY is the correct
|
|
# anti-loop tool for the Qwen/QwQ family (a repetition PENALTY worsens their looping).
|
|
CHARRP_REASONING_DRY=0.8
|
|
|
|
# ── ALTERNATES / FALLBACKS (all pre-pulled to /tank/aimodels/llm/rp/) ────────
|
|
# Reasoning-seat A/B losers, kept on disk: QwQ-32B-ArliAI-RpR-v4-Q5_K_M (no-DRY → 1/30 loop
|
|
# tail); Gryphe_Pantheon-Reasoning-27B-Q5_K_M (7/30 explicit refusals; also needs the newer
|
|
# ggml-org/llama.cpp:server-cuda image — Qwen3.6 won't load on llama-swap b8840);
|
|
# Gemma-4-Gembrain-31B (Gemma-4 reasoning-parser broken on llama.cpp — floored).
|
|
# One-model fallback (collapse the reasoning seat onto Magidonia, lighter /think reasoning):
|
|
# CHARRP_REASONING_MODEL=rp/TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf + drop the DRY/--reasoning flags.
|