From f49c4e40a38b83a83449669dde4c4a5040fc5962 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 8 Jul 2026 13:01:22 -0700 Subject: [PATCH] config(char-rp-gguf): char-rp to full 128K, funded by gen util 0.40->0.37 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completes the GPU0 rebalance discussed with the operator: - gen util 0.40 -> 0.37 (qwen36-27b-aeon stack .env) — frees ~2.9G of gen's IDLE KV headroom (gen KV usage runs 0-2%; concurrency-at-256K 4.74x -> 3.66x, invisible). - char-rp (Magidonia) 96K -> 131072 (full native 128K), q8_0 KV ~10.9G. - char-rp-reasoning unchanged at 40K (QwQ native max). - GPU0 ~4.4G margin, all 3 seats healthy, verified live. Deployed .env values already set on ana-ml2; this canonicalizes the intent + rationale. --- persistent-memory.md | 13 +++++++------ stacks/char-rp-gguf/.env.example | 8 +++++--- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/persistent-memory.md b/persistent-memory.md index 059cb43..35a87b5 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -113,7 +113,7 @@ _As of 2026-07-08 — OFF-THE-SHELF INFERENCE STACK is the active work (home-tra General-assistant seat. Served by `/opt/docker/compose/qwen36-27b-aeon/` (`.env` repointed: `AEON_GEN_MODEL`, `AEON_GEN_QUANT=modelopt`, `AEON_GEN_GPU_MEM_UTIL=0.40`; the `--quantization` line is now `${AEON_GEN_QUANT}`-parameterized). Served-names `qwen3.6-27b-aeon`/`-thinking` (gateway gen/gen-reasoning stay - transparent). **util 0.40 (~38G) → ~55–58G FREE on GPU0 for the RP seat.** AEON + the intermediate Pantheon-27B + transparent). **util 0.37 (~35G; dropped from 0.40 on 2026-07-08 to fund char-rp 128K — the RP seats are now co-resident, ~4.4G GPU0 margin).** AEON + the intermediate Pantheon-27B serves are RETIRED. - **char-rp (:8016) = `TheDrummer_Magidonia-24B-v4.3-Q6_K` GGUF — LIVE + EXCELLENT.** Magistral (Mistral) dark-romantasy RP tune via llama.cpp (`char-rp-gguf` stack, container `llama-charrp`). NON-thinking PROSE seat: @@ -138,11 +138,12 @@ _As of 2026-07-08 — OFF-THE-SHELF INFERENCE STACK is the active work (home-tra old AEON trace-not-surfacing gap). One-model fallback (Magidonia both, lighter reasoning) documented in the stack header/README. Requirements met: prose#1, ≥50tps#2, low-refusal#3, dense#4, GGUF-not-Ollama#5, fit-GPU0#6, thinking#7. Candidate GGUFs also on disk for A/B: Cydonia-R1-24B-v4.1, PaintedFantasy-v4.1-24B, RpR-v4 Q6_K. - **MAX CONTEXT (2026-07-08):** char-rp **96K** (Magidonia native 128K), char-rp-reasoning **40K** (QwQ native - 40960, YaRN-free max), **q8_0 KV cache both** (near-lossless, ~2× ctx/GB; verified coherent, no Qwen gibberish). - GPU0 ~93/97G, ~4.3G margin (gen fixed-util + static KV → stable, won't OOM). 64/50 tps at max ctx. `kv_unified` - → a single conversation gets the FULL n_ctx (4 slots share the pool, not a /4 split). 128K char-rp only by - dropping the reasoning seat's ctx (shared GPU0 budget). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE. + **MAX CONTEXT (2026-07-08):** char-rp **128K** (Magidonia FULL native 131072), char-rp-reasoning **40K** (QwQ + native 40960, YaRN-free max), **q8_0 KV cache both** (near-lossless, ~2× ctx/GB; verified coherent, no Qwen + gibberish). **Funded by gen util 0.40→0.37** (freed ~2.9G of gen's IDLE KV headroom — gen KV usage runs 0-2%, + so its 4.74×→3.66× concurrency-at-256K cut is invisible). GPU0 ~93/97G, **~4.4G margin** (gen fixed-util + + static KV → stable, won't OOM). 64/50 tps. `kv_unified` → a single conversation gets the FULL n_ctx (4 slots + share the pool, not a /4 split). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE / AEON_GEN_GPU_MEM_UTIL. - **✅ LiteLLM gateway sampling defaults WIRED for all 4 seats (2026-07-08)** — dvalin-smithy canonical, infra-ops triaged + char-rp A/B-validated. **gen** (+summarizer-large twin, shared :8015 deployment): temp 0.7 / top_p 0.8 / diff --git a/stacks/char-rp-gguf/.env.example b/stacks/char-rp-gguf/.env.example index 8d957c6..b7fd7bd 100644 --- a/stacks/char-rp-gguf/.env.example +++ b/stacks/char-rp-gguf/.env.example @@ -24,9 +24,11 @@ CHARRP_PORT=8016 CHARRP_MODEL=rp/TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf # Alternate prose model (more literary flair, looser POV adherence) — uncomment to swap: # CHARRP_MODEL=rp/MS3.2-PaintedFantasy-v4.1-24B-Q6_K.gguf -# Context. Magidonia native = 131072 (128K). 96K @ q8_0 KV ~ 8.4G VRAM; f16 KV doubles -# that. Raise toward 131072 only if you drop the reasoning seat's ctx (shared GPU-0 budget). -CHARRP_CTX=98304 +# Context. Magidonia native = 131072 (128K) — now at FULL 128K. 128K @ q8_0 KV ~ 10.9G VRAM. +# Funded 2026-07-08 by dropping gen's util 0.40->0.37 (AEON_GEN_GPU_MEM_UTIL in the +# qwen36-27b-aeon stack), which freed ~2.9G of gen's IDLE KV headroom (gen KV usage runs +# 0-2%). GPU0 ~4.4G margin at: char-rp 128K + char-rp-reasoning 40K + gen@0.37, all q8_0 KV. +CHARRP_CTX=131072 # KV cache dtype: q8_0 (~half VRAM, near-lossless) or f16 (max fidelity, 2x VRAM). CHARRP_KV_TYPE=q8_0