config(char-rp-gguf): char-rp to full 128K, funded by gen util 0.40->0.37

Completes the GPU0 rebalance discussed with the operator:
- gen util 0.40 -> 0.37 (qwen36-27b-aeon stack .env) — frees ~2.9G of gen's IDLE KV
  headroom (gen KV usage runs 0-2%; concurrency-at-256K 4.74x -> 3.66x, invisible).
- char-rp (Magidonia) 96K -> 131072 (full native 128K), q8_0 KV ~10.9G.
- char-rp-reasoning unchanged at 40K (QwQ native max).
- GPU0 ~4.4G margin, all 3 seats healthy, verified live.

Deployed .env values already set on ana-ml2; this canonicalizes the intent + rationale.
This commit is contained in:
2026-07-08 13:01:22 -07:00
parent d085604825
commit f49c4e40a3
2 changed files with 12 additions and 9 deletions
+7 -6
View File
@@ -113,7 +113,7 @@ _As of 2026-07-08 — OFF-THE-SHELF INFERENCE STACK is the active work (home-tra
General-assistant seat. Served by `/opt/docker/compose/qwen36-27b-aeon/` (`.env` repointed: `AEON_GEN_MODEL`,
`AEON_GEN_QUANT=modelopt`, `AEON_GEN_GPU_MEM_UTIL=0.40`; the `--quantization` line is now
`${AEON_GEN_QUANT}`-parameterized). Served-names `qwen3.6-27b-aeon`/`-thinking` (gateway gen/gen-reasoning stay
transparent). **util 0.40 (~38G) → ~5558G FREE on GPU0 for the RP seat.** AEON + the intermediate Pantheon-27B
transparent). **util 0.37 (~35G; dropped from 0.40 on 2026-07-08 to fund char-rp 128K — the RP seats are now co-resident, ~4.4G GPU0 margin).** AEON + the intermediate Pantheon-27B
serves are RETIRED.
- **char-rp (:8016) = `TheDrummer_Magidonia-24B-v4.3-Q6_K` GGUF — LIVE + EXCELLENT.** Magistral (Mistral)
dark-romantasy RP tune via llama.cpp (`char-rp-gguf` stack, container `llama-charrp`). NON-thinking PROSE seat:
@@ -138,11 +138,12 @@ _As of 2026-07-08 — OFF-THE-SHELF INFERENCE STACK is the active work (home-tra
old AEON trace-not-surfacing gap). One-model fallback (Magidonia both, lighter reasoning) documented in the stack
header/README. Requirements met: prose#1, ≥50tps#2, low-refusal#3, dense#4, GGUF-not-Ollama#5, fit-GPU0#6, thinking#7.
Candidate GGUFs also on disk for A/B: Cydonia-R1-24B-v4.1, PaintedFantasy-v4.1-24B, RpR-v4 Q6_K.
**MAX CONTEXT (2026-07-08):** char-rp **96K** (Magidonia native 128K), char-rp-reasoning **40K** (QwQ native
40960, YaRN-free max), **q8_0 KV cache both** (near-lossless, ~2× ctx/GB; verified coherent, no Qwen gibberish).
GPU0 ~93/97G, ~4.3G margin (gen fixed-util + static KV → stable, won't OOM). 64/50 tps at max ctx. `kv_unified`
→ a single conversation gets the FULL n_ctx (4 slots share the pool, not a /4 split). 128K char-rp only by
dropping the reasoning seat's ctx (shared GPU0 budget). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE.
**MAX CONTEXT (2026-07-08):** char-rp **128K** (Magidonia FULL native 131072), char-rp-reasoning **40K** (QwQ
native 40960, YaRN-free max), **q8_0 KV cache both** (near-lossless, ~2× ctx/GB; verified coherent, no Qwen
gibberish). **Funded by gen util 0.40→0.37** (freed ~2.9G of gen's IDLE KV headroom — gen KV usage runs 0-2%,
so its 4.74×→3.66× concurrency-at-256K cut is invisible). GPU0 ~93/97G, **~4.4G margin** (gen fixed-util +
static KV → stable, won't OOM). 64/50 tps. `kv_unified` → a single conversation gets the FULL n_ctx (4 slots
share the pool, not a /4 split). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE / AEON_GEN_GPU_MEM_UTIL.
- **✅ LiteLLM gateway sampling defaults WIRED for all 4 seats (2026-07-08)** — dvalin-smithy canonical, infra-ops
triaged + char-rp A/B-validated. **gen** (+summarizer-large twin, shared :8015 deployment): temp 0.7 / top_p 0.8 /
+5 -3
View File
@@ -24,9 +24,11 @@ CHARRP_PORT=8016
CHARRP_MODEL=rp/TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf
# Alternate prose model (more literary flair, looser POV adherence) — uncomment to swap:
# CHARRP_MODEL=rp/MS3.2-PaintedFantasy-v4.1-24B-Q6_K.gguf
# Context. Magidonia native = 131072 (128K). 96K @ q8_0 KV ~ 8.4G VRAM; f16 KV doubles
# that. Raise toward 131072 only if you drop the reasoning seat's ctx (shared GPU-0 budget).
CHARRP_CTX=98304
# Context. Magidonia native = 131072 (128K) — now at FULL 128K. 128K @ q8_0 KV ~ 10.9G VRAM.
# Funded 2026-07-08 by dropping gen's util 0.40->0.37 (AEON_GEN_GPU_MEM_UTIL in the
# qwen36-27b-aeon stack), which freed ~2.9G of gen's IDLE KV headroom (gen KV usage runs
# 0-2%). GPU0 ~4.4G margin at: char-rp 128K + char-rp-reasoning 40K + gen@0.37, all q8_0 KV.
CHARRP_CTX=131072
# KV cache dtype: q8_0 (~half VRAM, near-lossless) or f16 (max fidelity, 2x VRAM).
CHARRP_KV_TYPE=q8_0