config(char-rp-gguf): char-rp to full 128K, funded by gen util 0.40->0.37
Completes the GPU0 rebalance discussed with the operator: - gen util 0.40 -> 0.37 (qwen36-27b-aeon stack .env) — frees ~2.9G of gen's IDLE KV headroom (gen KV usage runs 0-2%; concurrency-at-256K 4.74x -> 3.66x, invisible). - char-rp (Magidonia) 96K -> 131072 (full native 128K), q8_0 KV ~10.9G. - char-rp-reasoning unchanged at 40K (QwQ native max). - GPU0 ~4.4G margin, all 3 seats healthy, verified live. Deployed .env values already set on ana-ml2; this canonicalizes the intent + rationale.
This commit is contained in:
@@ -113,7 +113,7 @@ _As of 2026-07-08 — OFF-THE-SHELF INFERENCE STACK is the active work (home-tra
|
||||
General-assistant seat. Served by `/opt/docker/compose/qwen36-27b-aeon/` (`.env` repointed: `AEON_GEN_MODEL`,
|
||||
`AEON_GEN_QUANT=modelopt`, `AEON_GEN_GPU_MEM_UTIL=0.40`; the `--quantization` line is now
|
||||
`${AEON_GEN_QUANT}`-parameterized). Served-names `qwen3.6-27b-aeon`/`-thinking` (gateway gen/gen-reasoning stay
|
||||
transparent). **util 0.40 (~38G) → ~55–58G FREE on GPU0 for the RP seat.** AEON + the intermediate Pantheon-27B
|
||||
transparent). **util 0.37 (~35G; dropped from 0.40 on 2026-07-08 to fund char-rp 128K — the RP seats are now co-resident, ~4.4G GPU0 margin).** AEON + the intermediate Pantheon-27B
|
||||
serves are RETIRED.
|
||||
- **char-rp (:8016) = `TheDrummer_Magidonia-24B-v4.3-Q6_K` GGUF — LIVE + EXCELLENT.** Magistral (Mistral)
|
||||
dark-romantasy RP tune via llama.cpp (`char-rp-gguf` stack, container `llama-charrp`). NON-thinking PROSE seat:
|
||||
@@ -138,11 +138,12 @@ _As of 2026-07-08 — OFF-THE-SHELF INFERENCE STACK is the active work (home-tra
|
||||
old AEON trace-not-surfacing gap). One-model fallback (Magidonia both, lighter reasoning) documented in the stack
|
||||
header/README. Requirements met: prose#1, ≥50tps#2, low-refusal#3, dense#4, GGUF-not-Ollama#5, fit-GPU0#6, thinking#7.
|
||||
Candidate GGUFs also on disk for A/B: Cydonia-R1-24B-v4.1, PaintedFantasy-v4.1-24B, RpR-v4 Q6_K.
|
||||
**MAX CONTEXT (2026-07-08):** char-rp **96K** (Magidonia native 128K), char-rp-reasoning **40K** (QwQ native
|
||||
40960, YaRN-free max), **q8_0 KV cache both** (near-lossless, ~2× ctx/GB; verified coherent, no Qwen gibberish).
|
||||
GPU0 ~93/97G, ~4.3G margin (gen fixed-util + static KV → stable, won't OOM). 64/50 tps at max ctx. `kv_unified`
|
||||
→ a single conversation gets the FULL n_ctx (4 slots share the pool, not a /4 split). 128K char-rp only by
|
||||
dropping the reasoning seat's ctx (shared GPU0 budget). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE.
|
||||
**MAX CONTEXT (2026-07-08):** char-rp **128K** (Magidonia FULL native 131072), char-rp-reasoning **40K** (QwQ
|
||||
native 40960, YaRN-free max), **q8_0 KV cache both** (near-lossless, ~2× ctx/GB; verified coherent, no Qwen
|
||||
gibberish). **Funded by gen util 0.40→0.37** (freed ~2.9G of gen's IDLE KV headroom — gen KV usage runs 0-2%,
|
||||
so its 4.74×→3.66× concurrency-at-256K cut is invisible). GPU0 ~93/97G, **~4.4G margin** (gen fixed-util +
|
||||
static KV → stable, won't OOM). 64/50 tps. `kv_unified` → a single conversation gets the FULL n_ctx (4 slots
|
||||
share the pool, not a /4 split). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE / AEON_GEN_GPU_MEM_UTIL.
|
||||
|
||||
- **✅ LiteLLM gateway sampling defaults WIRED for all 4 seats (2026-07-08)** — dvalin-smithy canonical, infra-ops
|
||||
triaged + char-rp A/B-validated. **gen** (+summarizer-large twin, shared :8015 deployment): temp 0.7 / top_p 0.8 /
|
||||
|
||||
@@ -24,9 +24,11 @@ CHARRP_PORT=8016
|
||||
CHARRP_MODEL=rp/TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf
|
||||
# Alternate prose model (more literary flair, looser POV adherence) — uncomment to swap:
|
||||
# CHARRP_MODEL=rp/MS3.2-PaintedFantasy-v4.1-24B-Q6_K.gguf
|
||||
# Context. Magidonia native = 131072 (128K). 96K @ q8_0 KV ~ 8.4G VRAM; f16 KV doubles
|
||||
# that. Raise toward 131072 only if you drop the reasoning seat's ctx (shared GPU-0 budget).
|
||||
CHARRP_CTX=98304
|
||||
# Context. Magidonia native = 131072 (128K) — now at FULL 128K. 128K @ q8_0 KV ~ 10.9G VRAM.
|
||||
# Funded 2026-07-08 by dropping gen's util 0.40->0.37 (AEON_GEN_GPU_MEM_UTIL in the
|
||||
# qwen36-27b-aeon stack), which freed ~2.9G of gen's IDLE KV headroom (gen KV usage runs
|
||||
# 0-2%). GPU0 ~4.4G margin at: char-rp 128K + char-rp-reasoning 40K + gen@0.37, all q8_0 KV.
|
||||
CHARRP_CTX=131072
|
||||
# KV cache dtype: q8_0 (~half VRAM, near-lossless) or f16 (max fidelity, 2x VRAM).
|
||||
CHARRP_KV_TYPE=q8_0
|
||||
|
||||
|
||||
Reference in New Issue
Block a user