diff --git a/persistent-memory.md b/persistent-memory.md index d30f7e2..7719bd8 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -138,6 +138,11 @@ _As of 2026-07-08 — OFF-THE-SHELF INFERENCE STACK is the active work (home-tra old AEON trace-not-surfacing gap). One-model fallback (Magidonia both, lighter reasoning) documented in the stack header/README. Requirements met: prose#1, ≥50tps#2, low-refusal#3, dense#4, GGUF-not-Ollama#5, fit-GPU0#6, thinking#7. Candidate GGUFs also on disk for A/B: Cydonia-R1-24B-v4.1, PaintedFantasy-v4.1-24B, RpR-v4 Q6_K. + **MAX CONTEXT (2026-07-08):** char-rp **96K** (Magidonia native 128K), char-rp-reasoning **40K** (QwQ native + 40960, YaRN-free max), **q8_0 KV cache both** (near-lossless, ~2× ctx/GB; verified coherent, no Qwen gibberish). + GPU0 ~93/97G, ~4.3G margin (gen fixed-util + static KV → stable, won't OOM). 64/50 tps at max ctx. `kv_unified` + → a single conversation gets the FULL n_ctx (4 slots share the pool, not a /4 split). 128K char-rp only by + dropping the reasoning seat's ctx (shared GPU0 budget). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE. - **⏳ PENDING — wire LiteLLM gateway sampling defaults for GEN** (operator: "wire those settings in"). Config: `ana-docker:/opt/docker/conf/litellm/config.yaml`. **gen (Qwen3.6)** — thinking diff --git a/stacks/char-rp-gguf/.env.example b/stacks/char-rp-gguf/.env.example index 0ca3670..8d957c6 100644 --- a/stacks/char-rp-gguf/.env.example +++ b/stacks/char-rp-gguf/.env.example @@ -24,7 +24,11 @@ CHARRP_PORT=8016 CHARRP_MODEL=rp/TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf # Alternate prose model (more literary flair, looser POV adherence) — uncomment to swap: # CHARRP_MODEL=rp/MS3.2-PaintedFantasy-v4.1-24B-Q6_K.gguf -CHARRP_CTX=16384 +# Context. Magidonia native = 131072 (128K). 96K @ q8_0 KV ~ 8.4G VRAM; f16 KV doubles +# that. Raise toward 131072 only if you drop the reasoning seat's ctx (shared GPU-0 budget). +CHARRP_CTX=98304 +# KV cache dtype: q8_0 (~half VRAM, near-lossless) or f16 (max fidelity, 2x VRAM). +CHARRP_KV_TYPE=q8_0 # ── REASONING seat (char-rp-reasoning) ────────────────────────────────────── CHARRP_REASONING_CONTAINER=llama-charrp-reasoning @@ -33,7 +37,10 @@ CHARRP_REASONING_PORT=8018 # llama.cpp-managed CoT). ~50 tok/s @ Q5. Use Q6_K (~46 tok/s) for a touch more # quality if speed is not binding. CHARRP_REASONING_MODEL=rp/QwQ-32B-ArliAI-RpR-v4-Q5_K_M.gguf -CHARRP_REASONING_CTX=16384 +# QwQ native ctx = 40960 (its max without YaRN). ~5.5G VRAM @ q8_0 KV. +CHARRP_REASONING_CTX=40960 +# KV cache dtype (Qwen-arch): q8_0 verified coherent here; f16 if a future model gibbers. +CHARRP_REASONING_KV_TYPE=q8_0 # Thinking-token cap (QwQ over-thinks otherwise → starves the prose). 300-500 = a # concise, useful scene-plan before the response. CHARRP_REASONING_BUDGET=400 diff --git a/stacks/char-rp-gguf/compose.yaml b/stacks/char-rp-gguf/compose.yaml index cedd0ef..9c47d6a 100644 --- a/stacks/char-rp-gguf/compose.yaml +++ b/stacks/char-rp-gguf/compose.yaml @@ -69,9 +69,15 @@ services: - --n-gpu-layers - "999" - --ctx-size - - "${CHARRP_CTX:-16384}" + - "${CHARRP_CTX:-98304}" - --flash-attn - on + # q8_0 KV cache ~halves KV VRAM (8-bit, near-lossless) → ~2x the context per GB. + # Mistral/Magistral handles q8 KV cleanly. Set f16 in .env to disable. + - --cache-type-k + - ${CHARRP_KV_TYPE:-q8_0} + - --cache-type-v + - ${CHARRP_KV_TYPE:-q8_0} - --jinja healthcheck: test: ["CMD-SHELL", "curl -fsS http://localhost:8080/health >/dev/null || exit 1"] @@ -115,9 +121,16 @@ services: - --n-gpu-layers - "999" - --ctx-size - - "${CHARRP_REASONING_CTX:-16384}" + - "${CHARRP_REASONING_CTX:-40960}" - --flash-attn - on + # QwQ native ctx = 40960 (beyond needs YaRN → quality loss; don't). Qwen-arch can be + # KV-quant-sensitive: q8_0 is verified coherent here, but flip to f16 in .env if a + # future model shows gibberish. + - --cache-type-k + - ${CHARRP_REASONING_KV_TYPE:-q8_0} + - --cache-type-v + - ${CHARRP_REASONING_KV_TYPE:-q8_0} - --jinja # QwQ reasoning is template-native → llama.cpp manages it. --reasoning on surfaces # the trace in reasoning_content (content stays clean prose); --reasoning-budget