feat(char-rp-gguf): max context — char-rp 96K, char-rp-reasoning 40K, q8_0 KV

Raise both RP seats to near-max context within the GPU0 budget using q8_0 KV cache
(near-lossless 8-bit, ~2x context/GB, flash-attn-backed). Verified coherent on both
(no Qwen KV-quant gibberish) at 64/50 tok/s.

- char-rp (Magidonia): 16K -> 96K (native 128K; 128K would starve the reasoning seat).
- char-rp-reasoning (QwQ): 16K -> 40960 (QwQ native max; beyond needs YaRN).
- kv_unified=true -> a single conversation gets the full n_ctx (slots share the pool).
- GPU0 ~93/97G, ~4.3G margin (gen fixed-util + static KV = stable, no OOM risk).
- New .env knobs: CHARRP_CTX / CHARRP_REASONING_CTX / CHARRP_KV_TYPE / CHARRP_REASONING_KV_TYPE.
This commit is contained in:
vh
2026-07-08 07:29:49 -07:00
parent b268f93035
commit f5706046b1
3 changed files with 29 additions and 4 deletions
+5
View File
@@ -138,6 +138,11 @@ _As of 2026-07-08 — OFF-THE-SHELF INFERENCE STACK is the active work (home-tra
old AEON trace-not-surfacing gap). One-model fallback (Magidonia both, lighter reasoning) documented in the stack
header/README. Requirements met: prose#1, ≥50tps#2, low-refusal#3, dense#4, GGUF-not-Ollama#5, fit-GPU0#6, thinking#7.
Candidate GGUFs also on disk for A/B: Cydonia-R1-24B-v4.1, PaintedFantasy-v4.1-24B, RpR-v4 Q6_K.
**MAX CONTEXT (2026-07-08):** char-rp **96K** (Magidonia native 128K), char-rp-reasoning **40K** (QwQ native
40960, YaRN-free max), **q8_0 KV cache both** (near-lossless, ~2× ctx/GB; verified coherent, no Qwen gibberish).
GPU0 ~93/97G, ~4.3G margin (gen fixed-util + static KV → stable, won't OOM). 64/50 tps at max ctx. `kv_unified`
→ a single conversation gets the FULL n_ctx (4 slots share the pool, not a /4 split). 128K char-rp only by
dropping the reasoning seat's ctx (shared GPU0 budget). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE.
- **⏳ PENDING — wire LiteLLM gateway sampling defaults for GEN** (operator: "wire those settings in"). Config:
`ana-docker:/opt/docker/conf/litellm/config.yaml`. **gen (Qwen3.6)** — thinking