feat(char-rp-gguf): max context — char-rp 96K, char-rp-reasoning 40K, q8_0 KV
Raise both RP seats to near-max context within the GPU0 budget using q8_0 KV cache (near-lossless 8-bit, ~2x context/GB, flash-attn-backed). Verified coherent on both (no Qwen KV-quant gibberish) at 64/50 tok/s. - char-rp (Magidonia): 16K -> 96K (native 128K; 128K would starve the reasoning seat). - char-rp-reasoning (QwQ): 16K -> 40960 (QwQ native max; beyond needs YaRN). - kv_unified=true -> a single conversation gets the full n_ctx (slots share the pool). - GPU0 ~93/97G, ~4.3G margin (gen fixed-util + static KV = stable, no OOM risk). - New .env knobs: CHARRP_CTX / CHARRP_REASONING_CTX / CHARRP_KV_TYPE / CHARRP_REASONING_KV_TYPE.
This commit is contained in:
@@ -138,6 +138,11 @@ _As of 2026-07-08 — OFF-THE-SHELF INFERENCE STACK is the active work (home-tra
|
||||
old AEON trace-not-surfacing gap). One-model fallback (Magidonia both, lighter reasoning) documented in the stack
|
||||
header/README. Requirements met: prose#1, ≥50tps#2, low-refusal#3, dense#4, GGUF-not-Ollama#5, fit-GPU0#6, thinking#7.
|
||||
Candidate GGUFs also on disk for A/B: Cydonia-R1-24B-v4.1, PaintedFantasy-v4.1-24B, RpR-v4 Q6_K.
|
||||
**MAX CONTEXT (2026-07-08):** char-rp **96K** (Magidonia native 128K), char-rp-reasoning **40K** (QwQ native
|
||||
40960, YaRN-free max), **q8_0 KV cache both** (near-lossless, ~2× ctx/GB; verified coherent, no Qwen gibberish).
|
||||
GPU0 ~93/97G, ~4.3G margin (gen fixed-util + static KV → stable, won't OOM). 64/50 tps at max ctx. `kv_unified`
|
||||
→ a single conversation gets the FULL n_ctx (4 slots share the pool, not a /4 split). 128K char-rp only by
|
||||
dropping the reasoning seat's ctx (shared GPU0 budget). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE.
|
||||
|
||||
- **⏳ PENDING — wire LiteLLM gateway sampling defaults for GEN** (operator: "wire those settings in"). Config:
|
||||
`ana-docker:/opt/docker/conf/litellm/config.yaml`. **gen (Qwen3.6)** — thinking
|
||||
|
||||
Reference in New Issue
Block a user