vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(litellm): wire canonical sampler defaults for all 4 gateway seats

Browse Source 
dvalin-smithy canonical set, infra-ops triaged + char-rp A/B-validated on the live serve.

- gen (+summarizer-large twin): presence_penalty 1.0 -> 1.5 (Qwen3.6 non-thinking rec).
- gen-reasoning: temp 0.6 -> 1.0, presence 1.0 -> 1.5 (Qwen general-thinking profile;
  the old 0.6 was the coding sub-profile).
- char-rp: temp 1.0 -> 1.1, min_p 0.03 -> 0.10, top_k 0, NO rep. A/B on 2 dark-romantasy
  prompts: min_p 0.10 richened imagery; repeat_penalty 1.05 REJECTED (injected a stray
  markdown title, hurts Drummer/Magistral RP creativity per the card + dvalin's own note).
- char-rp-reasoning: add explicit top_p 0.95 (else per the RpR card: no rep/DRY/XTC).

Canonical reference: docs/pfi/model-sampler-defaults.md (mirrors dvalin's derivation).
This commit is contained in:
vh
2026-07-08 11:10:28 -07:00
parent f5706046b1
commit aac4bcfa3e
3 changed files with 101 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+8 -6
View File
@@ -144,12 +144,14 @@ _As of 2026-07-08 — OFF-THE-SHELF INFERENCE STACK is the active work (home-tra
→ a single conversation gets the FULL n_ctx (4 slots share the pool, not a /4 split). 128K char-rp only by
dropping the reasoning seat's ctx (shared GPU0 budget). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE.
- **⏳ PENDING — wire LiteLLM gateway sampling defaults for GEN** (operator: "wire those settings in"). Config:
`ana-docker:/opt/docker/conf/litellm/config.yaml`. **gen (Qwen3.6)** — thinking
`temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 / presence_penalty 1.5 / rep 1.0`; non-thinking
`temp 0.7 / top_p 0.8 / top_k 20 / pp 1.5` (NO rep-penalty in thinking mode). NOT yet applied to gen/gen-reasoning.
**char-rp / char-rp-reasoning samplers ARE now wired** (Magistral: temp 1.0 / top_p 0.95 / min_p 0.03; QwQ RpR:
temp 1.0 / top_k 40 / min_p 0.02, no rep/DRY — 2026-07-08).
- **✅ LiteLLM gateway sampling defaults WIRED for all 4 seats (2026-07-08)** — dvalin-smithy canonical, infra-ops
triaged + char-rp A/B-validated. **gen** (+summarizer-large twin, shared :8015 deployment): temp 0.7 / top_p 0.8 /
top_k 20 / presence_penalty **1.5** (non-thinking). **gen-reasoning**: temp **1.0** / top_p 0.95 / top_k 20 /
presence_penalty **1.5** (Qwen general-thinking profile — the old temp 0.6 was the CODING sub-profile by mistake;
verbose ~9k-char CoT → callers need big max_tokens). **char-rp**: temp **1.1** / top_p 0.95 / min_p **0.10** /
top_k 0, NO rep (A/B: min_p 0.10 richened imagery vs 0.03; rep 1.05 REJECTED — injected a markdown title in a grief
scene, matches Drummer-no-rep). **char-rp-reasoning**: temp 1.0 / top_p **0.95** / top_k 40 / min_p 0.02, no rep/DRY.
Canonical doc `docs/pfi/model-sampler-defaults.md`; dvalin derivation `dvalin-smithy/hoard-drafts/pfi-gateway-sampler-defaults-20260708.md`.
- **NVFP4/GGUF quant tooling (proven, on ana-ml2):** `/tank/aimodels/quant-work/` — `nvfp4_quant.py`
(llm-compressor, MoE-aware `load_quantizable_moe`, Mistral-tuned ignore) + `.venv`; `qwen35_vision_nvfp4.py`