feat(litellm): wire canonical sampler defaults for all 4 gateway seats
dvalin-smithy canonical set, infra-ops triaged + char-rp A/B-validated on the live serve. - gen (+summarizer-large twin): presence_penalty 1.0 -> 1.5 (Qwen3.6 non-thinking rec). - gen-reasoning: temp 0.6 -> 1.0, presence 1.0 -> 1.5 (Qwen general-thinking profile; the old 0.6 was the coding sub-profile). - char-rp: temp 1.0 -> 1.1, min_p 0.03 -> 0.10, top_k 0, NO rep. A/B on 2 dark-romantasy prompts: min_p 0.10 richened imagery; repeat_penalty 1.05 REJECTED (injected a stray markdown title, hurts Drummer/Magistral RP creativity per the card + dvalin's own note). - char-rp-reasoning: add explicit top_p 0.95 (else per the RpR card: no rep/DRY/XTC). Canonical reference: docs/pfi/model-sampler-defaults.md (mirrors dvalin's derivation).
This commit is contained in:
@@ -0,0 +1,74 @@
|
||||
# Canonical sampler defaults — PFI/VastBlue LiteLLM gateway seats
|
||||
|
||||
**Applied:** 2026-07-08 · **Gateway:** `ana-docker:4000` · **Config:** `stacks/litellm/conf/config.yaml` → `/opt/docker/conf/litellm/config.yaml`
|
||||
|
||||
Canonical high-quality sampler defaults for the four model seats, **derived by
|
||||
dvalin-smithy-dev** (full rationale + sources: `dvalin-smithy/hoard-drafts/pfi-gateway-sampler-defaults-20260708.md`),
|
||||
**triaged + A/B-validated by infra-ops**, and wired into the gateway. These are the
|
||||
gateway *defaults*; callers may override per request.
|
||||
|
||||
Optimized for **output / prose quality** (not throughput or determinism).
|
||||
|
||||
## Engine surfaces
|
||||
|
||||
- **gen / gen-reasoning** — vLLM 0.24 (OpenAI sampler surface). No native DRY/XTC → anti-repetition via `presence_penalty`. Thinking split via `chat_template_kwargs.enable_thinking` on distinct `--served-model-name`s (avoids the shared-config-mutation footgun).
|
||||
- **char-rp / char-rp-reasoning** — llama.cpp / llama-server (supports `min_p`, `top_k`, DRY, XTC, dynatemp). `min_p` + `top_p` do the tail work; `top_k 0` disables top-k.
|
||||
|
||||
## The four seats (applied values)
|
||||
|
||||
### 1. gen — Qwen3.6-35B-A3B heretic (vLLM, non-thinking)
|
||||
Also governs **summarizer-large** (shares the same `qwen3.6-27b-aeon` @ :8015 deployment → kept identical).
|
||||
|
||||
| param | value |
|
||||
|---|---|
|
||||
| temperature | 0.7 |
|
||||
| top_p | 0.80 |
|
||||
| top_k | 20 |
|
||||
| presence_penalty | **1.5** |
|
||||
| repetition/frequency | 1.0 / 0.0 |
|
||||
| enable_thinking | false |
|
||||
|
||||
*Source:* Qwen3.6 README instruct/non-thinking rec. *Change:* presence_penalty 1.0 → 1.5.
|
||||
|
||||
### 2. gen-reasoning — same model (vLLM, thinking)
|
||||
|
||||
| param | value |
|
||||
|---|---|
|
||||
| temperature | **1.0** |
|
||||
| top_p | 0.95 |
|
||||
| top_k | 20 |
|
||||
| presence_penalty | **1.5** |
|
||||
| repetition/frequency | 1.0 / 0.0 |
|
||||
| enable_thinking | true |
|
||||
|
||||
*Source:* Qwen3.6 README **general** thinking profile (NOT the temp-0.6 coding sub-profile — the prior default was that coding profile by mistake). *Changes:* temperature 0.6 → 1.0, presence_penalty 1.0 → 1.5. Reasoning is verbose (~9k chars) → callers set generous `max_tokens` (catalog default 32768). Optional per-route coding override: temp 0.6 / presence 0.0.
|
||||
|
||||
### 3. char-rp — Magidonia-24B-v4.3 (llama.cpp, non-thinking prose RP)
|
||||
|
||||
| param | value |
|
||||
|---|---|
|
||||
| temperature | **1.1** |
|
||||
| top_p | 0.95 |
|
||||
| min_p | **0.10** |
|
||||
| top_k | 0 (disabled) |
|
||||
| repetition/DRY/XTC | **off** |
|
||||
|
||||
*Source:* dvalin canonical (Mistral-Small RP prose) **A/B-validated by infra-ops** on the live serve. *Changes:* temp 1.0 → 1.1, min_p 0.03 → 0.10. **min_p 0.10 richened imagery vs 0.03** with no incoherence at temp 1.1. **repeat_penalty 1.05 was REJECTED** — in the A/B it injected a stray markdown title into a grief scene; rep-style penalties hurt Drummer/Magistral RP creativity (matches the model card and dvalin's own note). Alt prose model: `MS3.2-PaintedFantasy-v4.1-24B` (swap via the `char-rp-gguf` stack `.env`).
|
||||
|
||||
### 4. char-rp-reasoning — QwQ-32B-ArliAI-RpR-v4 (llama.cpp, reasoning RP)
|
||||
|
||||
| param | value |
|
||||
|---|---|
|
||||
| temperature | 1.0 |
|
||||
| top_p | **0.95** |
|
||||
| top_k | 40 |
|
||||
| min_p | 0.02 |
|
||||
| repetition/DRY/XTC | **off** |
|
||||
|
||||
*Source:* ArliAI RpR v4 card — explicit **NO** rep/DRY/XTC penalties. *Change:* added explicit top_p 0.95. Reasoning is server-side (`--reasoning on`, budget-capped); the CoT surfaces in `reasoning_content` with clean prose in `content`. SillyTavern wiring (non-sampler): `include names = never`; exact `<think>`/`</think>` tokens.
|
||||
|
||||
## Changing a default
|
||||
|
||||
Edit the seat's `litellm_params` in `stacks/litellm/conf/config.yaml`, `scp` to
|
||||
`/opt/docker/conf/litellm/config.yaml` on ana-docker, `docker restart litellm`.
|
||||
(`gen` and `summarizer-large` must change together — same deployment.)
|
||||
@@ -144,12 +144,14 @@ _As of 2026-07-08 — OFF-THE-SHELF INFERENCE STACK is the active work (home-tra
|
||||
→ a single conversation gets the FULL n_ctx (4 slots share the pool, not a /4 split). 128K char-rp only by
|
||||
dropping the reasoning seat's ctx (shared GPU0 budget). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE.
|
||||
|
||||
- **⏳ PENDING — wire LiteLLM gateway sampling defaults for GEN** (operator: "wire those settings in"). Config:
|
||||
`ana-docker:/opt/docker/conf/litellm/config.yaml`. **gen (Qwen3.6)** — thinking
|
||||
`temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 / presence_penalty 1.5 / rep 1.0`; non-thinking
|
||||
`temp 0.7 / top_p 0.8 / top_k 20 / pp 1.5` (NO rep-penalty in thinking mode). NOT yet applied to gen/gen-reasoning.
|
||||
**char-rp / char-rp-reasoning samplers ARE now wired** (Magistral: temp 1.0 / top_p 0.95 / min_p 0.03; QwQ RpR:
|
||||
temp 1.0 / top_k 40 / min_p 0.02, no rep/DRY — 2026-07-08).
|
||||
- **✅ LiteLLM gateway sampling defaults WIRED for all 4 seats (2026-07-08)** — dvalin-smithy canonical, infra-ops
|
||||
triaged + char-rp A/B-validated. **gen** (+summarizer-large twin, shared :8015 deployment): temp 0.7 / top_p 0.8 /
|
||||
top_k 20 / presence_penalty **1.5** (non-thinking). **gen-reasoning**: temp **1.0** / top_p 0.95 / top_k 20 /
|
||||
presence_penalty **1.5** (Qwen general-thinking profile — the old temp 0.6 was the CODING sub-profile by mistake;
|
||||
verbose ~9k-char CoT → callers need big max_tokens). **char-rp**: temp **1.1** / top_p 0.95 / min_p **0.10** /
|
||||
top_k 0, NO rep (A/B: min_p 0.10 richened imagery vs 0.03; rep 1.05 REJECTED — injected a markdown title in a grief
|
||||
scene, matches Drummer-no-rep). **char-rp-reasoning**: temp 1.0 / top_p **0.95** / top_k 40 / min_p 0.02, no rep/DRY.
|
||||
Canonical doc `docs/pfi/model-sampler-defaults.md`; dvalin derivation `dvalin-smithy/hoard-drafts/pfi-gateway-sampler-defaults-20260708.md`.
|
||||
|
||||
- **NVFP4/GGUF quant tooling (proven, on ana-ml2):** `/tank/aimodels/quant-work/` — `nvfp4_quant.py`
|
||||
(llm-compressor, MoE-aware `load_quantizable_moe`, Mistral-tuned ignore) + `.venv`; `qwen35_vision_nvfp4.py`
|
||||
|
||||
@@ -62,7 +62,7 @@ model_list:
|
||||
model: hosted_vllm/qwen3.6-27b-aeon
|
||||
api_base: http://10.250.50.54:8015/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
presence_penalty: 1.0
|
||||
presence_penalty: 1.5
|
||||
temperature: 0.7
|
||||
top_p: 0.8
|
||||
extra_body:
|
||||
@@ -115,15 +115,16 @@ model_list:
|
||||
# RETIRED with the displacement (→ 404, callers migrate to gen): qwen3.5-122-a10b
|
||||
# [-reasoning] + qwen-large[-reasoning] — they named a 122B that no longer exists;
|
||||
# aliasing a 27B under those is the naming footgun the qwen36-vl stack warns against.
|
||||
# presence_penalty: 1.0 INHERITED from qwopus (same-family abliterated/NVFP4 anti-
|
||||
# repetition damper, operator 2026-06-27) — RE-VALIDATE for AEON; NOT yet confirmed
|
||||
# for this model's repetition behavior. ---
|
||||
# presence_penalty: 1.5 — Qwen3.6 README anti-repetition rec for BOTH non-thinking and
|
||||
# thinking (dvalin-smithy canonical 2026-07-08, validated vs Qwen guidance). gen
|
||||
# non-thinking temp 0.7/top_p 0.8; gen-reasoning thinking temp 1.0/top_p 0.95 (the
|
||||
# GENERAL thinking profile, not the 0.6 coding sub-profile). docs/pfi/model-sampler-defaults.md. ---
|
||||
- model_name: gen
|
||||
litellm_params:
|
||||
model: hosted_vllm/qwen3.6-27b-aeon
|
||||
api_base: http://10.250.50.54:8015/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
presence_penalty: 1.0
|
||||
presence_penalty: 1.5
|
||||
temperature: 0.7
|
||||
top_p: 0.8
|
||||
extra_body:
|
||||
@@ -139,8 +140,8 @@ model_list:
|
||||
model: hosted_vllm/qwen3.6-27b-aeon-thinking
|
||||
api_base: http://10.250.50.54:8015/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
presence_penalty: 1.0
|
||||
temperature: 0.6
|
||||
presence_penalty: 1.5
|
||||
temperature: 1.0
|
||||
top_p: 0.95
|
||||
extra_body:
|
||||
top_k: 20
|
||||
@@ -152,19 +153,23 @@ model_list:
|
||||
# ana-ml2 GPU 0). TheDrummer Magidonia-24B-v4.3 Q6_K — Magistral (Mistral) RP tune.
|
||||
# NON-thinking: elite literary prose, zero refusal on dark/explicit scenes, ~65 tok/s,
|
||||
# tight POV/instruction adherence (live-tested 2026-07-08). Replaced the broken Angel
|
||||
# NVFP4 serve AND the earlier AEON-rp MTP twin. Magistral is stable WITHOUT a repetition
|
||||
# penalty (dropped the old 1.05); min_p 0.03 is the anti-slop knob. No enable_thinking
|
||||
# kwarg — meaningless to the Mistral template. Alt prose model (swap via the stack .env):
|
||||
# MS3.2-PaintedFantasy-v4.1-24B. Callers may override the sampler.
|
||||
# NVFP4 serve AND the earlier AEON-rp MTP twin. Sampler A/B-tuned 2026-07-08 vs the
|
||||
# dvalin-smithy canonical: temp 1.1 / top_p 0.95 / min_p 0.10 / top_k 0, NO repetition
|
||||
# penalty. min_p 0.10 richened imagery vs 0.03; rep 1.05 REJECTED (injected a stray
|
||||
# markdown title in a grief scene — rep-style penalties hurt Drummer RP, matching the
|
||||
# card). No enable_thinking kwarg — meaningless to the Mistral template. Alt prose model
|
||||
# (swap via the stack .env): MS3.2-PaintedFantasy-v4.1-24B. Callers may override.
|
||||
# docs/pfi/model-sampler-defaults.md.
|
||||
- model_name: char-rp
|
||||
litellm_params:
|
||||
model: hosted_vllm/magidonia-24b-v4.3
|
||||
api_base: http://10.250.50.54:8016/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
temperature: 1.0
|
||||
temperature: 1.1
|
||||
top_p: 0.95
|
||||
extra_body:
|
||||
min_p: 0.03
|
||||
min_p: 0.10
|
||||
top_k: 0
|
||||
model_info:
|
||||
mode: chat
|
||||
# char-rp-reasoning -> GGUF managed-REASONING seat (:8018, llama.cpp, char-rp-gguf stack).
|
||||
@@ -181,6 +186,7 @@ model_list:
|
||||
api_base: http://10.250.50.54:8018/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
temperature: 1.0
|
||||
top_p: 0.95
|
||||
extra_body:
|
||||
top_k: 40
|
||||
min_p: 0.02
|
||||
|
||||
Reference in New Issue
Block a user