Files

T

vh aac4bcfa3e feat(litellm): wire canonical sampler defaults for all 4 gateway seats

dvalin-smithy canonical set, infra-ops triaged + char-rp A/B-validated on the live serve.

- gen (+summarizer-large twin): presence_penalty 1.0 -> 1.5 (Qwen3.6 non-thinking rec).
- gen-reasoning: temp 0.6 -> 1.0, presence 1.0 -> 1.5 (Qwen general-thinking profile;
  the old 0.6 was the coding sub-profile).
- char-rp: temp 1.0 -> 1.1, min_p 0.03 -> 0.10, top_k 0, NO rep. A/B on 2 dark-romantasy
  prompts: min_p 0.10 richened imagery; repeat_penalty 1.05 REJECTED (injected a stray
  markdown title, hurts Drummer/Magistral RP creativity per the card + dvalin's own note).
- char-rp-reasoning: add explicit top_p 0.95 (else per the RpR card: no rep/DRY/XTC).

Canonical reference: docs/pfi/model-sampler-defaults.md (mirrors dvalin's derivation).

2026-07-08 11:10:28 -07:00

3.7 KiB

Raw Blame History

Canonical sampler defaults — PFI/VastBlue LiteLLM gateway seats

Applied: 2026-07-08 · Gateway: ana-docker:4000 · Config: stacks/litellm/conf/config.yaml → /opt/docker/conf/litellm/config.yaml

Canonical high-quality sampler defaults for the four model seats, derived by dvalin-smithy-dev (full rationale + sources: dvalin-smithy/hoard-drafts/pfi-gateway-sampler-defaults-20260708.md), triaged + A/B-validated by infra-ops, and wired into the gateway. These are the gateway defaults; callers may override per request.

Optimized for output / prose quality (not throughput or determinism).

Engine surfaces

gen / gen-reasoning — vLLM 0.24 (OpenAI sampler surface). No native DRY/XTC → anti-repetition via presence_penalty. Thinking split via chat_template_kwargs.enable_thinking on distinct --served-model-names (avoids the shared-config-mutation footgun).
char-rp / char-rp-reasoning — llama.cpp / llama-server (supports min_p, top_k, DRY, XTC, dynatemp). min_p + top_p do the tail work; top_k 0 disables top-k.

The four seats (applied values)

1. gen — Qwen3.6-35B-A3B heretic (vLLM, non-thinking)

Also governs summarizer-large (shares the same qwen3.6-27b-aeon @ :8015 deployment → kept identical).

param	value
temperature	0.7
top_p	0.80
top_k	20
presence_penalty	1.5
repetition/frequency	1.0 / 0.0
enable_thinking	false

Source: Qwen3.6 README instruct/non-thinking rec. Change: presence_penalty 1.0 → 1.5.

2. gen-reasoning — same model (vLLM, thinking)

param	value
temperature	1.0
top_p	0.95
top_k	20
presence_penalty	1.5
repetition/frequency	1.0 / 0.0
enable_thinking	true

Source: Qwen3.6 README general thinking profile (NOT the temp-0.6 coding sub-profile — the prior default was that coding profile by mistake). Changes: temperature 0.6 → 1.0, presence_penalty 1.0 → 1.5. Reasoning is verbose (~9k chars) → callers set generous max_tokens (catalog default 32768). Optional per-route coding override: temp 0.6 / presence 0.0.

3. char-rp — Magidonia-24B-v4.3 (llama.cpp, non-thinking prose RP)

param	value
temperature	1.1
top_p	0.95
min_p	0.10
top_k	0 (disabled)
repetition/DRY/XTC	off

Source: dvalin canonical (Mistral-Small RP prose) A/B-validated by infra-ops on the live serve. Changes: temp 1.0 → 1.1, min_p 0.03 → 0.10. min_p 0.10 richened imagery vs 0.03 with no incoherence at temp 1.1. repeat_penalty 1.05 was REJECTED — in the A/B it injected a stray markdown title into a grief scene; rep-style penalties hurt Drummer/Magistral RP creativity (matches the model card and dvalin's own note). Alt prose model: MS3.2-PaintedFantasy-v4.1-24B (swap via the char-rp-gguf stack .env).

4. char-rp-reasoning — QwQ-32B-ArliAI-RpR-v4 (llama.cpp, reasoning RP)

param	value
temperature	1.0
top_p	0.95
top_k	40
min_p	0.02
repetition/DRY/XTC	off

Source: ArliAI RpR v4 card — explicit NO rep/DRY/XTC penalties. Change: added explicit top_p 0.95. Reasoning is server-side (--reasoning on, budget-capped); the CoT surfaces in reasoning_content with clean prose in content. SillyTavern wiring (non-sampler): include names = never; exact <think>/</think> tokens.

Changing a default

Edit the seat's litellm_params in stacks/litellm/conf/config.yaml, scp to /opt/docker/conf/litellm/config.yaml on ana-docker, docker restart litellm. (gen and summarizer-large must change together — same deployment.)

3.7 KiB Raw Blame History