vh
8894854127
The live gateway config has served char-rp-reasoning as deckard-pkd-27b (:8018) since the 2026-07-08 A/B; the standalone doc had frozen on QwQ-RpR-v4. Corrects seat 4 (backend + samplers + server-side DRY/reasoning-budget notes). Also snapshots session state in persistent-memory.md: phantom-qwen verified already-clean, ana-docker docker log-cap (logrotate copytruncate, no bounce), and the granite→gen memory_extractor bind live on demo+personal.
77 lines
4.2 KiB
Markdown
77 lines
4.2 KiB
Markdown
# Canonical sampler defaults — PFI/VastBlue LiteLLM gateway seats
|
|
|
|
**Applied:** 2026-07-08 · **Gateway:** `ana-docker:4000` · **Config:** `stacks/litellm/conf/config.yaml` → `/opt/docker/conf/litellm/config.yaml`
|
|
|
|
Canonical high-quality sampler defaults for the four model seats, **derived by
|
|
dvalin-smithy-dev** (full rationale + sources: `dvalin-smithy/hoard-drafts/pfi-gateway-sampler-defaults-20260708.md`),
|
|
**triaged + A/B-validated by infra-ops**, and wired into the gateway. These are the
|
|
gateway *defaults*; callers may override per request.
|
|
|
|
Optimized for **output / prose quality** (not throughput or determinism).
|
|
|
|
## Engine surfaces
|
|
|
|
- **gen / gen-reasoning** — vLLM 0.24 (OpenAI sampler surface). No native DRY/XTC → anti-repetition via `presence_penalty`. Thinking split via `chat_template_kwargs.enable_thinking` on distinct `--served-model-name`s (avoids the shared-config-mutation footgun).
|
|
- **char-rp / char-rp-reasoning** — llama.cpp / llama-server (supports `min_p`, `top_k`, DRY, XTC, dynatemp). `min_p` + `top_p` do the tail work; `top_k 0` disables top-k.
|
|
|
|
## The four seats (applied values)
|
|
|
|
### 1. gen — Qwen3.6-35B-A3B heretic (vLLM, non-thinking)
|
|
Also governs **summarizer-large** (shares the same `qwen3.6-27b-aeon` @ :8015 deployment → kept identical).
|
|
|
|
| param | value |
|
|
|---|---|
|
|
| temperature | 0.7 |
|
|
| top_p | 0.80 |
|
|
| top_k | 20 |
|
|
| presence_penalty | **1.5** |
|
|
| repetition/frequency | 1.0 / 0.0 |
|
|
| enable_thinking | false |
|
|
|
|
*Source:* Qwen3.6 README instruct/non-thinking rec. *Change:* presence_penalty 1.0 → 1.5.
|
|
|
|
### 2. gen-reasoning — same model (vLLM, thinking)
|
|
|
|
| param | value |
|
|
|---|---|
|
|
| temperature | **1.0** |
|
|
| top_p | 0.95 |
|
|
| top_k | 20 |
|
|
| presence_penalty | **1.5** |
|
|
| repetition/frequency | 1.0 / 0.0 |
|
|
| enable_thinking | true |
|
|
|
|
*Source:* Qwen3.6 README **general** thinking profile (NOT the temp-0.6 coding sub-profile — the prior default was that coding profile by mistake). *Changes:* temperature 0.6 → 1.0, presence_penalty 1.0 → 1.5. Reasoning is verbose (~9k chars) → callers set generous `max_tokens` (catalog default 32768). Optional per-route coding override: temp 0.6 / presence 0.0.
|
|
|
|
### 3. char-rp — Magidonia-24B-v4.3 (llama.cpp, non-thinking prose RP)
|
|
|
|
| param | value |
|
|
|---|---|
|
|
| temperature | **1.1** |
|
|
| top_p | 0.95 |
|
|
| min_p | **0.10** |
|
|
| top_k | 0 (disabled) |
|
|
| repetition/DRY/XTC | **off** |
|
|
|
|
*Source:* dvalin canonical (Mistral-Small RP prose) **A/B-validated by infra-ops** on the live serve. *Changes:* temp 1.0 → 1.1, min_p 0.03 → 0.10. **min_p 0.10 richened imagery vs 0.03** with no incoherence at temp 1.1. **repeat_penalty 1.05 was REJECTED** — in the A/B it injected a stray markdown title into a grief scene; rep-style penalties hurt Drummer/Magistral RP creativity (matches the model card and dvalin's own note). Alt prose model: `MS3.2-PaintedFantasy-v4.1-24B` (swap via the `char-rp-gguf` stack `.env`).
|
|
|
|
### 4. char-rp-reasoning — Qwen3.5-27B-Deckard-PKD (llama.cpp, managed-reasoning RP)
|
|
|
|
| param | value (request-level) |
|
|
|---|---|
|
|
| temperature | 1.0 |
|
|
| top_p | 0.95 |
|
|
| top_k | 40 |
|
|
| min_p | **0.05** |
|
|
| presence/repetition | **off** |
|
|
| DRY | **0.8 server-side** (base 1.75 / len 2, dry-after-temp) — not a request param |
|
|
| reasoning-budget | 400 (server-side) |
|
|
|
|
*Source:* dvalin-CONFIRMED canonical 2026-07-08 (thread 01KX1Y7P). **Corrected 2026-07-09:** this seat had lagged on QwQ-RpR-v4 — the A/B on 2026-07-08 replaced it with **Deckard-PKD-Heretic i1-Q5_K_M** (DavidAU, Qwen3.5-27B, :8018); the live gateway was always Deckard. Deckard won on brokkr's frozen scorer (0/30 loops, 0/30 refusals) over RpR-v4 (1/30 loop, forbids DRY) + Pantheon-27B (7/30 refusals). Reasoning ON server-side (`--reasoning on`, budget 400); CoT surfaces in `reasoning_content`, clean prose in `content`. Tuning ladder: flat prose→min_p 0.08, loops→DRY 0.9, over-damped→DRY 0.6/off. **Do NOT import RpR/QwQ sampler rules** (different family — QwQ hated DRY; Qwen3.5 benefits from it).
|
|
|
|
## Changing a default
|
|
|
|
Edit the seat's `litellm_params` in `stacks/litellm/conf/config.yaml`, `scp` to
|
|
`/opt/docker/conf/litellm/config.yaml` on ana-docker, `docker restart litellm`.
|
|
(`gen` and `summarizer-large` must change together — same deployment.)
|