diff --git a/docs/pfi/model-sampler-defaults.md b/docs/pfi/model-sampler-defaults.md new file mode 100644 index 0000000..250b605 --- /dev/null +++ b/docs/pfi/model-sampler-defaults.md @@ -0,0 +1,74 @@ +# Canonical sampler defaults — PFI/VastBlue LiteLLM gateway seats + +**Applied:** 2026-07-08 · **Gateway:** `ana-docker:4000` · **Config:** `stacks/litellm/conf/config.yaml` → `/opt/docker/conf/litellm/config.yaml` + +Canonical high-quality sampler defaults for the four model seats, **derived by +dvalin-smithy-dev** (full rationale + sources: `dvalin-smithy/hoard-drafts/pfi-gateway-sampler-defaults-20260708.md`), +**triaged + A/B-validated by infra-ops**, and wired into the gateway. These are the +gateway *defaults*; callers may override per request. + +Optimized for **output / prose quality** (not throughput or determinism). + +## Engine surfaces + +- **gen / gen-reasoning** — vLLM 0.24 (OpenAI sampler surface). No native DRY/XTC → anti-repetition via `presence_penalty`. Thinking split via `chat_template_kwargs.enable_thinking` on distinct `--served-model-name`s (avoids the shared-config-mutation footgun). +- **char-rp / char-rp-reasoning** — llama.cpp / llama-server (supports `min_p`, `top_k`, DRY, XTC, dynatemp). `min_p` + `top_p` do the tail work; `top_k 0` disables top-k. + +## The four seats (applied values) + +### 1. gen — Qwen3.6-35B-A3B heretic (vLLM, non-thinking) +Also governs **summarizer-large** (shares the same `qwen3.6-27b-aeon` @ :8015 deployment → kept identical). + +| param | value | +|---|---| +| temperature | 0.7 | +| top_p | 0.80 | +| top_k | 20 | +| presence_penalty | **1.5** | +| repetition/frequency | 1.0 / 0.0 | +| enable_thinking | false | + +*Source:* Qwen3.6 README instruct/non-thinking rec. *Change:* presence_penalty 1.0 → 1.5. + +### 2. gen-reasoning — same model (vLLM, thinking) + +| param | value | +|---|---| +| temperature | **1.0** | +| top_p | 0.95 | +| top_k | 20 | +| presence_penalty | **1.5** | +| repetition/frequency | 1.0 / 0.0 | +| enable_thinking | true | + +*Source:* Qwen3.6 README **general** thinking profile (NOT the temp-0.6 coding sub-profile — the prior default was that coding profile by mistake). *Changes:* temperature 0.6 → 1.0, presence_penalty 1.0 → 1.5. Reasoning is verbose (~9k chars) → callers set generous `max_tokens` (catalog default 32768). Optional per-route coding override: temp 0.6 / presence 0.0. + +### 3. char-rp — Magidonia-24B-v4.3 (llama.cpp, non-thinking prose RP) + +| param | value | +|---|---| +| temperature | **1.1** | +| top_p | 0.95 | +| min_p | **0.10** | +| top_k | 0 (disabled) | +| repetition/DRY/XTC | **off** | + +*Source:* dvalin canonical (Mistral-Small RP prose) **A/B-validated by infra-ops** on the live serve. *Changes:* temp 1.0 → 1.1, min_p 0.03 → 0.10. **min_p 0.10 richened imagery vs 0.03** with no incoherence at temp 1.1. **repeat_penalty 1.05 was REJECTED** — in the A/B it injected a stray markdown title into a grief scene; rep-style penalties hurt Drummer/Magistral RP creativity (matches the model card and dvalin's own note). Alt prose model: `MS3.2-PaintedFantasy-v4.1-24B` (swap via the `char-rp-gguf` stack `.env`). + +### 4. char-rp-reasoning — QwQ-32B-ArliAI-RpR-v4 (llama.cpp, reasoning RP) + +| param | value | +|---|---| +| temperature | 1.0 | +| top_p | **0.95** | +| top_k | 40 | +| min_p | 0.02 | +| repetition/DRY/XTC | **off** | + +*Source:* ArliAI RpR v4 card — explicit **NO** rep/DRY/XTC penalties. *Change:* added explicit top_p 0.95. Reasoning is server-side (`--reasoning on`, budget-capped); the CoT surfaces in `reasoning_content` with clean prose in `content`. SillyTavern wiring (non-sampler): `include names = never`; exact ``/`` tokens. + +## Changing a default + +Edit the seat's `litellm_params` in `stacks/litellm/conf/config.yaml`, `scp` to +`/opt/docker/conf/litellm/config.yaml` on ana-docker, `docker restart litellm`. +(`gen` and `summarizer-large` must change together — same deployment.) diff --git a/persistent-memory.md b/persistent-memory.md index 7719bd8..48b55c2 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -144,12 +144,14 @@ _As of 2026-07-08 — OFF-THE-SHELF INFERENCE STACK is the active work (home-tra → a single conversation gets the FULL n_ctx (4 slots share the pool, not a /4 split). 128K char-rp only by dropping the reasoning seat's ctx (shared GPU0 budget). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE. -- **⏳ PENDING — wire LiteLLM gateway sampling defaults for GEN** (operator: "wire those settings in"). Config: - `ana-docker:/opt/docker/conf/litellm/config.yaml`. **gen (Qwen3.6)** — thinking - `temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 / presence_penalty 1.5 / rep 1.0`; non-thinking - `temp 0.7 / top_p 0.8 / top_k 20 / pp 1.5` (NO rep-penalty in thinking mode). NOT yet applied to gen/gen-reasoning. - **char-rp / char-rp-reasoning samplers ARE now wired** (Magistral: temp 1.0 / top_p 0.95 / min_p 0.03; QwQ RpR: - temp 1.0 / top_k 40 / min_p 0.02, no rep/DRY — 2026-07-08). +- **✅ LiteLLM gateway sampling defaults WIRED for all 4 seats (2026-07-08)** — dvalin-smithy canonical, infra-ops + triaged + char-rp A/B-validated. **gen** (+summarizer-large twin, shared :8015 deployment): temp 0.7 / top_p 0.8 / + top_k 20 / presence_penalty **1.5** (non-thinking). **gen-reasoning**: temp **1.0** / top_p 0.95 / top_k 20 / + presence_penalty **1.5** (Qwen general-thinking profile — the old temp 0.6 was the CODING sub-profile by mistake; + verbose ~9k-char CoT → callers need big max_tokens). **char-rp**: temp **1.1** / top_p 0.95 / min_p **0.10** / + top_k 0, NO rep (A/B: min_p 0.10 richened imagery vs 0.03; rep 1.05 REJECTED — injected a markdown title in a grief + scene, matches Drummer-no-rep). **char-rp-reasoning**: temp 1.0 / top_p **0.95** / top_k 40 / min_p 0.02, no rep/DRY. + Canonical doc `docs/pfi/model-sampler-defaults.md`; dvalin derivation `dvalin-smithy/hoard-drafts/pfi-gateway-sampler-defaults-20260708.md`. - **NVFP4/GGUF quant tooling (proven, on ana-ml2):** `/tank/aimodels/quant-work/` — `nvfp4_quant.py` (llm-compressor, MoE-aware `load_quantizable_moe`, Mistral-tuned ignore) + `.venv`; `qwen35_vision_nvfp4.py` diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index f1fbfc5..69f71de 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -62,7 +62,7 @@ model_list: model: hosted_vllm/qwen3.6-27b-aeon api_base: http://10.250.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY - presence_penalty: 1.0 + presence_penalty: 1.5 temperature: 0.7 top_p: 0.8 extra_body: @@ -115,15 +115,16 @@ model_list: # RETIRED with the displacement (→ 404, callers migrate to gen): qwen3.5-122-a10b # [-reasoning] + qwen-large[-reasoning] — they named a 122B that no longer exists; # aliasing a 27B under those is the naming footgun the qwen36-vl stack warns against. - # presence_penalty: 1.0 INHERITED from qwopus (same-family abliterated/NVFP4 anti- - # repetition damper, operator 2026-06-27) — RE-VALIDATE for AEON; NOT yet confirmed - # for this model's repetition behavior. --- + # presence_penalty: 1.5 — Qwen3.6 README anti-repetition rec for BOTH non-thinking and + # thinking (dvalin-smithy canonical 2026-07-08, validated vs Qwen guidance). gen + # non-thinking temp 0.7/top_p 0.8; gen-reasoning thinking temp 1.0/top_p 0.95 (the + # GENERAL thinking profile, not the 0.6 coding sub-profile). docs/pfi/model-sampler-defaults.md. --- - model_name: gen litellm_params: model: hosted_vllm/qwen3.6-27b-aeon api_base: http://10.250.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY - presence_penalty: 1.0 + presence_penalty: 1.5 temperature: 0.7 top_p: 0.8 extra_body: @@ -139,8 +140,8 @@ model_list: model: hosted_vllm/qwen3.6-27b-aeon-thinking api_base: http://10.250.50.54:8015/v1 api_key: os.environ/VLLM_API_KEY - presence_penalty: 1.0 - temperature: 0.6 + presence_penalty: 1.5 + temperature: 1.0 top_p: 0.95 extra_body: top_k: 20 @@ -152,19 +153,23 @@ model_list: # ana-ml2 GPU 0). TheDrummer Magidonia-24B-v4.3 Q6_K — Magistral (Mistral) RP tune. # NON-thinking: elite literary prose, zero refusal on dark/explicit scenes, ~65 tok/s, # tight POV/instruction adherence (live-tested 2026-07-08). Replaced the broken Angel - # NVFP4 serve AND the earlier AEON-rp MTP twin. Magistral is stable WITHOUT a repetition - # penalty (dropped the old 1.05); min_p 0.03 is the anti-slop knob. No enable_thinking - # kwarg — meaningless to the Mistral template. Alt prose model (swap via the stack .env): - # MS3.2-PaintedFantasy-v4.1-24B. Callers may override the sampler. + # NVFP4 serve AND the earlier AEON-rp MTP twin. Sampler A/B-tuned 2026-07-08 vs the + # dvalin-smithy canonical: temp 1.1 / top_p 0.95 / min_p 0.10 / top_k 0, NO repetition + # penalty. min_p 0.10 richened imagery vs 0.03; rep 1.05 REJECTED (injected a stray + # markdown title in a grief scene — rep-style penalties hurt Drummer RP, matching the + # card). No enable_thinking kwarg — meaningless to the Mistral template. Alt prose model + # (swap via the stack .env): MS3.2-PaintedFantasy-v4.1-24B. Callers may override. + # docs/pfi/model-sampler-defaults.md. - model_name: char-rp litellm_params: model: hosted_vllm/magidonia-24b-v4.3 api_base: http://10.250.50.54:8016/v1 api_key: os.environ/VLLM_API_KEY - temperature: 1.0 + temperature: 1.1 top_p: 0.95 extra_body: - min_p: 0.03 + min_p: 0.10 + top_k: 0 model_info: mode: chat # char-rp-reasoning -> GGUF managed-REASONING seat (:8018, llama.cpp, char-rp-gguf stack). @@ -181,6 +186,7 @@ model_list: api_base: http://10.250.50.54:8018/v1 api_key: os.environ/VLLM_API_KEY temperature: 1.0 + top_p: 0.95 extra_body: top_k: 40 min_p: 0.02