diff --git a/docs/pfi/recommended-model-settings.md b/docs/pfi/recommended-model-settings.md index a42a35c..280f643 100644 --- a/docs/pfi/recommended-model-settings.md +++ b/docs/pfi/recommended-model-settings.md @@ -16,6 +16,7 @@ 6. [Quick Reference Cards](#6-quick-reference-cards) 7. [Critical Warnings by Model](#7-critical-warnings-by-model) 8. [Models Without KB Settings](#8-models-without-kb-settings) +9. [PFI LiteLLM Gateway — Deployed Sampling Defaults](#9-pfi-litellm-gateway--deployed-sampling-defaults) --- @@ -539,6 +540,51 @@ The following model families are deployed in the Infrastructure-PFI environment --- +## 9. PFI LiteLLM Gateway — Deployed Sampling Defaults + +> **Live as of 2026-06-27** on the PFI gateway (`ana-docker:4000`; canonical config +> `eshpfi-management/stacks/litellm/conf/config.yaml`). Unlike §§1–8 (general vendor +> reference), this section is the **deployed reality** — keep it in sync when gateway +> sampling changes. + +These are **overrideable defaults**: any caller that passes its own sampling param +wins; callers that omit one inherit the value below. (Verified — vLLM rejected an +out-of-range `presence_penalty=5.0`, proving per-request values reach the backend and +override the config default.) Values set per the `dvalin-smithy-dev` research pass +(provenance-cited in-thread, corroborated by §3 above). vLLM-only params (`top_k`, +`repetition_penalty`) ride in `extra_body` so LiteLLM's `drop_params` can't strip them. + +| Gateway model(s) | temp | top_p | top_k | presence_penalty | repetition_penalty | Source | +|---|---|---|---|---|---|---| +| `granite-4.1-8b`, `summarizer`, `classifier` | **0** | — | — | — | — | IBM-canonical (temp 0 for inferencing) | +| `gen`, `summarizer-large`, `qwen-large`, `qwen3.5-122-a10b` (non-thinking) | **0.7** | 0.8 | 20 | **1.0** | — | Qwen3 non-thinking + operator anti-repetition | +| `gen-reasoning`, `qwen-large-reasoning`, `qwen3.5-122-a10b-reasoning` (thinking) | **0.6** | 0.95 | 20 | **1.0** | — | Qwen3 thinking | +| `qwen-image-bench`, `image-judge` | **0** | 1.0 | 1 | — | 1.05 | Qwen-Image-Bench judge reproducibility table | +| `selene-1-mini-8b`, `chat-judge` | **0.6** | 0.9 | — | — | — | Selene `generation_config` | +| `glm-5.1`, `glm-5.2`, `glm-5-turbo`, `glm-4.7`, `gen-frontier` | **1.0** | 0.95 | — | — | — | z.ai API defaults (5.x / 4.7 series) | +| `glm-4.5-air` | **0.6** | 0.95 | — | — | — | z.ai API default (4.5 series) | +| `qwen3-embedding`, `qwen3-reranker`, `reranker` | — | — | — | — | — | no sampling (embedding / rerank) | + +**Notes:** +- **qwen "gen" family `presence_penalty: 1.0`** — operator-set anti-repetition for the + abliterated/NVFP4 Qwopus 122B-A10B. Qwen documents `presence_penalty` (0–2) as *the* + repetition lever; 1.0 is conservative (the §3 vendor general value is 1.5 — step up to + 1.5 if loops persist). Do **not** use `repetition_penalty` for the Qwen3 family. +- **GLM (z.ai cloud) — only `temperature` + `top_p` are set.** z.ai's chat API schema + accepts no `top_k` / `min_p` / penalties, so they're deliberately not sent (would be + silently dropped). These temps match z.ai's own API defaults (explicit-over-implicit / + future-proofing). +- **Both `temp 0` values (granite, image-judge) are research-confirmed, not heuristic.** + Greedy is correct for constrained summ/classify (IBM) and for judge reproducibility + (Qwen judge card + LLM-as-judge practice). `temp 0.1` was explicitly evaluated and + rejected: it adds sampling noise without fixing loops, and *reduces* run-to-run score + consistency on the judge. If granite ever loops in production, fix via + `repetition_penalty` / `presence_penalty` / `max_tokens`, not a temperature floor. +- **`qwen-image-bench` / `image-judge` is arbo's hero-judge** (comfy-dev consumer) — + sampling changes there are a coordination item, not a unilateral gateway edit. + +--- + ## KB Source Documents | Document | Path in KB |