docs(pfi): sync recommended-model-settings KB to deployed gateway defaults

Add §9 "PFI LiteLLM Gateway — Deployed Sampling Defaults": the live fleet
sampling table (granite/qwen/judges/GLM) with provenance, overrideable-default
semantics, the GLM API-accepted-subset caveat, and the research-confirmed temp-0
rationale for granite + image-judge. Accepts the dvalin-smithy-dev recommendations
as deployed. §§1-8 vendor reference left intact.
This commit is contained in:
2026-06-27 09:45:10 -07:00
parent b9dcbc199f
commit 809c51e095
+46
View File
@@ -16,6 +16,7 @@
6. [Quick Reference Cards](#6-quick-reference-cards)
7. [Critical Warnings by Model](#7-critical-warnings-by-model)
8. [Models Without KB Settings](#8-models-without-kb-settings)
9. [PFI LiteLLM Gateway — Deployed Sampling Defaults](#9-pfi-litellm-gateway--deployed-sampling-defaults)
---
@@ -539,6 +540,51 @@ The following model families are deployed in the Infrastructure-PFI environment
---
## 9. PFI LiteLLM Gateway — Deployed Sampling Defaults
> **Live as of 2026-06-27** on the PFI gateway (`ana-docker:4000`; canonical config
> `eshpfi-management/stacks/litellm/conf/config.yaml`). Unlike §§18 (general vendor
> reference), this section is the **deployed reality** — keep it in sync when gateway
> sampling changes.
These are **overrideable defaults**: any caller that passes its own sampling param
wins; callers that omit one inherit the value below. (Verified — vLLM rejected an
out-of-range `presence_penalty=5.0`, proving per-request values reach the backend and
override the config default.) Values set per the `dvalin-smithy-dev` research pass
(provenance-cited in-thread, corroborated by §3 above). vLLM-only params (`top_k`,
`repetition_penalty`) ride in `extra_body` so LiteLLM's `drop_params` can't strip them.
| Gateway model(s) | temp | top_p | top_k | presence_penalty | repetition_penalty | Source |
|---|---|---|---|---|---|---|
| `granite-4.1-8b`, `summarizer`, `classifier` | **0** | — | — | — | — | IBM-canonical (temp 0 for inferencing) |
| `gen`, `summarizer-large`, `qwen-large`, `qwen3.5-122-a10b` (non-thinking) | **0.7** | 0.8 | 20 | **1.0** | — | Qwen3 non-thinking + operator anti-repetition |
| `gen-reasoning`, `qwen-large-reasoning`, `qwen3.5-122-a10b-reasoning` (thinking) | **0.6** | 0.95 | 20 | **1.0** | — | Qwen3 thinking |
| `qwen-image-bench`, `image-judge` | **0** | 1.0 | 1 | — | 1.05 | Qwen-Image-Bench judge reproducibility table |
| `selene-1-mini-8b`, `chat-judge` | **0.6** | 0.9 | — | — | — | Selene `generation_config` |
| `glm-5.1`, `glm-5.2`, `glm-5-turbo`, `glm-4.7`, `gen-frontier` | **1.0** | 0.95 | — | — | — | z.ai API defaults (5.x / 4.7 series) |
| `glm-4.5-air` | **0.6** | 0.95 | — | — | — | z.ai API default (4.5 series) |
| `qwen3-embedding`, `qwen3-reranker`, `reranker` | — | — | — | — | — | no sampling (embedding / rerank) |
**Notes:**
- **qwen "gen" family `presence_penalty: 1.0`** — operator-set anti-repetition for the
abliterated/NVFP4 Qwopus 122B-A10B. Qwen documents `presence_penalty` (02) as *the*
repetition lever; 1.0 is conservative (the §3 vendor general value is 1.5 — step up to
1.5 if loops persist). Do **not** use `repetition_penalty` for the Qwen3 family.
- **GLM (z.ai cloud) — only `temperature` + `top_p` are set.** z.ai's chat API schema
accepts no `top_k` / `min_p` / penalties, so they're deliberately not sent (would be
silently dropped). These temps match z.ai's own API defaults (explicit-over-implicit /
future-proofing).
- **Both `temp 0` values (granite, image-judge) are research-confirmed, not heuristic.**
Greedy is correct for constrained summ/classify (IBM) and for judge reproducibility
(Qwen judge card + LLM-as-judge practice). `temp 0.1` was explicitly evaluated and
rejected: it adds sampling noise without fixing loops, and *reduces* run-to-run score
consistency on the judge. If granite ever loops in production, fix via
`repetition_penalty` / `presence_penalty` / `max_tokens`, not a temperature floor.
- **`qwen-image-bench` / `image-judge` is arbo's hero-judge** (comfy-dev consumer) —
sampling changes there are a coordination item, not a unilateral gateway edit.
---
## KB Source Documents
| Document | Path in KB |