docs(pfi): sync recommended-model-settings KB to deployed gateway defaults
Add §9 "PFI LiteLLM Gateway — Deployed Sampling Defaults": the live fleet sampling table (granite/qwen/judges/GLM) with provenance, overrideable-default semantics, the GLM API-accepted-subset caveat, and the research-confirmed temp-0 rationale for granite + image-judge. Accepts the dvalin-smithy-dev recommendations as deployed. §§1-8 vendor reference left intact.
This commit is contained in:
@@ -16,6 +16,7 @@
|
||||
6. [Quick Reference Cards](#6-quick-reference-cards)
|
||||
7. [Critical Warnings by Model](#7-critical-warnings-by-model)
|
||||
8. [Models Without KB Settings](#8-models-without-kb-settings)
|
||||
9. [PFI LiteLLM Gateway — Deployed Sampling Defaults](#9-pfi-litellm-gateway--deployed-sampling-defaults)
|
||||
|
||||
---
|
||||
|
||||
@@ -539,6 +540,51 @@ The following model families are deployed in the Infrastructure-PFI environment
|
||||
|
||||
---
|
||||
|
||||
## 9. PFI LiteLLM Gateway — Deployed Sampling Defaults
|
||||
|
||||
> **Live as of 2026-06-27** on the PFI gateway (`ana-docker:4000`; canonical config
|
||||
> `eshpfi-management/stacks/litellm/conf/config.yaml`). Unlike §§1–8 (general vendor
|
||||
> reference), this section is the **deployed reality** — keep it in sync when gateway
|
||||
> sampling changes.
|
||||
|
||||
These are **overrideable defaults**: any caller that passes its own sampling param
|
||||
wins; callers that omit one inherit the value below. (Verified — vLLM rejected an
|
||||
out-of-range `presence_penalty=5.0`, proving per-request values reach the backend and
|
||||
override the config default.) Values set per the `dvalin-smithy-dev` research pass
|
||||
(provenance-cited in-thread, corroborated by §3 above). vLLM-only params (`top_k`,
|
||||
`repetition_penalty`) ride in `extra_body` so LiteLLM's `drop_params` can't strip them.
|
||||
|
||||
| Gateway model(s) | temp | top_p | top_k | presence_penalty | repetition_penalty | Source |
|
||||
|---|---|---|---|---|---|---|
|
||||
| `granite-4.1-8b`, `summarizer`, `classifier` | **0** | — | — | — | — | IBM-canonical (temp 0 for inferencing) |
|
||||
| `gen`, `summarizer-large`, `qwen-large`, `qwen3.5-122-a10b` (non-thinking) | **0.7** | 0.8 | 20 | **1.0** | — | Qwen3 non-thinking + operator anti-repetition |
|
||||
| `gen-reasoning`, `qwen-large-reasoning`, `qwen3.5-122-a10b-reasoning` (thinking) | **0.6** | 0.95 | 20 | **1.0** | — | Qwen3 thinking |
|
||||
| `qwen-image-bench`, `image-judge` | **0** | 1.0 | 1 | — | 1.05 | Qwen-Image-Bench judge reproducibility table |
|
||||
| `selene-1-mini-8b`, `chat-judge` | **0.6** | 0.9 | — | — | — | Selene `generation_config` |
|
||||
| `glm-5.1`, `glm-5.2`, `glm-5-turbo`, `glm-4.7`, `gen-frontier` | **1.0** | 0.95 | — | — | — | z.ai API defaults (5.x / 4.7 series) |
|
||||
| `glm-4.5-air` | **0.6** | 0.95 | — | — | — | z.ai API default (4.5 series) |
|
||||
| `qwen3-embedding`, `qwen3-reranker`, `reranker` | — | — | — | — | — | no sampling (embedding / rerank) |
|
||||
|
||||
**Notes:**
|
||||
- **qwen "gen" family `presence_penalty: 1.0`** — operator-set anti-repetition for the
|
||||
abliterated/NVFP4 Qwopus 122B-A10B. Qwen documents `presence_penalty` (0–2) as *the*
|
||||
repetition lever; 1.0 is conservative (the §3 vendor general value is 1.5 — step up to
|
||||
1.5 if loops persist). Do **not** use `repetition_penalty` for the Qwen3 family.
|
||||
- **GLM (z.ai cloud) — only `temperature` + `top_p` are set.** z.ai's chat API schema
|
||||
accepts no `top_k` / `min_p` / penalties, so they're deliberately not sent (would be
|
||||
silently dropped). These temps match z.ai's own API defaults (explicit-over-implicit /
|
||||
future-proofing).
|
||||
- **Both `temp 0` values (granite, image-judge) are research-confirmed, not heuristic.**
|
||||
Greedy is correct for constrained summ/classify (IBM) and for judge reproducibility
|
||||
(Qwen judge card + LLM-as-judge practice). `temp 0.1` was explicitly evaluated and
|
||||
rejected: it adds sampling noise without fixing loops, and *reduces* run-to-run score
|
||||
consistency on the judge. If granite ever loops in production, fix via
|
||||
`repetition_penalty` / `presence_penalty` / `max_tokens`, not a temperature floor.
|
||||
- **`qwen-image-bench` / `image-judge` is arbo's hero-judge** (comfy-dev consumer) —
|
||||
sampling changes there are a coordination item, not a unilateral gateway edit.
|
||||
|
||||
---
|
||||
|
||||
## KB Source Documents
|
||||
|
||||
| Document | Path in KB |
|
||||
|
||||
Reference in New Issue
Block a user