36 lines
1.3 KiB
Diff
36 lines
1.3 KiB
Diff
Subject: [PATCH 0004] chat: LLM_DEFAULT_CONTEXT_WINDOW for model names the OpenAI table does not know
|
|
|
|
OpenAIService.GetContextWindow knows only OpenAI's own names (gpt-4o, gpt-4, gpt-3.5)
|
|
and returns 4096 for anything else. Behind an OpenAI-compatible gateway every model is
|
|
"anything else" (here LiteLLM's summarizer and gen-large, 262k-context vLLM seats), so
|
|
chat refuses any transcript the handler estimates above 3,596 tokens (chars/4, minus 500
|
|
kept for the reply): a 22k-character meeting was refused on 2026-10-19 with "limit 4096".
|
|
The default branch now reads LLM_DEFAULT_CONTEXT_WINDOW (a positive integer) and keeps
|
|
4096 when it is unset or invalid; the known OpenAI names are unchanged.
|
|
|
|
---
|
|
--- a/internal/llm/openai.go
|
|
+++ b/internal/llm/openai.go
|
|
@@ -9,6 +9,8 @@
|
|
"io"
|
|
"log"
|
|
"net/http"
|
|
+ "os"
|
|
+ "strconv"
|
|
"strings"
|
|
"time"
|
|
)
|
|
@@ -337,7 +339,11 @@
|
|
case strings.HasPrefix(model, "gpt-3.5-turbo"):
|
|
return 16385, nil // Most recent gpt-3.5-turbo is 16k
|
|
default:
|
|
- // Default fallback
|
|
+ // Not an OpenAI model name (a LiteLLM alias, a vLLM served name): the operator's
|
|
+ // LLM_DEFAULT_CONTEXT_WINDOW when set to a positive integer, else upstream's 4096.
|
|
+ if v, err := strconv.Atoi(strings.TrimSpace(os.Getenv("LLM_DEFAULT_CONTEXT_WINDOW"))); err == nil && v > 0 {
|
|
+ return v, nil
|
|
+ }
|
|
return 4096, nil
|
|
}
|
|
}
|