docs(claude.md): refresh LiteLLM gateway LLM roster to the live gateway
The summarizer/classifier endpoint is now the 'summarizer' (+ 'summarizer-large') / 'classifier' alias backed by qwen3.6-35b-a3b-heretic (2026-07 GPU re-org), not granite-4.1-8b. GLM family up to 5.2 (+ -reasoning/-turbo/-air) with gen-frontier -> GLM 5.2; kimi now kimi-k3 / kimi-k3-gen-api. Added coder-fast, lfm2.5-2.6b, the Selene judge, current rerankers, and ext-tts -> dots. Retired granite-4.1-8b and the qwen3.5-9b-fp8 vision seat from the roster. Shared-key / paid-spend caveats unchanged.
This commit is contained in:
+27
-18
@@ -482,23 +482,32 @@ note says otherwise. On other boxes, check first.)
|
||||
(curated geography), doesn't replace it. Pilot-validated on Worldtree
|
||||
2026-06-10.
|
||||
|
||||
- **LiteLLM gateway (Granite 4.1 + friends)** — OpenAI-compatible gateway at
|
||||
`http://10.250.50.70:4000/v1` (Logs UI `:4000/ui`) fronting vLLM services.
|
||||
Essentially-**free local compute** for low-caliber, high-volume,
|
||||
parallelizable LLM work. The **always-available summarizer / classifier
|
||||
endpoint** is `granite-4.1-8b` — the go-to for summarization, naming/
|
||||
labeling, classification, and triage (fast, parallelizes well on vLLM):
|
||||
served FP8 on ana-ml2 GPU 1, production-stable, 131k context (rebalanced
|
||||
2026-06-13). Treat it as a standing dependency you can reach for any time.
|
||||
Also behind the gateway: `qwen3.5-9b-fp8` (vision / multimodal — image +
|
||||
text chat, GPU 1), `qwen3-embedding`, `qwen3-reranker`, `glm-5.1` / `glm-4.7`
|
||||
(via z.ai passthrough), and `kimi-k3` (Moonshot passthrough, added
|
||||
2026-07-25 for Heid's 4th cross-frontier panel arm — 1M context, a
|
||||
**reasoning** model: final answer in `content`, chain-of-thought in
|
||||
`reasoning_content`, and it 400s on any `temperature` other than 1, so omit
|
||||
the field entirely rather than sending 0. Sibling variants `kimi-k2.6` /
|
||||
`kimi-k2.7-code` / `kimi-k2.7-code-highspeed` are provisioned on the same
|
||||
key on request). The gateway 401s without a **virtual API key**. **Shared
|
||||
- **LiteLLM gateway (local vLLM + GLM/Kimi passthrough)** — OpenAI-compatible gateway at
|
||||
`http://10.250.50.70:4000/v1` (Logs UI `:4000/ui`) fronting ana-ml2 / irv-ml1 vLLM services
|
||||
plus paid passthroughs. Essentially-**free local compute** for low-caliber, high-volume,
|
||||
parallelizable LLM work. The **always-available summarizer / classifier endpoint** is the
|
||||
`summarizer` (+ `summarizer-large`) / `classifier` alias — the go-to for summarization,
|
||||
naming/labeling, classification, and triage. Since the 2026-07 GPU re-org it's backed by
|
||||
**`qwen3.6-35b-a3b-heretic`** (efficient MoE, ~3B active) on ana-ml2 `:8015` — the same "gen"
|
||||
seat that also serves `gen` (general chat), `gen-reasoning` (thinking variant), `image-judge`,
|
||||
and `qwen-image-bench`. (`granite-4.1-8b` and the old `qwen3.5-9b-fp8` vision seat are retired
|
||||
from the roster.) Treat the summarizer as a standing dependency you can reach any time. The
|
||||
roster drifts — query the live list at `:4000/v1/models` (~30 entries as of 2026-08-14); the
|
||||
current tool-relevant set:
|
||||
- **Embeddings / rerankers (local, free):** `qwen3-embedding`, `qwen3-reranker`, `reranker`
|
||||
(BAAI bge-v2-m3), `reranker-a4-gte-modernbert`.
|
||||
- **Small fast local (free):** `coder-fast` (qwen2.5-coder-1.5b), `lfm2.5-2.6b` (Liquid) —
|
||||
cheap for high-volume mechanical work.
|
||||
- **Judge (local, free):** `selene-1-mini-8b` / `chat-judge` (Atla Selene reward/quality judge).
|
||||
- **Frontier passthrough (PAID — spends vendor credits):** `gen-frontier` /
|
||||
`gen-frontier-reasoning` → GLM 5.2; the GLM family `glm-5.2` (+`-reasoning`), `glm-5.1`
|
||||
(+`-reasoning`), `glm-5-turbo`, `glm-4.7`, `glm-4.5-air` (z.ai); and `kimi-k3` /
|
||||
`kimi-k3-gen-api` (Moonshot — 1M-context **reasoning** models: answer in `content`,
|
||||
chain-of-thought in `reasoning_content`, and they 400 on any `temperature` other than 1,
|
||||
so omit the field entirely rather than sending 0).
|
||||
- **TTS:** `ext-tts` → the fleet dots gateway (irv-ml1 `:8198`; voices donut/emmie/glados/
|
||||
miranda/sindra + emotion variants, or OpenAI aliases nova/alloy — repointed off the parked
|
||||
zonos 2026-08-13; an unknown voice 404s and can trip the router cooldown). The gateway 401s without a **virtual API key**. **Shared
|
||||
all-agents key** — `sk-eA_XOdcs6nIkyYXonohtEQ` (alias `all-agents-local`):
|
||||
reaches **every** model behind the gateway, paid passthroughs included
|
||||
(corrected 2026-07-25 — this entry previously read "scoped to free local
|
||||
@@ -511,7 +520,7 @@ note says otherwise. On other boxes, check first.)
|
||||
the always-available free local endpoints — no per-project provisioning
|
||||
needed. Example: `curl http://10.250.50.70:4000/v1/chat/completions -H
|
||||
"Authorization: Bearer sk-eA_XOdcs6nIkyYXonohtEQ" -d
|
||||
'{"model":"granite-4.1-8b","messages":[…]}'`. For spend isolated from the
|
||||
'{"model":"summarizer","messages":[…]}'`. For spend isolated from the
|
||||
shared pool — auditable per-project cost, blast-radius containment — still
|
||||
request a project-scoped key from infra-ops via althing; that remains the
|
||||
right move for anything where "shows up on someone else's tab" matters, it's
|
||||
|
||||
Reference in New Issue
Block a user