chore(vllm): retire LFM2.5-2.6B permanently; audit finds nevermore on the harmful reranker
Operator directive: lfm2.5-2.6b goes down permanently.
- stacks/vllm/compose.yaml vllm-lfm25 service removed (replaced by a
tombstone comment), pushed live to ana-ml2
- ana-ml2 container docker rm -f'd, 8,721 MiB freed on GPU1
(95,388 -> 86,667 of 97,887)
- litellm config lfm2.5-2.6b alias deleted, live + canonical,
28 -> 27 models
It was an EVAL-ONLY bake-off seat against granite-4.1-8b that never received
the operator ruling it was pending; the comparator was retired from the roster
on 2026-08-15; it was deliberately never wired into any default or fallback
routing chain; and spend logs show 0 calls in the 4-day window to 2026-08-21.
Weights stay in the shared HF cache -- nothing deleted from disk.
The gateway restart that makes the alias deletion take effect is HELD so it can
batch with a pending reranker change. Until then the name is still routable
in-memory and will error against a dead backend.
Auditing the three reranker seats while answering "why do we have three" turned
up a real problem. The design is one production, one rollback, one fallback --
but the traffic is backwards:
:8013 A3 bge-v2-m3 PRODUCTION, backs `reranker` 0 calls / 4 days
:8002 Qwen3-Reranker RETIRED incumbent, rollback only 7 calls, 12-hourly
:8014 A4 gte-modernbert "fallback" no alias at all
nevermore is hard-wired to the incumbent by name (NEVERMORE_RERANK_MODEL=
qwen3-reranker), so the R43 cutover never moved it -- the cutover repointed the
`reranker` alias and correctly left `qwen3-reranker` naming the Qwen model.
Brokkr R43 measured that model harming 80/90 fleet queries, so nevermore's
twice-daily rerank pass is likely degrading its own briefing.
Fix is one line in nevermore's .env plus a nevermore restart, and it must land
before :8002 is retired. Recorded in persistent-memory with the A4 alias also
noted as absent (global CLAUDE.md names reranker-a4-gte-modernbert; it does not
exist).
This commit is contained in:
@@ -513,22 +513,12 @@ model_list:
|
||||
# model names. Removed so unknown models now fail loudly (404). Re-add an
|
||||
# explicit per-model entry if a swappable zoo ever returns. ---
|
||||
|
||||
# --- lfm2.5-2.6b -> LiquidAI LFM2.5-2.6B (ana-ml2 GPU1 :8021, vLLM). NON-PROD bake-off
|
||||
# vs granite-4.1-8b (brokkr R-target 2026-08-10). LFM Open License v1.0 (<USD 10M-rev
|
||||
# commercial) - EVAL-ONLY pending operator ruling; NOT in any default/fallback chain.
|
||||
# Reasoning model served raw (no vLLM reasoning-parser) so content is non-empty.
|
||||
# Vendor sampling (temp 0.1 / top_k 50 / rep_pen 1.1) baked as the alias default. ---
|
||||
- model_name: lfm2.5-2.6b
|
||||
litellm_params:
|
||||
model: hosted_vllm/lfm2.5-2.6b
|
||||
api_base: http://10.250.50.54:8021/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
temperature: 0.1
|
||||
extra_body:
|
||||
top_k: 50
|
||||
repetition_penalty: 1.1
|
||||
model_info:
|
||||
mode: chat
|
||||
# --- lfm2.5-2.6b -> RETIRED PERMANENTLY 2026-08-20 (operator directive). The
|
||||
# LiquidAI LFM2.5-2.6B seat (ana-ml2 GPU1 :8021) was an EVAL-ONLY bake-off
|
||||
# against granite-4.1-8b that never got its operator ruling; its comparator
|
||||
# was retired 2026-08-15 and spend logs showed 0 calls in the 4 days to
|
||||
# 2026-08-21. Container removed, service deleted from stacks/vllm. The alias
|
||||
# is deleted rather than repointed so the name 404s cleanly. ---
|
||||
|
||||
general_settings:
|
||||
master_key: os.environ/LITELLM_MASTER_KEY
|
||||
|
||||
@@ -342,78 +342,15 @@ services:
|
||||
- homepage.description=Qwen2.5-Coder-1.5B FIM code-completion (ana-ml2, Zed edit-predictions)
|
||||
- homepage.href=http://10.250.50.54:${CODER_PORT}/docs
|
||||
|
||||
vllm-lfm25:
|
||||
image: vllm/vllm-openai:${VLLM_VERSION}
|
||||
container_name: vllm-lfm25
|
||||
restart: unless-stopped
|
||||
ipc: host
|
||||
ports:
|
||||
- "${LFM25_PORT}:8000"
|
||||
volumes:
|
||||
- /tank/aimodels/huggingface:/hfcache
|
||||
environment:
|
||||
- HF_HOME=/hfcache
|
||||
- HF_HUB_CACHE=/hfcache/hub
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||||
- VLLM_API_KEY=${API_KEY:-}
|
||||
command:
|
||||
# LiquidAI/LFM2.5-2.6B (BF16, Lfm2ForCausalLM). NON-PRODUCTION bake-off alias
|
||||
# vs granite-4.1-8b on structured extraction / classification / tool-call
|
||||
# formatting (brokkr R-target, 2026-08-10). LFM Open License v1.0 (<USD 10M-rev
|
||||
# commercial, not OSI) — EVAL-ONLY pending an operator ruling on production; NOT
|
||||
# wired into any default/fallback routing chain. Reasoning model: the chat
|
||||
# template opens <think>; deliberately NO --reasoning-parser, so the full
|
||||
# generation (thinking + answer) lands in `content` non-empty — brokkr's explicit
|
||||
# requirement (an empty content with the answer stranded in reasoning_content
|
||||
# reads as a degenerate model). Vendor sampling (temp 0.1 / top_k 50 / rep_pen 1.1)
|
||||
# is the LiteLLM alias default, not a launch arg.
|
||||
- ${LFM25_MODEL}
|
||||
- --served-model-name
|
||||
- ${LFM25_SERVED_NAME}
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- "8000"
|
||||
- --gpu-memory-utilization
|
||||
- ${LFM25_GPU_MEM_UTIL}
|
||||
- --max-model-len
|
||||
- ${LFM25_MAX_MODEL_LEN}
|
||||
- --max-num-seqs
|
||||
- ${LFM25_MAX_NUM_SEQS}
|
||||
- --dtype
|
||||
- auto
|
||||
- --kv-cache-dtype
|
||||
- ${LFM25_KV_CACHE_DTYPE}
|
||||
- --enable-prefix-caching
|
||||
# Split the <think> reasoning (delimited by </think>; the opening tag is
|
||||
# injected by the chat template into the prompt) into reasoning_content, so
|
||||
# `content` is the clean post-</think> answer — scoreable JSON for brokkr's
|
||||
# structured-extraction bake-off (raw-served, reasoning prepended, is not).
|
||||
- --reasoning-parser
|
||||
- deepseek_r1
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids:
|
||||
- "${LFM25_GPU_ID}"
|
||||
capabilities:
|
||||
- gpu
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 300s
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI - Inference
|
||||
- homepage.name=vLLM LFM2.5-2.6B (bake-off)
|
||||
- homepage.icon=mdi-flask-outline
|
||||
- homepage.description=LiquidAI LFM2.5-2.6B non-prod bake-off vs granite (ana-ml2)
|
||||
- homepage.href=http://10.250.50.54:${LFM25_PORT}/docs
|
||||
# vllm-lfm25 (LiquidAI/LFM2.5-2.6B, :8021) — RETIRED PERMANENTLY 2026-08-20 by
|
||||
# operator directive. It was an EVAL-ONLY bake-off seat against granite-4.1-8b
|
||||
# (brokkr R-target, 2026-08-10) that never received the operator ruling it was
|
||||
# pending. Its comparator is gone (granite retired from the roster 2026-08-15),
|
||||
# it was deliberately never wired into any default/fallback routing chain, and
|
||||
# LiteLLM spend logs showed 0 calls in the 4-day window ending 2026-08-21.
|
||||
# Freed 8,772 MiB on ana-ml2 GPU1. The `lfm2.5-2.6b` gateway alias was removed
|
||||
# in the same pass so the name 404s cleanly rather than erroring against a dead
|
||||
# backend. Weights remain in the shared HF cache; nothing was deleted from disk.
|
||||
|
||||
networks:
|
||||
tnet:
|
||||
|
||||
Reference in New Issue
Block a user