chore(vllm): retire LFM2.5-2.6B permanently; audit finds nevermore on the harmful reranker

Operator directive: lfm2.5-2.6b goes down permanently.

  - stacks/vllm/compose.yaml   vllm-lfm25 service removed (replaced by a
                               tombstone comment), pushed live to ana-ml2
  - ana-ml2                    container docker rm -f'd, 8,721 MiB freed on GPU1
                               (95,388 -> 86,667 of 97,887)
  - litellm config             lfm2.5-2.6b alias deleted, live + canonical,
                               28 -> 27 models

It was an EVAL-ONLY bake-off seat against granite-4.1-8b that never received
the operator ruling it was pending; the comparator was retired from the roster
on 2026-08-15; it was deliberately never wired into any default or fallback
routing chain; and spend logs show 0 calls in the 4-day window to 2026-08-21.
Weights stay in the shared HF cache -- nothing deleted from disk.

The gateway restart that makes the alias deletion take effect is HELD so it can
batch with a pending reranker change. Until then the name is still routable
in-memory and will error against a dead backend.

Auditing the three reranker seats while answering "why do we have three" turned
up a real problem. The design is one production, one rollback, one fallback --
but the traffic is backwards:

  :8013 A3 bge-v2-m3      PRODUCTION, backs `reranker`     0 calls / 4 days
  :8002 Qwen3-Reranker    RETIRED incumbent, rollback only 7 calls, 12-hourly
  :8014 A4 gte-modernbert "fallback"                       no alias at all

nevermore is hard-wired to the incumbent by name (NEVERMORE_RERANK_MODEL=
qwen3-reranker), so the R43 cutover never moved it -- the cutover repointed the
`reranker` alias and correctly left `qwen3-reranker` naming the Qwen model.
Brokkr R43 measured that model harming 80/90 fleet queries, so nevermore's
twice-daily rerank pass is likely degrading its own briefing.

Fix is one line in nevermore's .env plus a nevermore restart, and it must land
before :8002 is retired. Recorded in persistent-memory with the A4 alias also
noted as absent (global CLAUDE.md names reranker-a4-gte-modernbert; it does not
exist).
This commit is contained in:
vh
2026-08-20 23:23:22 -07:00
parent e3ce713f7f
commit b990951d80
3 changed files with 26 additions and 90 deletions
+9 -72
View File
@@ -342,78 +342,15 @@ services:
- homepage.description=Qwen2.5-Coder-1.5B FIM code-completion (ana-ml2, Zed edit-predictions)
- homepage.href=http://10.250.50.54:${CODER_PORT}/docs
vllm-lfm25:
image: vllm/vllm-openai:${VLLM_VERSION}
container_name: vllm-lfm25
restart: unless-stopped
ipc: host
ports:
- "${LFM25_PORT}:8000"
volumes:
- /tank/aimodels/huggingface:/hfcache
environment:
- HF_HOME=/hfcache
- HF_HUB_CACHE=/hfcache/hub
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
- VLLM_API_KEY=${API_KEY:-}
command:
# LiquidAI/LFM2.5-2.6B (BF16, Lfm2ForCausalLM). NON-PRODUCTION bake-off alias
# vs granite-4.1-8b on structured extraction / classification / tool-call
# formatting (brokkr R-target, 2026-08-10). LFM Open License v1.0 (<USD 10M-rev
# commercial, not OSI) — EVAL-ONLY pending an operator ruling on production; NOT
# wired into any default/fallback routing chain. Reasoning model: the chat
# template opens <think>; deliberately NO --reasoning-parser, so the full
# generation (thinking + answer) lands in `content` non-empty — brokkr's explicit
# requirement (an empty content with the answer stranded in reasoning_content
# reads as a degenerate model). Vendor sampling (temp 0.1 / top_k 50 / rep_pen 1.1)
# is the LiteLLM alias default, not a launch arg.
- ${LFM25_MODEL}
- --served-model-name
- ${LFM25_SERVED_NAME}
- --host
- 0.0.0.0
- --port
- "8000"
- --gpu-memory-utilization
- ${LFM25_GPU_MEM_UTIL}
- --max-model-len
- ${LFM25_MAX_MODEL_LEN}
- --max-num-seqs
- ${LFM25_MAX_NUM_SEQS}
- --dtype
- auto
- --kv-cache-dtype
- ${LFM25_KV_CACHE_DTYPE}
- --enable-prefix-caching
# Split the <think> reasoning (delimited by </think>; the opening tag is
# injected by the chat template into the prompt) into reasoning_content, so
# `content` is the clean post-</think> answer — scoreable JSON for brokkr's
# structured-extraction bake-off (raw-served, reasoning prepended, is not).
- --reasoning-parser
- deepseek_r1
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids:
- "${LFM25_GPU_ID}"
capabilities:
- gpu
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 300s
networks:
- tnet
labels:
- homepage.group=AI - Inference
- homepage.name=vLLM LFM2.5-2.6B (bake-off)
- homepage.icon=mdi-flask-outline
- homepage.description=LiquidAI LFM2.5-2.6B non-prod bake-off vs granite (ana-ml2)
- homepage.href=http://10.250.50.54:${LFM25_PORT}/docs
# vllm-lfm25 (LiquidAI/LFM2.5-2.6B, :8021) — RETIRED PERMANENTLY 2026-08-20 by
# operator directive. It was an EVAL-ONLY bake-off seat against granite-4.1-8b
# (brokkr R-target, 2026-08-10) that never received the operator ruling it was
# pending. Its comparator is gone (granite retired from the roster 2026-08-15),
# it was deliberately never wired into any default/fallback routing chain, and
# LiteLLM spend logs showed 0 calls in the 4-day window ending 2026-08-21.
# Freed 8,772 MiB on ana-ml2 GPU1. The `lfm2.5-2.6b` gateway alias was removed
# in the same pass so the name 404s cleanly rather than erroring against a dead
# backend. Weights remain in the shared HF cache; nothing was deleted from disk.
networks:
tnet: