fix(nevermore): repoint onto live aliases — its LLM pass had been dead 8 days

nevermore pinned LLAMA_SWAP_MODEL=granite-4.1-8b, an alias retired with the granite seat on 2026-08-12. Every summarization call since then failed: 67 consecutive status=failure rows, 0 tokens, twice daily, entirely silently. The briefing had been rendering with no LLM pass at all. Nothing alerts on status=failure in the spend logs, so it took an unrelated question about reranker VRAM to surface it. It was also pinned to NEVERMORE_RERANK_MODEL=qwen3-reranker -- the incumbent Brokkr R43 measured harming 80/90 fleet queries -- and was its ONLY caller, while the production `reranker` alias sat at 0 calls for 4 days. The R43 cutover repointed the alias but never moved the consumer. nevermore/.env LLAMA_SWAP_MODEL granite-4.1-8b -> summarizer NEVERMORE_RERANK_MODEL qwen3-reranker -> reranker (server-only; .env is excluded from the mirror both ways) Verified against nevermore's exact call shape: summarizer returns clean content with 0 reasoning chars at temperature 0.2 / max_tokens 4000; reranker scores 0.95 on-topic vs ~1e-5 off-topic; embedding returns dim-1024. Retired alongside it: vllm-rerank :8002 Qwen3-Reranker-0.6B + the qwen3-reranker alias vllm-rerank-a4 :8014 gte-reranker-modernbert + its alias vllm-granite :8004 Exited 8 days, dead service block and vllm-rerank-a3 was promoted from a throwaway `docker run` into this stack (the selection ledger's own open follow-up). Healthy in 55s. It keeps the bake-off arm name so the ledger, memory and R43 record stay valid. VLLM_VERSION is pinned latest -> v0.24.0. Every service in the stack shares that one variable, so a bare `compose up -d` could have silently upgraded all of them at once; both tags resolved to the same local image (4091d5593f77), so the pin changed nothing at runtime. GPU1 is down to 81,448 of 97,887 MiB -- 13.9 GB reclaimed tonight. Correction: an earlier claim that A4 had no gateway alias was wrong. It did. LiteLLM serves both config-defined and DB-defined models -- live showed 32 against config.yaml's 26 -- and grepping the file cannot see the difference. /v1/models and /model/info (which flags db_model) are the ground truth. DB models delete hot via POST /model/delete with no restart. Left alone: reranker-a3-bge-v2-m3, a zero-call duplicate of `reranker` on the same backend. It is Brokkr's cutover-verification handle -- redundant rather than broken, and another agent's tooling is not mine to delete unilaterally.
2026-08-20 23:50:53 -07:00
parent b990951d80
commit 1d3b80169a
4 changed files with 91 additions and 179 deletions
@@ -4,12 +4,19 @@
 #   # edit .env with real values
 #   docker compose up -d

-# Image version — pin for reproducibility (`latest` for edge)
-VLLM_VERSION=latest
+# Image version — PINNED. Do not put `latest` here: every service in this file
+# shares this one variable, so a bare `docker compose up -d` would silently
+# upgrade the whole stack's vLLM at once. Pinned 2026-08-20 to the version all
+# seats were already running (`latest` and `v0.24.0` were the same local image,
+# 4091d5593f77, so the pin changed nothing at runtime). Bump deliberately.
+VLLM_VERSION=v0.24.0

 # Host ports (container always listens on 8000 internally)
 EMBED_PORT=8001
-RERANK_PORT=8002
+# 8013 — the reranker moved here 2026-08-20 when bge-v2-m3 (the R43 winner, which
+# had been running as a throwaway `docker run` on this port) was promoted into
+# this stack and the Qwen incumbent on :8002 was retired.
+RERANK_PORT=8013
 REWARD_PORT=8003

 # GPU assignment — all services share this GPU
@@ -18,7 +25,10 @@ GPU_ID=1

 # Models — reference by full repo name in API requests
 EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
-RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B
+# bge-reranker-v2-m3 — the R43 bake-off winner, replacing Qwen3-Reranker-0.6B
+# (measured HARMING 80/90 fleet queries). Multilingual cross-encoder; needs no
+# --hf-overrides, unlike the Qwen reranker it displaced.
+RERANK_MODEL=BAAI/bge-reranker-v2-m3
 # Skywork is a local-path AWQ output, not from HF Hub. Bind-mounted into the
 # reward container at /local-models — see compose.yaml. No env var here for
 # the model path itself since it's hard-coded in the compose command.
@@ -72,62 +82,12 @@ API_KEY=
 # services (embed/rerank). Reward is local-path, ignores this.
 HF_TOKEN=

-# === granite-4.1-8b (production summarizer / dreaming agent) ===
-# Replaced phi4-mini 2026-06-05 (Granite 4.1 8B beat phi4 on precision in
-# brokkr's R15 P03 model-fitness eval). Same GPU-1 slot, reusing phi4's port.
-GRANITE_PORT=8004
-# GPU 1 — co-located with the embed/rerank/reward trio. With phi4 retired, GPU 1
-# has ~19 GB free; granite at 56K + FP8 KV needs ~15-16 GB → ~2-3 GB margin.
-# NOTE: llama-swap also uses GPU 1 dynamically — a large swap-in could contend;
-# pin llama-swap to GPU 0 for clean separation (follow-up).
-GRANITE_GPU_ID=1
-# Official IBM pre-quantized FP8 (compressed-tensors) — calibrated, ~9.6 GB,
-# loaded directly (FP8 native on Blackwell cc 12.0). Fallback to vLLM-native dynamic FP8
-# from BF16: GRANITE_MODEL=ibm-granite/granite-4.1-8b + GRANITE_QUANT=fp8.
-GRANITE_MODEL=ibm-granite/granite-4.1-8b-fp8
-GRANITE_QUANT=compressed-tensors
-GRANITE_SERVED_NAME=granite-4.1-8b
-# 131072 ctx — MAXED 2026-06-13 (was 51200/50K). GPU-1 rebalance: granite shares
-# the card with the trio + qwen35-vl (vision). Qwen was over-provisioned on KV
-# (20x conc @ 32k), so trimming it freed room for granite's ~305k-token pool here.
-# PagedAttention allocates KV per ACTUAL token, so 131072 is only a CEILING — a 1k
-# summarize turn uses ~1k tokens, so the pool holds ~300 concurrently; the "2.33x"
-# headline is worst-case (every request maxing 131k). Granite 4.1 supports 131072.
-# 65536 — REDUCED 2026-06-14 (was 131072) to free GPU-1 room for the FP8 vision
-# model (Qwen3.6-35B-A3B, stacks/qwen36-vl, ~34 GB weights). Summarizer load is
-# short parallel calls, so the 64K cap is ample.
-# 131072 — RESTORED 2026-07-16 (native max) for full-chapter summarization; GPU-1
-# freed by the image-bench evict + qwen36-vl gone, so the 128K ctx fits again.
-# 16384 — SHRUNK 2026-07-27 (operator: granite is being phased out) to free GPU-1
-# room for vllm-coder (the Zed FIM seat). Full-chapter ctx dropped; util 0.13 holds
-# 16K cleanly (32768 @ util 0.12 crash-looped: KV est-max was only 19376 tokens).
-GRANITE_MAX_MODEL_LEN=16384
-# FP8 KV cache (native on Blackwell cc 12.0). At 50K ≈ ~4.2 GB (vs ~8.4 GB at fp16).
-GRANITE_KV_CACHE_DTYPE=fp8
-# util 0.35 (~33.6 GB) — tuned 2026-06-13 to leave ~3.5 GB free on GPU 1 alongside
-# the trio + qwen co-tenants. On this shared card vLLM needs free >= util*total at
-# startup, and START ORDER matters: trim qwen FIRST, then grow granite, else granite
-# OOMs against the full card. (0.37 overshot to 1.7 GB free; 0.35 lands ~3.7 GB.)
-# 0.24 — REBALANCED 2026-06-14 (was 0.35) for the FP8 vision cutover. GPU-1 budget:
-# qwen36-vl 0.46 + granite 0.24 + reward 0.10 + embed/rerank 0.03 ≈ 0.90 total,
-# ~7.5 GB headroom (the OOM buffer; held under 20-concurrent load test). granite
-# gets a 169K-token KV pool = 2.58x concurrency @ 64K. Bring qwen36 up LAST.
-# 0.18 — RIGHT-SIZED 2026-07-16 (was 0.34 live; qwen36-vl no longer a GPU-1 tenant,
-# image-bench evicted) to free ~10.5 GB for relocating a GPU0 model onto GPU1. KV
-# 6.45 GiB = 84,528 tokens = 1.29x concurrency @ 65536 (summarizer = short parallel
-# calls; ample). Effective slope on this shared card ≈ 950 MiB KV per 0.01 util, and
-# KV must hold >= 1x max-model-len — util 0.15 undershot (crash: est max-len 47184 <
-# 65536), 0.18 lands the target cleanly.
-# 0.27 — RE-GROWN 2026-07-16 (same session) after char-rp moved onto GPU-1: spend the
-# leftover room on full-chapter context (max-len 131072). KV 15.0 GiB = 196,560 tokens
-# = 1.50x @ 131072; GPU-1 lands ~6.7 GB headroom (char-rp 30 + granite 27 + selene 17 + trio).
-# 0.13 — SHRUNK 2026-07-27 (was 0.27) with the max-len drop; frees ~14 GB on GPU-1 for
-# vllm-coder. granite phasing out, so no longer worth the big KV pool. (0.12 was too
-# small for even 16K KV; 0.13 gives ~1.4x @ 16384.)
-GRANITE_GPU_MEM_UTIL=0.13
-# Concurrency cap. Was 1024 (fan-out summarizer); DROPPED to 256 on 2026-07-27 with the
-# phasing-out shrink (256 is ample for the reduced summarizer load; smaller sched state).
-GRANITE_MAX_NUM_SEQS=256
+# === granite-4.1-8b — RETIRED 2026-08-12, vars removed 2026-08-20 ===
+# Was the production summarizer. The `summarizer` / `classifier` gateway aliases
+# were repointed at the gen seat and the container stopped; the service block and
+# these tunables are now gone. See the tombstone in compose.yaml for the lesson
+# this cost us (nevermore stayed pinned to the dead `granite-4.1-8b` alias and
+# failed silently for 8 days).

 # Qwen2.5-Coder-1.5B (BASE) — FIM code-completion seat for Zed editor inline
 # edit-predictions (deep-research pick 2026-07-27, Apache-2.0). GPU-1, alongside the
@@ -143,17 +103,8 @@ CODER_KV_CACHE_DTYPE=fp8
 CODER_GPU_MEM_UTIL=0.06
 CODER_MAX_NUM_SEQS=32

-# LFM2.5-2.6B — NON-PRODUCTION bake-off alias vs granite (brokkr R-target 2026-08-10).
-# LiquidAI LFM Open License v1.0 (<USD 10M-rev commercial, not OSI) — eval-only pending
-# an operator production ruling. Reasoning model (emits <think>); served raw (no vLLM
-# reasoning-parser) so content is non-empty. util 0.09 (~8.6GB) fits GPU1's free slack
-# without touching production reservations; max-len 16384 keeps KV small (bake-off
-# doesn't need the 131k ceiling). Vendor sampling lives in the LiteLLM alias.
-LFM25_PORT=8021
-LFM25_GPU_ID=1
-LFM25_MODEL=LiquidAI/LFM2.5-2.6B
-LFM25_SERVED_NAME=lfm2.5-2.6b
-LFM25_MAX_MODEL_LEN=16384
-LFM25_KV_CACHE_DTYPE=auto
-LFM25_GPU_MEM_UTIL=0.09
-LFM25_MAX_NUM_SEQS=8
+# === LFM2.5-2.6B — RETIRED PERMANENTLY 2026-08-20 (operator directive) ===
+# Eval-only bake-off seat vs granite-4.1-8b that never got its production ruling;
+# its comparator was retired first, and it logged 0 calls in the 4 days before it
+# came down. Container removed, service block and tunables deleted, gateway alias
+# dropped. Weights remain in the shared HF cache.
@@ -83,9 +83,27 @@ services:
      - homepage.description=Qwen3 Embedding via vLLM (ana-ml2)
      - homepage.href=http://10.250.50.54:${EMBED_PORT}/docs

-  vllm-rerank:
+  # THE fleet reranker. Backs the LiteLLM `reranker` alias, which is what every
+  # consumer should name — never the model, never a bake-off arm name.
+  #
+  # It won the Brokkr R43 bake-off (docs/pfi/reranker-selection-ledger.md) and
+  # replaced Qwen3-Reranker-0.6B, which was measured HARMING 80/90 fleet queries
+  # (no-reranker beat it 89/90 vs 56/90). The R42 v13 acceptance gate went
+  # 56/90 -> 90/90 on the cutover, its first-ever PASS.
+  #
+  # Promoted from a throwaway `docker run` to this service 2026-08-20 (the
+  # ledger's own open follow-up). It carried the bake-off's arm name from the
+  # start and KEEPS it: the ledger, persistent-memory and the R43 record all say
+  # `vllm-rerank-a3`, and renaming for tidiness would orphan every one of those
+  # references. The name carries its provenance.
+  #
+  # Retired alongside this promotion: `vllm-rerank` (Qwen3-Reranker-0.6B, :8002 —
+  # the rollback path, kept warm 13 days) and `vllm-rerank-a4`
+  # (gte-reranker-modernbert, :8014 — a documented throughput fallback that was
+  # never given a gateway alias, so it was unreachable the whole time).
+  vllm-rerank-a3:
    image: vllm/vllm-openai:${VLLM_VERSION}
-    container_name: vllm-rerank
+    container_name: vllm-rerank-a3
    restart: unless-stopped
    ipc: host
    ports:
@@ -103,8 +121,9 @@ services:
      - ${RERANK_MODEL}
      - --runner
      - pooling
-      - --hf-overrides
-      - '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}'
+      # No --hf-overrides here. The Qwen reranker needed one to be coerced into a
+      # sequence-classification head; bge-reranker-v2-m3 is natively a
+      # cross-encoder and vLLM resolves it directly.
      - --host
      - 0.0.0.0
      - --port
@@ -134,9 +153,9 @@ services:
      - tnet
    labels:
      - homepage.group=AI - Eval & Retrieval
-      - homepage.name=vLLM Rerank (Qwen3)
+      - homepage.name=vLLM Rerank (bge-v2-m3)
      - homepage.icon=mdi-sort-variant
-      - homepage.description=Qwen3 Reranker via vLLM (ana-ml2)
+      - homepage.description=BAAI bge-reranker-v2-m3 — the fleet reranker, backs the `reranker` alias (ana-ml2)
      - homepage.href=http://10.250.50.54:${RERANK_PORT}/docs

  vllm-reward:
@@ -197,86 +216,24 @@ services:
      - homepage.description=Skywork-Reward-V2 8B classifier via vLLM (ana-ml2)
      - homepage.href=http://10.250.50.54:${REWARD_PORT}/docs

-  # Granite 4.1 8B (FP8) — production summarizer (replaced phi4-mini
-  # 2026-06-05, which had superseded the llama-swap granite-4-small pin).
-  # Generative chat model (OpenAI /v1/chat/completions), so NO --runner
-  # pooling. FP8 on RTX PRO 6000 Blackwell (cc 12.0): near-lossless, ~1.2x, ~6 GB.
-  vllm-granite:
-    image: vllm/vllm-openai:${VLLM_VERSION}
-    container_name: vllm-granite
-    restart: unless-stopped
-    ipc: host
-    ports:
-      - "${GRANITE_PORT}:8000"
-    volumes:
-      - /tank/aimodels/huggingface:/hfcache
-    environment:
-      - HF_HOME=/hfcache
-      - HF_HUB_CACHE=/hfcache/hub
-      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
-      - VLLM_API_KEY=${API_KEY:-}
-    command:
-      # Production summarizer (replaced phi4-mini 2026-06-05). Default = official
-      # IBM pre-quantized FP8 (compressed-tensors), loaded directly; FP8 is native
-      # on the RTX PRO 6000 Blackwell (cc 12.0). Fallback to vLLM-native dynamic FP8 from
-      # BF16: GRANITE_MODEL=ibm-granite/granite-4.1-8b + GRANITE_QUANT=fp8.
-      - ${GRANITE_MODEL}
-      - --served-model-name
-      - ${GRANITE_SERVED_NAME}
-      - --quantization
-      - ${GRANITE_QUANT}
-      - --host
-      - 0.0.0.0
-      - --port
-      - "8000"
-      - --gpu-memory-utilization
-      - ${GRANITE_GPU_MEM_UTIL}
-      - --max-model-len
-      - ${GRANITE_MAX_MODEL_LEN}
-      # Very high so the KV pool (not the seq cap) is the only concurrency bound —
-      # granite is the fleet fan-out summarizer/classifier (many concurrent SHORT
-      # calls). vLLM's default resolves to 128, capping below the KV bound
-      # (~192 @ 1K-tok); 1024 unblocks it (VRAM-neutral — KV pool is util-bound).
-      - --max-num-seqs
-      - ${GRANITE_MAX_NUM_SEQS}
-      - --dtype
-      - auto
-      # CUDA graphs ENABLED (no --enforce-eager) for decode throughput. Made
-      # room 2026-06-05 by right-sizing the embed/rerank/reward trio's KV pools
-      # (they were over-provisioned at 5.9x/2.0x/3.9x concurrency); GPU 1 now has
-      # ~17 GB free after granite, so graph-capture buffers fit. If the trio
-      # ever grows back, granite may need --enforce-eager again on this card.
-      # FP8 KV cache — halves KV memory; near-lossless on Blackwell (cc 12.0).
-      - --kv-cache-dtype
-      - ${GRANITE_KV_CACHE_DTYPE}
-      # Prefix caching pinned EXPLICIT (vLLM v1 defaults it on, but pin so a
-      # version flip can't silently disable it). Benched 2026-06-13: ~6.5x faster
-      # TTFT (45ms vs 292ms) on a shared ~4.5k-token summarizer template; soft/
-      # evictable KV, neutral when prefixes don't repeat — pure win for granite.
-      - --enable-prefix-caching
-    deploy:
-      resources:
-        reservations:
-          devices:
-            - driver: nvidia
-              device_ids:
-                - "${GRANITE_GPU_ID}"
-              capabilities:
-                - gpu
-    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-      start_period: 180s
-    networks:
-      - tnet
-    labels:
-      - homepage.group=AI - Inference
-      - homepage.name=vLLM Granite 4.1 8B (summarizer)
-      - homepage.icon=mdi-text-box-outline
-      - homepage.description=Granite 4.1 8B FP8 via vLLM (ana-ml2)
-      - homepage.href=http://10.250.50.54:${GRANITE_PORT}/docs
+  # vllm-granite (ibm-granite/granite-4.1-8b-fp8, :8004) — RETIRED 2026-08-12,
+  # service block removed 2026-08-20. It was the fleet summarizer until the
+  # `summarizer` and `classifier` aliases were repointed at the gen seat; the
+  # container was stopped then and sat Exited for 8 days while this block still
+  # claimed it as production.
+  #
+  # ⚠️ Retiring the SEAT did not retire its CONSUMERS, and nobody checked. The
+  # `granite-4.1-8b` gateway alias was deleted at the same time, but `nevermore`
+  # was still pinned to that alias by name — so its LLM summarization pass failed
+  # 67 consecutive times over 8 days, 0 tokens, entirely silently, because a
+  # failed gateway call still returns 200-shaped spend-log rows and nothing
+  # alerts on status=failure. Found 2026-08-20 only because someone asked an
+  # unrelated question about reranker VRAM.
+  #
+  # The rule this earns: RETIRING A MODEL IS A TWO-SIDED OPERATION. Grep every
+  # consumer's config for the alias BEFORE deleting it, and prefer that consumers
+  # pin stable ALIASES (`summarizer`) over model names (`granite-4.1-8b`) so the
+  # gateway can repoint them without anyone editing a downstream .env.

  vllm-coder:
    image: vllm/vllm-openai:${VLLM_VERSION}