fix(nevermore): repoint onto live aliases — its LLM pass had been dead 8 days

nevermore pinned LLAMA_SWAP_MODEL=granite-4.1-8b, an alias retired with the
granite seat on 2026-08-12. Every summarization call since then failed: 67
consecutive status=failure rows, 0 tokens, twice daily, entirely silently. The
briefing had been rendering with no LLM pass at all. Nothing alerts on
status=failure in the spend logs, so it took an unrelated question about
reranker VRAM to surface it.

It was also pinned to NEVERMORE_RERANK_MODEL=qwen3-reranker -- the incumbent
Brokkr R43 measured harming 80/90 fleet queries -- and was its ONLY caller,
while the production `reranker` alias sat at 0 calls for 4 days. The R43
cutover repointed the alias but never moved the consumer.

  nevermore/.env  LLAMA_SWAP_MODEL       granite-4.1-8b -> summarizer
                  NEVERMORE_RERANK_MODEL qwen3-reranker -> reranker
                  (server-only; .env is excluded from the mirror both ways)

Verified against nevermore's exact call shape: summarizer returns clean content
with 0 reasoning chars at temperature 0.2 / max_tokens 4000; reranker scores
0.95 on-topic vs ~1e-5 off-topic; embedding returns dim-1024.

Retired alongside it:

  vllm-rerank    :8002  Qwen3-Reranker-0.6B + the qwen3-reranker alias
  vllm-rerank-a4 :8014  gte-reranker-modernbert + its alias
  vllm-granite   :8004  Exited 8 days, dead service block

and vllm-rerank-a3 was promoted from a throwaway `docker run` into this stack
(the selection ledger's own open follow-up). Healthy in 55s. It keeps the
bake-off arm name so the ledger, memory and R43 record stay valid.

VLLM_VERSION is pinned latest -> v0.24.0. Every service in the stack shares that
one variable, so a bare `compose up -d` could have silently upgraded all of
them at once; both tags resolved to the same local image (4091d5593f77), so the
pin changed nothing at runtime.

GPU1 is down to 81,448 of 97,887 MiB -- 13.9 GB reclaimed tonight.

Correction: an earlier claim that A4 had no gateway alias was wrong. It did.
LiteLLM serves both config-defined and DB-defined models -- live showed 32
against config.yaml's 26 -- and grepping the file cannot see the difference.
/v1/models and /model/info (which flags db_model) are the ground truth. DB
models delete hot via POST /model/delete with no restart.

Left alone: reranker-a3-bge-v2-m3, a zero-call duplicate of `reranker` on the
same backend. It is Brokkr's cutover-verification handle -- redundant rather
than broken, and another agent's tooling is not mine to delete unilaterally.

persistent-memory.md

@@ -134,16 +134,18 @@ _As of 2026-08-20 23:30 — **the Heretic-300 session** (see the 🔴 entry abov
 
 - **🟢 LITELLM — upgraded v1.91.0→v1.97.0, spend-log DB purged 6GB→16MB + CAPPED (2026-08-17).** `store_prompts_in_spend_logs:false` + `maximum_spend_logs_retention_period:7d`. ⚠ **1.8GB pre-upgrade pg_dump still on ana-docker `/opt/docker/compose/litellm/` — deletable now the upgrade is proven** (operator was going to call it). Commit `01b5ad9`.
 
-- **⚠️ GPU zero-sum (both cards ~94–95/97.9 GB).** GPU0: gen + meromero. GPU1: fablefusion + utility cluster. Any util bump on either seat of a shared card must be checked against the co-tenant (starved meromero into a crash-loop once at 0.45). **⚠️ BOOT ORDER IS PART OF THE STATE (2026-08-20).** `--gpu-memory-utilization` sets the target as a fraction of **TOTAL** VRAM, but vLLM **refuses to start unless that whole target is FREE right now** — so at ~96.4/97.9 GB the GPU0 pair coexists *only* in the order it was originally brought up. **Restore/reboot order: `vllm-meromero-rp` to `healthy` FIRST, then `vllm-gen`** — meromero (0.52 = 49.38 GiB) is the one that cannot fit in the remainder. "First" means **observed healthy**, not a `sleep`: a 10s gap against a 2–3 min weight load cost a 7-restart crash-loop. Verify a restore against **KV-pool size** (`GPU KV cache size` / `Maximum concurrency` in the container log), not `nvidia-smi` used-MiB — the latter swings ~7 GB on allocator slack with identical serving capacity. Baselines: gen ≈14.36 GiB / 403k tok / 1.54× (h300 build: 401,550 tok / 1.53×); meromero 542,202 tok. **📊 MEASURED VRAM CENSUS 2026-08-20 23:20** (nvidia-smi PID→container, not util-fraction guesses) — **GPU0 92,572/97,887 MiB (94.6%), 5.2 GB free:** meromero 50,072 + gen 42,500. **GPU1 86,667/97,887 MiB (88.5%), 11.0 GB free** *(after the lfm25 retirement below; was 95,388/2.4 GB)*: fablefusion-probe **43,452** + selene 16,870 + reward 9,512 + coder 6,158 + rerank 3,586 + embed 3,304 + rerank-a3 2,314 + rerank-a4 1,420. **Still no room for a ~22 GB PPL probe seat on either card without stopping something.** ⚠ **fablefusion is the single biggest reclaimable block (43.4 GB) and is nearly idle** — LiteLLM spend logs show `char-rp-probe` at **4 calls, last 2026-08-19 08:52**, vs `char-rp` (meromero) at 129 calls, last 2026-08-20 15:28.
+- **⚠️ GPU zero-sum (both cards ~94–95/97.9 GB).** GPU0: gen + meromero. GPU1: fablefusion + utility cluster. Any util bump on either seat of a shared card must be checked against the co-tenant (starved meromero into a crash-loop once at 0.45). **⚠️ BOOT ORDER IS PART OF THE STATE (2026-08-20).** `--gpu-memory-utilization` sets the target as a fraction of **TOTAL** VRAM, but vLLM **refuses to start unless that whole target is FREE right now** — so at ~96.4/97.9 GB the GPU0 pair coexists *only* in the order it was originally brought up. **Restore/reboot order: `vllm-meromero-rp` to `healthy` FIRST, then `vllm-gen`** — meromero (0.52 = 49.38 GiB) is the one that cannot fit in the remainder. "First" means **observed healthy**, not a `sleep`: a 10s gap against a 2–3 min weight load cost a 7-restart crash-loop. Verify a restore against **KV-pool size** (`GPU KV cache size` / `Maximum concurrency` in the container log), not `nvidia-smi` used-MiB — the latter swings ~7 GB on allocator slack with identical serving capacity. Baselines: gen ≈14.36 GiB / 403k tok / 1.54× (h300 build: 401,550 tok / 1.53×); meromero 542,202 tok. **📊 MEASURED VRAM CENSUS 2026-08-20 23:20** (nvidia-smi PID→container, not util-fraction guesses) — **GPU0 92,572/97,887 MiB (94.6%), 5.2 GB free:** meromero 50,072 + gen 42,500. **GPU1 81,448/97,887 MiB (83.2%), 16.1 GB free** *(after the lfm25 + reranker cleanups below; was 95,388 / 2.4 GB free — **13.9 GB reclaimed in one night**)*: fablefusion-probe **43,452** + selene 16,870 + reward 9,512 + coder 6,158 + embed 3,304 + rerank-a3 2,112. **Still no room for a ~22 GB PPL probe seat — fablefusion is the only remaining block big enough.** ⚠ **fablefusion is the single biggest reclaimable block (43.4 GB) and is nearly idle** — LiteLLM spend logs show `char-rp-probe` at **4 calls, last 2026-08-19 08:52**, vs `char-rp` (meromero) at 129 calls, last 2026-08-20 15:28.
 
 - **FLEET RERANKER** = A3 (bge-reranker-v2-m3) PROD ana-ml2 GPU1 :8013. Passive watch; levers = A4 :8014 / util / 2nd replica; incumbent :8002 warm. `docs/pfi/reranker-selection-ledger.md`.
 
-- **🔴 RERANKER FLEET AUDIT 2026-08-20 — the R43 cutover is only half-landed; `nevermore` is still running on the reranker Brokkr measured as HARMFUL.** Three reranker seats are up, and the usage is backwards from the design:
-  - `:8013` **A3 bge-reranker-v2-m3 — PRODUCTION**, backs the `reranker` alias. **0 calls** in the 4-day spend window to 2026-08-21. 2,314 MiB.
-  - `:8002` **Qwen3-Reranker-0.6B — the RETIRED incumbent**, kept warm only as a rollback path. **STILL SERVING LIVE TRAFFIC: 7 calls, 12-hourly at 03:00/15:00**, most recently 2026-08-21T03:00:31, all from api_key `dc0af5ba431b…`. 3,586 MiB.
-  - `:8014` **A4 gte-reranker-modernbert — "throughput fallback"**, but it has **NO LiteLLM alias at all** (config has only `qwen3-reranker` and `reranker`; the `reranker-a4-gte-modernbert` name in global `CLAUDE.md` is **STALE — it does not exist**). Unreachable through the gateway; 1,420 MiB serving nobody.
-  - **Root cause of the live-traffic finding:** `nevermore` is hard-wired to the incumbent **by name** — `/opt/docker/compose/nevermore/.env` has `NEVERMORE_RERANK_MODEL=qwen3-reranker` (+ `NEVERMORE_EMBED_MODEL=qwen3-embedding`, which is fine, that's the live embedder) pointed at the gateway. The R43 cutover repointed the `reranker` alias but **deliberately left `qwen3-reranker` naming the Qwen model** (correct under the no-false-aliases rule) — so nevermore never moved. Brokkr R43 measured that model **harming 80/90 fleet queries** (no-reranker beat it 89/90 vs 56/90), so nevermore's twice-daily rerank pass is very likely degrading its own briefing.
-  - **Fix is one line in nevermore's `.env`** (`NEVERMORE_RERANK_MODEL=reranker`) + a nevermore restart — NOT a gateway restart. Do that BEFORE retiring `:8002`, or nevermore's rerank pass breaks. **Then** `:8002` and `:8014` are both genuinely free to retire (~5.0 GB).
+- **🟢 RERANKER + NEVERMORE CLEANUP — RESOLVED 2026-08-20 (found while answering "why do we have 3 rerankers?").** The R43 cutover was only half-landed: it repointed the `reranker` alias but never moved the one consumer that mattered.
+  - **★ THE BIG ONE — `nevermore`'s LLM summarization had been DEAD for 8 days and nothing noticed.** Its `.env` pinned `LLAMA_SWAP_MODEL=granite-4.1-8b`, an alias retired 2026-08-12 with the granite seat. Result: **67 consecutive failures, 0 tokens, status=failure**, twice daily, silently — the twice-daily briefing was rendering with no LLM pass at all. Nothing alerts on `status=failure` in the spend logs, so this was invisible until someone went looking for something else.
+  - **Also found:** `NEVERMORE_RERANK_MODEL=qwen3-reranker` — nevermore was the ONLY caller of the retired Qwen incumbent (7 calls, 12-hourly at 03:00/15:00 UTC = its 08:00/20:00 PDT cron), the model Brokkr measured **harming 80/90 fleet queries**. Its rerank calls succeeded; they were just running through the bad model. Meanwhile the production `reranker` alias had **0 calls in 4 days**.
+  - **★ THE RULE THIS EARNS: retiring a model is a TWO-SIDED operation.** Grep every consumer's config for the alias *before* deleting it. And consumers must pin stable **capability aliases** (`summarizer`, `reranker`) never model names (`granite-4.1-8b`, `qwen3-reranker`) — then the gateway can repoint without anyone editing a downstream `.env`. Both of nevermore's breakages are the same bug.
+  - **FIXED:** nevermore `.env` → `LLAMA_SWAP_MODEL=summarizer` + `NEVERMORE_RERANK_MODEL=reranker` (backup `.env.bak-pre-model-repoint-20260820`), worker recreated, all three deps verified live — summarizer returns clean content with 0 reasoning chars at nevermore's exact call shape (`temperature 0.2`, `max_tokens 4000`), reranker scores 0.95 on-topic vs ~1e-5 off-topic, embedding returns dim-1024. ⚠ `NEVERMORE_EMBED_MODEL=qwen3-embedding` was already correct — left alone. ⚠ nevermore's `.env` is **server-only** (`.env` is excluded from the mirror both ways), so this fix is not in git.
+  - **RETIRED:** `vllm-rerank` (:8002 Qwen incumbent) + its `qwen3-reranker` alias; `vllm-rerank-a4` (:8014) + its alias; `vllm-granite` (Exited 8 days, dead service block). **A3 PROMOTED** from a throwaway `docker run` into `stacks/vllm` as service `vllm-rerank-a3` (the ledger's own open follow-up) — healthy in 55s, image pinned. Container name **deliberately keeps the bake-off arm name** so the ledger/memory/R43 references stay valid.
+  - **⚠️ CORRECTION — my "A4 has no alias at all" claim was WRONG.** `reranker-a4-gte-modernbert` **did** exist; I grepped `config.yaml` and concluded absence. **LiteLLM serves BOTH config-defined AND DB-defined models** — live was **32** models, `config.yaml` only **26**. The 6 DB-only ones: `ext-tts`, `gpt-4o-mini-tts`, `tts-1`, `tts-1-hd`, `reranker-a3-bge-v2-m3`, `reranker-a4-gte-modernbert`. **`/v1/models` (or `/model/info`, which flags `db_model: true`) is the ground truth — never `config.yaml` alone.** DB models delete hot via `POST /model/delete {"id": …}` with **no gateway restart**; config models need a file edit + `docker restart litellm` (~84s). A4's alias was deleted that way once its backend was gone.
+  - **⏳ REMAINING:** `reranker-a3-bge-v2-m3` (DB-defined, id `1f08a73e-c09c-463c-8f18-5edea51fb736`) still exists as a **duplicate of `reranker` on the same backend**, 0 calls. It's Brokkr's cutover-verification handle (prod == arm at maxdiff 0.000000), so **not removed unilaterally** — it is another agent's tooling, and it is redundant rather than broken. Ask Brokkr before deleting.
 
 - **EVIDENCE HOLD (partial):** WT #394 FILE half STILL STANDS — do NOT delete on-disk gen dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`); rex393-fiction-* + r42-gate-* KEEP.
 
@@ -172,9 +174,9 @@ _As of 2026-08-20 23:30 — **the Heretic-300 session** (see the 🔴 entry abov
   ⚠ **Never set the ZFS cachefile on one pool.** The runbook's `zpool set cachefile=… nvme` was a trap: populating a cache flips the host from import-by-scan to import-by-cache, so a cache holding only `nvme` leaves `ssd`+`tank` unimported and empties every CT 103 export. Set on all three 2026-08-18, verified in the 11,976-byte cache.
   ⚠ **Migrate FIRST, patch after** — a signed kernel would land in `/boot` on the 1.3 GB root. ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on this host and serves `hard` NFS to esh-docker-vm and esh-pve — quiesce both before any reboot** or you wedge esh-docker-vm into D-state. Off-box at `nh3-dev:~/backups/esh-pve-nas/`: DOM image `dom-sdq-20260818.img.zst` (2.38 GiB, crash-consistent), clean `bootchain-20260818.tar.gz`, config snapshot `…20260818T051*.tar.gz`. Runbook `docs/runbooks/esh-pve-nas-boot-migration.md`; detail → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`.
 
-- **⚪ LFM2.5-2.6B RETIRED PERMANENTLY 2026-08-20 (operator directive).** `vllm-lfm25` (:8021, GPU1) removed: service deleted from `stacks/vllm/compose.yaml` + pushed live (backup `compose.yaml.bak-pre-lfm25-retire-20260820`), container `docker rm -f`'d, `lfm2.5-2.6b` alias deleted from the LiteLLM config (live + canonical; backup `config.yaml.bak-pre-lfm25-retire-20260820`, 28→27 models). **Freed 8,721 MiB on GPU1.** Justification: it was an EVAL-ONLY bake-off seat vs `granite-4.1-8b` (brokkr R-target 2026-08-10) that never received the operator ruling it was pending; the comparator was retired from the roster 2026-08-15; it was deliberately never in any default/fallback routing chain; and spend logs showed **0 calls in the 4-day window**. Weights remain in the shared HF cache — nothing deleted from disk. ⚠ **The gateway restart that makes the alias-deletion take effect was HELD** so it could batch with a reranker change — until `docker restart litellm` runs, `lfm2.5-2.6b` is still routable in-memory and will error against a dead backend. ⚠ `vllm-granite` is **still a defined service** in `stacks/vllm/compose.yaml` though the model was retired 2026-08-12 — dead config, 0 VRAM (stopped), worth the same cleanup pass.
+- **⚪ LFM2.5-2.6B RETIRED PERMANENTLY 2026-08-20 (operator directive).** `vllm-lfm25` (:8021, GPU1) removed: service deleted from `stacks/vllm/compose.yaml` + pushed live (backup `compose.yaml.bak-pre-lfm25-retire-20260820`), container `docker rm -f`'d, `lfm2.5-2.6b` alias deleted from the LiteLLM config (live + canonical; backup `config.yaml.bak-pre-lfm25-retire-20260820`, 28→27 models). **Freed 8,721 MiB on GPU1.** Justification: it was an EVAL-ONLY bake-off seat vs `granite-4.1-8b` (brokkr R-target 2026-08-10) that never received the operator ruling it was pending; the comparator was retired from the roster 2026-08-15; it was deliberately never in any default/fallback routing chain; and spend logs showed **0 calls in the 4-day window**. Weights remain in the shared HF cache — nothing deleted from disk. The held gateway restart fired at 23:46 alongside the `qwen3-reranker` removal (one ~84s blip covered both); `lfm2.5-2.6b`, `qwen3-reranker` and `granite-4.1-8b` all now 400 cleanly. `vllm-granite`'s dead service block was removed in the same pass.
 
-- **OPEN FOLLOW-UPS (parked):** repoint `nevermore` off the retired reranker (see the 🔴 reranker audit) then retire `:8002`/`:8014`; delete the dead `vllm-granite` service block; move gen seat off pinned-nightly to stable once #51113 ships; Lobe one-time TTS UI pass; delete the 1.8GB litellm dump; `harden-esh-docker-vm` (park id 28, PROMOTED — Tier-1 done, `/mnt/books` stays hard w/ watchdog); chatterbox-fast build-context divergence; #363 research-wing ingest (no deadline); optionally attach our MTP reproducer to vllm#47087 (needs a GitHub identity — operator's call).
+- **OPEN FOLLOW-UPS (parked):** ask Brokkr whether the duplicate `reranker-a3-bge-v2-m3` alias can go; move gen seat off pinned-nightly to stable once #51113 ships; Lobe one-time TTS UI pass; delete the 1.8GB litellm dump; `harden-esh-docker-vm` (park id 28, PROMOTED — Tier-1 done, `/mnt/books` stays hard w/ watchdog); chatterbox-fast build-context divergence; #363 research-wing ingest (no deadline); optionally attach our MTP reproducer to vllm#47087 (needs a GitHub identity — operator's call).
 
 - **althing monitor** ARMED (handle `infra-ops`). ⚠️ Re-arm ONLY after a real FIRE (rc0), never after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, never chained with `&` (orphans it — hit this twice 2026-08-17, `stop-monitor` reclaims).