fix(nevermore): repoint onto live aliases — its LLM pass had been dead 8 days
nevermore pinned LLAMA_SWAP_MODEL=granite-4.1-8b, an alias retired with the
granite seat on 2026-08-12. Every summarization call since then failed: 67
consecutive status=failure rows, 0 tokens, twice daily, entirely silently. The
briefing had been rendering with no LLM pass at all. Nothing alerts on
status=failure in the spend logs, so it took an unrelated question about
reranker VRAM to surface it.
It was also pinned to NEVERMORE_RERANK_MODEL=qwen3-reranker -- the incumbent
Brokkr R43 measured harming 80/90 fleet queries -- and was its ONLY caller,
while the production `reranker` alias sat at 0 calls for 4 days. The R43
cutover repointed the alias but never moved the consumer.
nevermore/.env LLAMA_SWAP_MODEL granite-4.1-8b -> summarizer
NEVERMORE_RERANK_MODEL qwen3-reranker -> reranker
(server-only; .env is excluded from the mirror both ways)
Verified against nevermore's exact call shape: summarizer returns clean content
with 0 reasoning chars at temperature 0.2 / max_tokens 4000; reranker scores
0.95 on-topic vs ~1e-5 off-topic; embedding returns dim-1024.
Retired alongside it:
vllm-rerank :8002 Qwen3-Reranker-0.6B + the qwen3-reranker alias
vllm-rerank-a4 :8014 gte-reranker-modernbert + its alias
vllm-granite :8004 Exited 8 days, dead service block
and vllm-rerank-a3 was promoted from a throwaway `docker run` into this stack
(the selection ledger's own open follow-up). Healthy in 55s. It keeps the
bake-off arm name so the ledger, memory and R43 record stay valid.
VLLM_VERSION is pinned latest -> v0.24.0. Every service in the stack shares that
one variable, so a bare `compose up -d` could have silently upgraded all of
them at once; both tags resolved to the same local image (4091d5593f77), so the
pin changed nothing at runtime.
GPU1 is down to 81,448 of 97,887 MiB -- 13.9 GB reclaimed tonight.
Correction: an earlier claim that A4 had no gateway alias was wrong. It did.
LiteLLM serves both config-defined and DB-defined models -- live showed 32
against config.yaml's 26 -- and grepping the file cannot see the difference.
/v1/models and /model/info (which flags db_model) are the ground truth. DB
models delete hot via POST /model/delete with no restart.
Left alone: reranker-a3-bge-v2-m3, a zero-call duplicate of `reranker` on the
same backend. It is Brokkr's cutover-verification handle -- redundant rather
than broken, and another agent's tooling is not mine to delete unilaterally.
This commit is contained in:
+11
-9
@@ -134,16 +134,18 @@ _As of 2026-08-20 23:30 — **the Heretic-300 session** (see the 🔴 entry abov
|
||||
|
||||
- **🟢 LITELLM — upgraded v1.91.0→v1.97.0, spend-log DB purged 6GB→16MB + CAPPED (2026-08-17).** `store_prompts_in_spend_logs:false` + `maximum_spend_logs_retention_period:7d`. ⚠ **1.8GB pre-upgrade pg_dump still on ana-docker `/opt/docker/compose/litellm/` — deletable now the upgrade is proven** (operator was going to call it). Commit `01b5ad9`.
|
||||
|
||||
- **⚠️ GPU zero-sum (both cards ~94–95/97.9 GB).** GPU0: gen + meromero. GPU1: fablefusion + utility cluster. Any util bump on either seat of a shared card must be checked against the co-tenant (starved meromero into a crash-loop once at 0.45). **⚠️ BOOT ORDER IS PART OF THE STATE (2026-08-20).** `--gpu-memory-utilization` sets the target as a fraction of **TOTAL** VRAM, but vLLM **refuses to start unless that whole target is FREE right now** — so at ~96.4/97.9 GB the GPU0 pair coexists *only* in the order it was originally brought up. **Restore/reboot order: `vllm-meromero-rp` to `healthy` FIRST, then `vllm-gen`** — meromero (0.52 = 49.38 GiB) is the one that cannot fit in the remainder. "First" means **observed healthy**, not a `sleep`: a 10s gap against a 2–3 min weight load cost a 7-restart crash-loop. Verify a restore against **KV-pool size** (`GPU KV cache size` / `Maximum concurrency` in the container log), not `nvidia-smi` used-MiB — the latter swings ~7 GB on allocator slack with identical serving capacity. Baselines: gen ≈14.36 GiB / 403k tok / 1.54× (h300 build: 401,550 tok / 1.53×); meromero 542,202 tok. **📊 MEASURED VRAM CENSUS 2026-08-20 23:20** (nvidia-smi PID→container, not util-fraction guesses) — **GPU0 92,572/97,887 MiB (94.6%), 5.2 GB free:** meromero 50,072 + gen 42,500. **GPU1 86,667/97,887 MiB (88.5%), 11.0 GB free** *(after the lfm25 retirement below; was 95,388/2.4 GB)*: fablefusion-probe **43,452** + selene 16,870 + reward 9,512 + coder 6,158 + rerank 3,586 + embed 3,304 + rerank-a3 2,314 + rerank-a4 1,420. **Still no room for a ~22 GB PPL probe seat on either card without stopping something.** ⚠ **fablefusion is the single biggest reclaimable block (43.4 GB) and is nearly idle** — LiteLLM spend logs show `char-rp-probe` at **4 calls, last 2026-08-19 08:52**, vs `char-rp` (meromero) at 129 calls, last 2026-08-20 15:28.
|
||||
- **⚠️ GPU zero-sum (both cards ~94–95/97.9 GB).** GPU0: gen + meromero. GPU1: fablefusion + utility cluster. Any util bump on either seat of a shared card must be checked against the co-tenant (starved meromero into a crash-loop once at 0.45). **⚠️ BOOT ORDER IS PART OF THE STATE (2026-08-20).** `--gpu-memory-utilization` sets the target as a fraction of **TOTAL** VRAM, but vLLM **refuses to start unless that whole target is FREE right now** — so at ~96.4/97.9 GB the GPU0 pair coexists *only* in the order it was originally brought up. **Restore/reboot order: `vllm-meromero-rp` to `healthy` FIRST, then `vllm-gen`** — meromero (0.52 = 49.38 GiB) is the one that cannot fit in the remainder. "First" means **observed healthy**, not a `sleep`: a 10s gap against a 2–3 min weight load cost a 7-restart crash-loop. Verify a restore against **KV-pool size** (`GPU KV cache size` / `Maximum concurrency` in the container log), not `nvidia-smi` used-MiB — the latter swings ~7 GB on allocator slack with identical serving capacity. Baselines: gen ≈14.36 GiB / 403k tok / 1.54× (h300 build: 401,550 tok / 1.53×); meromero 542,202 tok. **📊 MEASURED VRAM CENSUS 2026-08-20 23:20** (nvidia-smi PID→container, not util-fraction guesses) — **GPU0 92,572/97,887 MiB (94.6%), 5.2 GB free:** meromero 50,072 + gen 42,500. **GPU1 81,448/97,887 MiB (83.2%), 16.1 GB free** *(after the lfm25 + reranker cleanups below; was 95,388 / 2.4 GB free — **13.9 GB reclaimed in one night**)*: fablefusion-probe **43,452** + selene 16,870 + reward 9,512 + coder 6,158 + embed 3,304 + rerank-a3 2,112. **Still no room for a ~22 GB PPL probe seat — fablefusion is the only remaining block big enough.** ⚠ **fablefusion is the single biggest reclaimable block (43.4 GB) and is nearly idle** — LiteLLM spend logs show `char-rp-probe` at **4 calls, last 2026-08-19 08:52**, vs `char-rp` (meromero) at 129 calls, last 2026-08-20 15:28.
|
||||
|
||||
- **FLEET RERANKER** = A3 (bge-reranker-v2-m3) PROD ana-ml2 GPU1 :8013. Passive watch; levers = A4 :8014 / util / 2nd replica; incumbent :8002 warm. `docs/pfi/reranker-selection-ledger.md`.
|
||||
|
||||
- **🔴 RERANKER FLEET AUDIT 2026-08-20 — the R43 cutover is only half-landed; `nevermore` is still running on the reranker Brokkr measured as HARMFUL.** Three reranker seats are up, and the usage is backwards from the design:
|
||||
- `:8013` **A3 bge-reranker-v2-m3 — PRODUCTION**, backs the `reranker` alias. **0 calls** in the 4-day spend window to 2026-08-21. 2,314 MiB.
|
||||
- `:8002` **Qwen3-Reranker-0.6B — the RETIRED incumbent**, kept warm only as a rollback path. **STILL SERVING LIVE TRAFFIC: 7 calls, 12-hourly at 03:00/15:00**, most recently 2026-08-21T03:00:31, all from api_key `dc0af5ba431b…`. 3,586 MiB.
|
||||
- `:8014` **A4 gte-reranker-modernbert — "throughput fallback"**, but it has **NO LiteLLM alias at all** (config has only `qwen3-reranker` and `reranker`; the `reranker-a4-gte-modernbert` name in global `CLAUDE.md` is **STALE — it does not exist**). Unreachable through the gateway; 1,420 MiB serving nobody.
|
||||
- **Root cause of the live-traffic finding:** `nevermore` is hard-wired to the incumbent **by name** — `/opt/docker/compose/nevermore/.env` has `NEVERMORE_RERANK_MODEL=qwen3-reranker` (+ `NEVERMORE_EMBED_MODEL=qwen3-embedding`, which is fine, that's the live embedder) pointed at the gateway. The R43 cutover repointed the `reranker` alias but **deliberately left `qwen3-reranker` naming the Qwen model** (correct under the no-false-aliases rule) — so nevermore never moved. Brokkr R43 measured that model **harming 80/90 fleet queries** (no-reranker beat it 89/90 vs 56/90), so nevermore's twice-daily rerank pass is very likely degrading its own briefing.
|
||||
- **Fix is one line in nevermore's `.env`** (`NEVERMORE_RERANK_MODEL=reranker`) + a nevermore restart — NOT a gateway restart. Do that BEFORE retiring `:8002`, or nevermore's rerank pass breaks. **Then** `:8002` and `:8014` are both genuinely free to retire (~5.0 GB).
|
||||
- **🟢 RERANKER + NEVERMORE CLEANUP — RESOLVED 2026-08-20 (found while answering "why do we have 3 rerankers?").** The R43 cutover was only half-landed: it repointed the `reranker` alias but never moved the one consumer that mattered.
|
||||
- **★ THE BIG ONE — `nevermore`'s LLM summarization had been DEAD for 8 days and nothing noticed.** Its `.env` pinned `LLAMA_SWAP_MODEL=granite-4.1-8b`, an alias retired 2026-08-12 with the granite seat. Result: **67 consecutive failures, 0 tokens, status=failure**, twice daily, silently — the twice-daily briefing was rendering with no LLM pass at all. Nothing alerts on `status=failure` in the spend logs, so this was invisible until someone went looking for something else.
|
||||
- **Also found:** `NEVERMORE_RERANK_MODEL=qwen3-reranker` — nevermore was the ONLY caller of the retired Qwen incumbent (7 calls, 12-hourly at 03:00/15:00 UTC = its 08:00/20:00 PDT cron), the model Brokkr measured **harming 80/90 fleet queries**. Its rerank calls succeeded; they were just running through the bad model. Meanwhile the production `reranker` alias had **0 calls in 4 days**.
|
||||
- **★ THE RULE THIS EARNS: retiring a model is a TWO-SIDED operation.** Grep every consumer's config for the alias *before* deleting it. And consumers must pin stable **capability aliases** (`summarizer`, `reranker`) never model names (`granite-4.1-8b`, `qwen3-reranker`) — then the gateway can repoint without anyone editing a downstream `.env`. Both of nevermore's breakages are the same bug.
|
||||
- **FIXED:** nevermore `.env` → `LLAMA_SWAP_MODEL=summarizer` + `NEVERMORE_RERANK_MODEL=reranker` (backup `.env.bak-pre-model-repoint-20260820`), worker recreated, all three deps verified live — summarizer returns clean content with 0 reasoning chars at nevermore's exact call shape (`temperature 0.2`, `max_tokens 4000`), reranker scores 0.95 on-topic vs ~1e-5 off-topic, embedding returns dim-1024. ⚠ `NEVERMORE_EMBED_MODEL=qwen3-embedding` was already correct — left alone. ⚠ nevermore's `.env` is **server-only** (`.env` is excluded from the mirror both ways), so this fix is not in git.
|
||||
- **RETIRED:** `vllm-rerank` (:8002 Qwen incumbent) + its `qwen3-reranker` alias; `vllm-rerank-a4` (:8014) + its alias; `vllm-granite` (Exited 8 days, dead service block). **A3 PROMOTED** from a throwaway `docker run` into `stacks/vllm` as service `vllm-rerank-a3` (the ledger's own open follow-up) — healthy in 55s, image pinned. Container name **deliberately keeps the bake-off arm name** so the ledger/memory/R43 references stay valid.
|
||||
- **⚠️ CORRECTION — my "A4 has no alias at all" claim was WRONG.** `reranker-a4-gte-modernbert` **did** exist; I grepped `config.yaml` and concluded absence. **LiteLLM serves BOTH config-defined AND DB-defined models** — live was **32** models, `config.yaml` only **26**. The 6 DB-only ones: `ext-tts`, `gpt-4o-mini-tts`, `tts-1`, `tts-1-hd`, `reranker-a3-bge-v2-m3`, `reranker-a4-gte-modernbert`. **`/v1/models` (or `/model/info`, which flags `db_model: true`) is the ground truth — never `config.yaml` alone.** DB models delete hot via `POST /model/delete {"id": …}` with **no gateway restart**; config models need a file edit + `docker restart litellm` (~84s). A4's alias was deleted that way once its backend was gone.
|
||||
- **⏳ REMAINING:** `reranker-a3-bge-v2-m3` (DB-defined, id `1f08a73e-c09c-463c-8f18-5edea51fb736`) still exists as a **duplicate of `reranker` on the same backend**, 0 calls. It's Brokkr's cutover-verification handle (prod == arm at maxdiff 0.000000), so **not removed unilaterally** — it is another agent's tooling, and it is redundant rather than broken. Ask Brokkr before deleting.
|
||||
|
||||
- **EVIDENCE HOLD (partial):** WT #394 FILE half STILL STANDS — do NOT delete on-disk gen dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`); rex393-fiction-* + r42-gate-* KEEP.
|
||||
|
||||
@@ -172,9 +174,9 @@ _As of 2026-08-20 23:30 — **the Heretic-300 session** (see the 🔴 entry abov
|
||||
⚠ **Never set the ZFS cachefile on one pool.** The runbook's `zpool set cachefile=… nvme` was a trap: populating a cache flips the host from import-by-scan to import-by-cache, so a cache holding only `nvme` leaves `ssd`+`tank` unimported and empties every CT 103 export. Set on all three 2026-08-18, verified in the 11,976-byte cache.
|
||||
⚠ **Migrate FIRST, patch after** — a signed kernel would land in `/boot` on the 1.3 GB root. ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on this host and serves `hard` NFS to esh-docker-vm and esh-pve — quiesce both before any reboot** or you wedge esh-docker-vm into D-state. Off-box at `nh3-dev:~/backups/esh-pve-nas/`: DOM image `dom-sdq-20260818.img.zst` (2.38 GiB, crash-consistent), clean `bootchain-20260818.tar.gz`, config snapshot `…20260818T051*.tar.gz`. Runbook `docs/runbooks/esh-pve-nas-boot-migration.md`; detail → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`.
|
||||
|
||||
- **⚪ LFM2.5-2.6B RETIRED PERMANENTLY 2026-08-20 (operator directive).** `vllm-lfm25` (:8021, GPU1) removed: service deleted from `stacks/vllm/compose.yaml` + pushed live (backup `compose.yaml.bak-pre-lfm25-retire-20260820`), container `docker rm -f`'d, `lfm2.5-2.6b` alias deleted from the LiteLLM config (live + canonical; backup `config.yaml.bak-pre-lfm25-retire-20260820`, 28→27 models). **Freed 8,721 MiB on GPU1.** Justification: it was an EVAL-ONLY bake-off seat vs `granite-4.1-8b` (brokkr R-target 2026-08-10) that never received the operator ruling it was pending; the comparator was retired from the roster 2026-08-15; it was deliberately never in any default/fallback routing chain; and spend logs showed **0 calls in the 4-day window**. Weights remain in the shared HF cache — nothing deleted from disk. ⚠ **The gateway restart that makes the alias-deletion take effect was HELD** so it could batch with a reranker change — until `docker restart litellm` runs, `lfm2.5-2.6b` is still routable in-memory and will error against a dead backend. ⚠ `vllm-granite` is **still a defined service** in `stacks/vllm/compose.yaml` though the model was retired 2026-08-12 — dead config, 0 VRAM (stopped), worth the same cleanup pass.
|
||||
- **⚪ LFM2.5-2.6B RETIRED PERMANENTLY 2026-08-20 (operator directive).** `vllm-lfm25` (:8021, GPU1) removed: service deleted from `stacks/vllm/compose.yaml` + pushed live (backup `compose.yaml.bak-pre-lfm25-retire-20260820`), container `docker rm -f`'d, `lfm2.5-2.6b` alias deleted from the LiteLLM config (live + canonical; backup `config.yaml.bak-pre-lfm25-retire-20260820`, 28→27 models). **Freed 8,721 MiB on GPU1.** Justification: it was an EVAL-ONLY bake-off seat vs `granite-4.1-8b` (brokkr R-target 2026-08-10) that never received the operator ruling it was pending; the comparator was retired from the roster 2026-08-15; it was deliberately never in any default/fallback routing chain; and spend logs showed **0 calls in the 4-day window**. Weights remain in the shared HF cache — nothing deleted from disk. The held gateway restart fired at 23:46 alongside the `qwen3-reranker` removal (one ~84s blip covered both); `lfm2.5-2.6b`, `qwen3-reranker` and `granite-4.1-8b` all now 400 cleanly. `vllm-granite`'s dead service block was removed in the same pass.
|
||||
|
||||
- **OPEN FOLLOW-UPS (parked):** repoint `nevermore` off the retired reranker (see the 🔴 reranker audit) then retire `:8002`/`:8014`; delete the dead `vllm-granite` service block; move gen seat off pinned-nightly to stable once #51113 ships; Lobe one-time TTS UI pass; delete the 1.8GB litellm dump; `harden-esh-docker-vm` (park id 28, PROMOTED — Tier-1 done, `/mnt/books` stays hard w/ watchdog); chatterbox-fast build-context divergence; #363 research-wing ingest (no deadline); optionally attach our MTP reproducer to vllm#47087 (needs a GitHub identity — operator's call).
|
||||
- **OPEN FOLLOW-UPS (parked):** ask Brokkr whether the duplicate `reranker-a3-bge-v2-m3` alias can go; move gen seat off pinned-nightly to stable once #51113 ships; Lobe one-time TTS UI pass; delete the 1.8GB litellm dump; `harden-esh-docker-vm` (park id 28, PROMOTED — Tier-1 done, `/mnt/books` stays hard w/ watchdog); chatterbox-fast build-context divergence; #363 research-wing ingest (no deadline); optionally attach our MTP reproducer to vllm#47087 (needs a GitHub identity — operator's call).
|
||||
|
||||
- **althing monitor** ARMED (handle `infra-ops`). ⚠️ Re-arm ONLY after a real FIRE (rc0), never after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, never chained with `&` (orphans it — hit this twice 2026-08-17, `stop-monitor` reclaims).
|
||||
|
||||
|
||||
@@ -8,7 +8,8 @@
|
||||
# Model-name → upstream mapping:
|
||||
# phi4-mini → vLLM :8004 (generative chat)
|
||||
# qwen3-embedding → vLLM :8001 (/v1/embeddings)
|
||||
# qwen3-reranker → vLLM :8002 (/rerank)
|
||||
# reranker → vLLM :8013 (/rerank; bge-v2-m3. The old qwen3-reranker
|
||||
# alias on :8002 was retired 2026-08-20.)
|
||||
# * (wildcard) → llama-swap :9292 (the swappable generative zoo)
|
||||
#
|
||||
# The wildcard fronts llama-swap so its whole model zoo logs through the
|
||||
@@ -313,14 +314,14 @@ model_list:
|
||||
model_info:
|
||||
mode: embedding
|
||||
|
||||
# --- Qwen3 reranker (proxy /rerank route) ---
|
||||
- model_name: qwen3-reranker
|
||||
litellm_params:
|
||||
model: hosted_vllm/Qwen/Qwen3-Reranker-0.6B
|
||||
api_base: http://10.250.50.54:8002/v1
|
||||
api_key: os.environ/VLLM_API_KEY
|
||||
model_info:
|
||||
mode: rerank
|
||||
# --- qwen3-reranker RETIRED 2026-08-20. It named Qwen3-Reranker-0.6B on :8002,
|
||||
# the incumbent the R43 bake-off replaced on 2026-08-06 after measuring it
|
||||
# HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). The alias
|
||||
# was kept as the rollback path and, for 13 days, was the ONLY reranker
|
||||
# actually receiving traffic: nevermore was pinned to it by name, so the
|
||||
# cutover moved `reranker` but never moved nevermore. Fixed at the consumer
|
||||
# (nevermore now pins `reranker`), then the seat and this alias were retired.
|
||||
# Use `reranker` -> bge-reranker-v2-m3 :8013. ---
|
||||
|
||||
# --- Worldtree capability aliases (role→capability gateway swaps, ADR-0012).
|
||||
# Stable role-named aliases so consumers bind the CAPABILITY, not a concrete
|
||||
@@ -337,7 +338,8 @@ model_list:
|
||||
top_p: 0.9
|
||||
model_info:
|
||||
mode: chat
|
||||
# reranker → generic capability name for rerank (currently qwen3-reranker).
|
||||
# reranker → generic capability name for rerank. THE reranker alias — the only
|
||||
# one left as of 2026-08-20. Every consumer pins this name, never a model name.
|
||||
- model_name: reranker
|
||||
litellm_params:
|
||||
model: hosted_vllm/BAAI/bge-reranker-v2-m3
|
||||
|
||||
+25
-74
@@ -4,12 +4,19 @@
|
||||
# # edit .env with real values
|
||||
# docker compose up -d
|
||||
|
||||
# Image version — pin for reproducibility (`latest` for edge)
|
||||
VLLM_VERSION=latest
|
||||
# Image version — PINNED. Do not put `latest` here: every service in this file
|
||||
# shares this one variable, so a bare `docker compose up -d` would silently
|
||||
# upgrade the whole stack's vLLM at once. Pinned 2026-08-20 to the version all
|
||||
# seats were already running (`latest` and `v0.24.0` were the same local image,
|
||||
# 4091d5593f77, so the pin changed nothing at runtime). Bump deliberately.
|
||||
VLLM_VERSION=v0.24.0
|
||||
|
||||
# Host ports (container always listens on 8000 internally)
|
||||
EMBED_PORT=8001
|
||||
RERANK_PORT=8002
|
||||
# 8013 — the reranker moved here 2026-08-20 when bge-v2-m3 (the R43 winner, which
|
||||
# had been running as a throwaway `docker run` on this port) was promoted into
|
||||
# this stack and the Qwen incumbent on :8002 was retired.
|
||||
RERANK_PORT=8013
|
||||
REWARD_PORT=8003
|
||||
|
||||
# GPU assignment — all services share this GPU
|
||||
@@ -18,7 +25,10 @@ GPU_ID=1
|
||||
|
||||
# Models — reference by full repo name in API requests
|
||||
EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B
|
||||
RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B
|
||||
# bge-reranker-v2-m3 — the R43 bake-off winner, replacing Qwen3-Reranker-0.6B
|
||||
# (measured HARMING 80/90 fleet queries). Multilingual cross-encoder; needs no
|
||||
# --hf-overrides, unlike the Qwen reranker it displaced.
|
||||
RERANK_MODEL=BAAI/bge-reranker-v2-m3
|
||||
# Skywork is a local-path AWQ output, not from HF Hub. Bind-mounted into the
|
||||
# reward container at /local-models — see compose.yaml. No env var here for
|
||||
# the model path itself since it's hard-coded in the compose command.
|
||||
@@ -72,62 +82,12 @@ API_KEY=
|
||||
# services (embed/rerank). Reward is local-path, ignores this.
|
||||
HF_TOKEN=
|
||||
|
||||
# === granite-4.1-8b (production summarizer / dreaming agent) ===
|
||||
# Replaced phi4-mini 2026-06-05 (Granite 4.1 8B beat phi4 on precision in
|
||||
# brokkr's R15 P03 model-fitness eval). Same GPU-1 slot, reusing phi4's port.
|
||||
GRANITE_PORT=8004
|
||||
# GPU 1 — co-located with the embed/rerank/reward trio. With phi4 retired, GPU 1
|
||||
# has ~19 GB free; granite at 56K + FP8 KV needs ~15-16 GB → ~2-3 GB margin.
|
||||
# NOTE: llama-swap also uses GPU 1 dynamically — a large swap-in could contend;
|
||||
# pin llama-swap to GPU 0 for clean separation (follow-up).
|
||||
GRANITE_GPU_ID=1
|
||||
# Official IBM pre-quantized FP8 (compressed-tensors) — calibrated, ~9.6 GB,
|
||||
# loaded directly (FP8 native on Blackwell cc 12.0). Fallback to vLLM-native dynamic FP8
|
||||
# from BF16: GRANITE_MODEL=ibm-granite/granite-4.1-8b + GRANITE_QUANT=fp8.
|
||||
GRANITE_MODEL=ibm-granite/granite-4.1-8b-fp8
|
||||
GRANITE_QUANT=compressed-tensors
|
||||
GRANITE_SERVED_NAME=granite-4.1-8b
|
||||
# 131072 ctx — MAXED 2026-06-13 (was 51200/50K). GPU-1 rebalance: granite shares
|
||||
# the card with the trio + qwen35-vl (vision). Qwen was over-provisioned on KV
|
||||
# (20x conc @ 32k), so trimming it freed room for granite's ~305k-token pool here.
|
||||
# PagedAttention allocates KV per ACTUAL token, so 131072 is only a CEILING — a 1k
|
||||
# summarize turn uses ~1k tokens, so the pool holds ~300 concurrently; the "2.33x"
|
||||
# headline is worst-case (every request maxing 131k). Granite 4.1 supports 131072.
|
||||
# 65536 — REDUCED 2026-06-14 (was 131072) to free GPU-1 room for the FP8 vision
|
||||
# model (Qwen3.6-35B-A3B, stacks/qwen36-vl, ~34 GB weights). Summarizer load is
|
||||
# short parallel calls, so the 64K cap is ample.
|
||||
# 131072 — RESTORED 2026-07-16 (native max) for full-chapter summarization; GPU-1
|
||||
# freed by the image-bench evict + qwen36-vl gone, so the 128K ctx fits again.
|
||||
# 16384 — SHRUNK 2026-07-27 (operator: granite is being phased out) to free GPU-1
|
||||
# room for vllm-coder (the Zed FIM seat). Full-chapter ctx dropped; util 0.13 holds
|
||||
# 16K cleanly (32768 @ util 0.12 crash-looped: KV est-max was only 19376 tokens).
|
||||
GRANITE_MAX_MODEL_LEN=16384
|
||||
# FP8 KV cache (native on Blackwell cc 12.0). At 50K ≈ ~4.2 GB (vs ~8.4 GB at fp16).
|
||||
GRANITE_KV_CACHE_DTYPE=fp8
|
||||
# util 0.35 (~33.6 GB) — tuned 2026-06-13 to leave ~3.5 GB free on GPU 1 alongside
|
||||
# the trio + qwen co-tenants. On this shared card vLLM needs free >= util*total at
|
||||
# startup, and START ORDER matters: trim qwen FIRST, then grow granite, else granite
|
||||
# OOMs against the full card. (0.37 overshot to 1.7 GB free; 0.35 lands ~3.7 GB.)
|
||||
# 0.24 — REBALANCED 2026-06-14 (was 0.35) for the FP8 vision cutover. GPU-1 budget:
|
||||
# qwen36-vl 0.46 + granite 0.24 + reward 0.10 + embed/rerank 0.03 ≈ 0.90 total,
|
||||
# ~7.5 GB headroom (the OOM buffer; held under 20-concurrent load test). granite
|
||||
# gets a 169K-token KV pool = 2.58x concurrency @ 64K. Bring qwen36 up LAST.
|
||||
# 0.18 — RIGHT-SIZED 2026-07-16 (was 0.34 live; qwen36-vl no longer a GPU-1 tenant,
|
||||
# image-bench evicted) to free ~10.5 GB for relocating a GPU0 model onto GPU1. KV
|
||||
# 6.45 GiB = 84,528 tokens = 1.29x concurrency @ 65536 (summarizer = short parallel
|
||||
# calls; ample). Effective slope on this shared card ≈ 950 MiB KV per 0.01 util, and
|
||||
# KV must hold >= 1x max-model-len — util 0.15 undershot (crash: est max-len 47184 <
|
||||
# 65536), 0.18 lands the target cleanly.
|
||||
# 0.27 — RE-GROWN 2026-07-16 (same session) after char-rp moved onto GPU-1: spend the
|
||||
# leftover room on full-chapter context (max-len 131072). KV 15.0 GiB = 196,560 tokens
|
||||
# = 1.50x @ 131072; GPU-1 lands ~6.7 GB headroom (char-rp 30 + granite 27 + selene 17 + trio).
|
||||
# 0.13 — SHRUNK 2026-07-27 (was 0.27) with the max-len drop; frees ~14 GB on GPU-1 for
|
||||
# vllm-coder. granite phasing out, so no longer worth the big KV pool. (0.12 was too
|
||||
# small for even 16K KV; 0.13 gives ~1.4x @ 16384.)
|
||||
GRANITE_GPU_MEM_UTIL=0.13
|
||||
# Concurrency cap. Was 1024 (fan-out summarizer); DROPPED to 256 on 2026-07-27 with the
|
||||
# phasing-out shrink (256 is ample for the reduced summarizer load; smaller sched state).
|
||||
GRANITE_MAX_NUM_SEQS=256
|
||||
# === granite-4.1-8b — RETIRED 2026-08-12, vars removed 2026-08-20 ===
|
||||
# Was the production summarizer. The `summarizer` / `classifier` gateway aliases
|
||||
# were repointed at the gen seat and the container stopped; the service block and
|
||||
# these tunables are now gone. See the tombstone in compose.yaml for the lesson
|
||||
# this cost us (nevermore stayed pinned to the dead `granite-4.1-8b` alias and
|
||||
# failed silently for 8 days).
|
||||
|
||||
# Qwen2.5-Coder-1.5B (BASE) — FIM code-completion seat for Zed editor inline
|
||||
# edit-predictions (deep-research pick 2026-07-27, Apache-2.0). GPU-1, alongside the
|
||||
@@ -143,17 +103,8 @@ CODER_KV_CACHE_DTYPE=fp8
|
||||
CODER_GPU_MEM_UTIL=0.06
|
||||
CODER_MAX_NUM_SEQS=32
|
||||
|
||||
# LFM2.5-2.6B — NON-PRODUCTION bake-off alias vs granite (brokkr R-target 2026-08-10).
|
||||
# LiquidAI LFM Open License v1.0 (<USD 10M-rev commercial, not OSI) — eval-only pending
|
||||
# an operator production ruling. Reasoning model (emits <think>); served raw (no vLLM
|
||||
# reasoning-parser) so content is non-empty. util 0.09 (~8.6GB) fits GPU1's free slack
|
||||
# without touching production reservations; max-len 16384 keeps KV small (bake-off
|
||||
# doesn't need the 131k ceiling). Vendor sampling lives in the LiteLLM alias.
|
||||
LFM25_PORT=8021
|
||||
LFM25_GPU_ID=1
|
||||
LFM25_MODEL=LiquidAI/LFM2.5-2.6B
|
||||
LFM25_SERVED_NAME=lfm2.5-2.6b
|
||||
LFM25_MAX_MODEL_LEN=16384
|
||||
LFM25_KV_CACHE_DTYPE=auto
|
||||
LFM25_GPU_MEM_UTIL=0.09
|
||||
LFM25_MAX_NUM_SEQS=8
|
||||
# === LFM2.5-2.6B — RETIRED PERMANENTLY 2026-08-20 (operator directive) ===
|
||||
# Eval-only bake-off seat vs granite-4.1-8b that never got its production ruling;
|
||||
# its comparator was retired first, and it logged 0 calls in the 4 days before it
|
||||
# came down. Container removed, service block and tunables deleted, gateway alias
|
||||
# dropped. Weights remain in the shared HF cache.
|
||||
|
||||
+43
-86
@@ -83,9 +83,27 @@ services:
|
||||
- homepage.description=Qwen3 Embedding via vLLM (ana-ml2)
|
||||
- homepage.href=http://10.250.50.54:${EMBED_PORT}/docs
|
||||
|
||||
vllm-rerank:
|
||||
# THE fleet reranker. Backs the LiteLLM `reranker` alias, which is what every
|
||||
# consumer should name — never the model, never a bake-off arm name.
|
||||
#
|
||||
# It won the Brokkr R43 bake-off (docs/pfi/reranker-selection-ledger.md) and
|
||||
# replaced Qwen3-Reranker-0.6B, which was measured HARMING 80/90 fleet queries
|
||||
# (no-reranker beat it 89/90 vs 56/90). The R42 v13 acceptance gate went
|
||||
# 56/90 -> 90/90 on the cutover, its first-ever PASS.
|
||||
#
|
||||
# Promoted from a throwaway `docker run` to this service 2026-08-20 (the
|
||||
# ledger's own open follow-up). It carried the bake-off's arm name from the
|
||||
# start and KEEPS it: the ledger, persistent-memory and the R43 record all say
|
||||
# `vllm-rerank-a3`, and renaming for tidiness would orphan every one of those
|
||||
# references. The name carries its provenance.
|
||||
#
|
||||
# Retired alongside this promotion: `vllm-rerank` (Qwen3-Reranker-0.6B, :8002 —
|
||||
# the rollback path, kept warm 13 days) and `vllm-rerank-a4`
|
||||
# (gte-reranker-modernbert, :8014 — a documented throughput fallback that was
|
||||
# never given a gateway alias, so it was unreachable the whole time).
|
||||
vllm-rerank-a3:
|
||||
image: vllm/vllm-openai:${VLLM_VERSION}
|
||||
container_name: vllm-rerank
|
||||
container_name: vllm-rerank-a3
|
||||
restart: unless-stopped
|
||||
ipc: host
|
||||
ports:
|
||||
@@ -103,8 +121,9 @@ services:
|
||||
- ${RERANK_MODEL}
|
||||
- --runner
|
||||
- pooling
|
||||
- --hf-overrides
|
||||
- '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}'
|
||||
# No --hf-overrides here. The Qwen reranker needed one to be coerced into a
|
||||
# sequence-classification head; bge-reranker-v2-m3 is natively a
|
||||
# cross-encoder and vLLM resolves it directly.
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
@@ -134,9 +153,9 @@ services:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI - Eval & Retrieval
|
||||
- homepage.name=vLLM Rerank (Qwen3)
|
||||
- homepage.name=vLLM Rerank (bge-v2-m3)
|
||||
- homepage.icon=mdi-sort-variant
|
||||
- homepage.description=Qwen3 Reranker via vLLM (ana-ml2)
|
||||
- homepage.description=BAAI bge-reranker-v2-m3 — the fleet reranker, backs the `reranker` alias (ana-ml2)
|
||||
- homepage.href=http://10.250.50.54:${RERANK_PORT}/docs
|
||||
|
||||
vllm-reward:
|
||||
@@ -197,86 +216,24 @@ services:
|
||||
- homepage.description=Skywork-Reward-V2 8B classifier via vLLM (ana-ml2)
|
||||
- homepage.href=http://10.250.50.54:${REWARD_PORT}/docs
|
||||
|
||||
# Granite 4.1 8B (FP8) — production summarizer (replaced phi4-mini
|
||||
# 2026-06-05, which had superseded the llama-swap granite-4-small pin).
|
||||
# Generative chat model (OpenAI /v1/chat/completions), so NO --runner
|
||||
# pooling. FP8 on RTX PRO 6000 Blackwell (cc 12.0): near-lossless, ~1.2x, ~6 GB.
|
||||
vllm-granite:
|
||||
image: vllm/vllm-openai:${VLLM_VERSION}
|
||||
container_name: vllm-granite
|
||||
restart: unless-stopped
|
||||
ipc: host
|
||||
ports:
|
||||
- "${GRANITE_PORT}:8000"
|
||||
volumes:
|
||||
- /tank/aimodels/huggingface:/hfcache
|
||||
environment:
|
||||
- HF_HOME=/hfcache
|
||||
- HF_HUB_CACHE=/hfcache/hub
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||||
- VLLM_API_KEY=${API_KEY:-}
|
||||
command:
|
||||
# Production summarizer (replaced phi4-mini 2026-06-05). Default = official
|
||||
# IBM pre-quantized FP8 (compressed-tensors), loaded directly; FP8 is native
|
||||
# on the RTX PRO 6000 Blackwell (cc 12.0). Fallback to vLLM-native dynamic FP8 from
|
||||
# BF16: GRANITE_MODEL=ibm-granite/granite-4.1-8b + GRANITE_QUANT=fp8.
|
||||
- ${GRANITE_MODEL}
|
||||
- --served-model-name
|
||||
- ${GRANITE_SERVED_NAME}
|
||||
- --quantization
|
||||
- ${GRANITE_QUANT}
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- "8000"
|
||||
- --gpu-memory-utilization
|
||||
- ${GRANITE_GPU_MEM_UTIL}
|
||||
- --max-model-len
|
||||
- ${GRANITE_MAX_MODEL_LEN}
|
||||
# Very high so the KV pool (not the seq cap) is the only concurrency bound —
|
||||
# granite is the fleet fan-out summarizer/classifier (many concurrent SHORT
|
||||
# calls). vLLM's default resolves to 128, capping below the KV bound
|
||||
# (~192 @ 1K-tok); 1024 unblocks it (VRAM-neutral — KV pool is util-bound).
|
||||
- --max-num-seqs
|
||||
- ${GRANITE_MAX_NUM_SEQS}
|
||||
- --dtype
|
||||
- auto
|
||||
# CUDA graphs ENABLED (no --enforce-eager) for decode throughput. Made
|
||||
# room 2026-06-05 by right-sizing the embed/rerank/reward trio's KV pools
|
||||
# (they were over-provisioned at 5.9x/2.0x/3.9x concurrency); GPU 1 now has
|
||||
# ~17 GB free after granite, so graph-capture buffers fit. If the trio
|
||||
# ever grows back, granite may need --enforce-eager again on this card.
|
||||
# FP8 KV cache — halves KV memory; near-lossless on Blackwell (cc 12.0).
|
||||
- --kv-cache-dtype
|
||||
- ${GRANITE_KV_CACHE_DTYPE}
|
||||
# Prefix caching pinned EXPLICIT (vLLM v1 defaults it on, but pin so a
|
||||
# version flip can't silently disable it). Benched 2026-06-13: ~6.5x faster
|
||||
# TTFT (45ms vs 292ms) on a shared ~4.5k-token summarizer template; soft/
|
||||
# evictable KV, neutral when prefixes don't repeat — pure win for granite.
|
||||
- --enable-prefix-caching
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids:
|
||||
- "${GRANITE_GPU_ID}"
|
||||
capabilities:
|
||||
- gpu
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 180s
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI - Inference
|
||||
- homepage.name=vLLM Granite 4.1 8B (summarizer)
|
||||
- homepage.icon=mdi-text-box-outline
|
||||
- homepage.description=Granite 4.1 8B FP8 via vLLM (ana-ml2)
|
||||
- homepage.href=http://10.250.50.54:${GRANITE_PORT}/docs
|
||||
# vllm-granite (ibm-granite/granite-4.1-8b-fp8, :8004) — RETIRED 2026-08-12,
|
||||
# service block removed 2026-08-20. It was the fleet summarizer until the
|
||||
# `summarizer` and `classifier` aliases were repointed at the gen seat; the
|
||||
# container was stopped then and sat Exited for 8 days while this block still
|
||||
# claimed it as production.
|
||||
#
|
||||
# ⚠️ Retiring the SEAT did not retire its CONSUMERS, and nobody checked. The
|
||||
# `granite-4.1-8b` gateway alias was deleted at the same time, but `nevermore`
|
||||
# was still pinned to that alias by name — so its LLM summarization pass failed
|
||||
# 67 consecutive times over 8 days, 0 tokens, entirely silently, because a
|
||||
# failed gateway call still returns 200-shaped spend-log rows and nothing
|
||||
# alerts on status=failure. Found 2026-08-20 only because someone asked an
|
||||
# unrelated question about reranker VRAM.
|
||||
#
|
||||
# The rule this earns: RETIRING A MODEL IS A TWO-SIDED OPERATION. Grep every
|
||||
# consumer's config for the alias BEFORE deleting it, and prefer that consumers
|
||||
# pin stable ALIASES (`summarizer`) over model names (`granite-4.1-8b`) so the
|
||||
# gateway can repoint them without anyone editing a downstream .env.
|
||||
|
||||
vllm-coder:
|
||||
image: vllm/vllm-openai:${VLLM_VERSION}
|
||||
|
||||
Reference in New Issue
Block a user