diff --git a/persistent-memory.md b/persistent-memory.md index f02b402..a6ecfe9 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -134,16 +134,18 @@ _As of 2026-08-20 23:30 β€” **the Heretic-300 session** (see the πŸ”΄ entry abov - **🟒 LITELLM β€” upgraded v1.91.0β†’v1.97.0, spend-log DB purged 6GBβ†’16MB + CAPPED (2026-08-17).** `store_prompts_in_spend_logs:false` + `maximum_spend_logs_retention_period:7d`. ⚠ **1.8GB pre-upgrade pg_dump still on ana-docker `/opt/docker/compose/litellm/` β€” deletable now the upgrade is proven** (operator was going to call it). Commit `01b5ad9`. -- **⚠️ GPU zero-sum (both cards ~94–95/97.9 GB).** GPU0: gen + meromero. GPU1: fablefusion + utility cluster. Any util bump on either seat of a shared card must be checked against the co-tenant (starved meromero into a crash-loop once at 0.45). **⚠️ BOOT ORDER IS PART OF THE STATE (2026-08-20).** `--gpu-memory-utilization` sets the target as a fraction of **TOTAL** VRAM, but vLLM **refuses to start unless that whole target is FREE right now** β€” so at ~96.4/97.9 GB the GPU0 pair coexists *only* in the order it was originally brought up. **Restore/reboot order: `vllm-meromero-rp` to `healthy` FIRST, then `vllm-gen`** β€” meromero (0.52 = 49.38 GiB) is the one that cannot fit in the remainder. "First" means **observed healthy**, not a `sleep`: a 10s gap against a 2–3 min weight load cost a 7-restart crash-loop. Verify a restore against **KV-pool size** (`GPU KV cache size` / `Maximum concurrency` in the container log), not `nvidia-smi` used-MiB β€” the latter swings ~7 GB on allocator slack with identical serving capacity. Baselines: gen β‰ˆ14.36 GiB / 403k tok / 1.54Γ— (h300 build: 401,550 tok / 1.53Γ—); meromero 542,202 tok. **πŸ“Š MEASURED VRAM CENSUS 2026-08-20 23:20** (nvidia-smi PIDβ†’container, not util-fraction guesses) β€” **GPU0 92,572/97,887 MiB (94.6%), 5.2 GB free:** meromero 50,072 + gen 42,500. **GPU1 86,667/97,887 MiB (88.5%), 11.0 GB free** *(after the lfm25 retirement below; was 95,388/2.4 GB)*: fablefusion-probe **43,452** + selene 16,870 + reward 9,512 + coder 6,158 + rerank 3,586 + embed 3,304 + rerank-a3 2,314 + rerank-a4 1,420. **Still no room for a ~22 GB PPL probe seat on either card without stopping something.** ⚠ **fablefusion is the single biggest reclaimable block (43.4 GB) and is nearly idle** β€” LiteLLM spend logs show `char-rp-probe` at **4 calls, last 2026-08-19 08:52**, vs `char-rp` (meromero) at 129 calls, last 2026-08-20 15:28. +- **⚠️ GPU zero-sum (both cards ~94–95/97.9 GB).** GPU0: gen + meromero. GPU1: fablefusion + utility cluster. Any util bump on either seat of a shared card must be checked against the co-tenant (starved meromero into a crash-loop once at 0.45). **⚠️ BOOT ORDER IS PART OF THE STATE (2026-08-20).** `--gpu-memory-utilization` sets the target as a fraction of **TOTAL** VRAM, but vLLM **refuses to start unless that whole target is FREE right now** β€” so at ~96.4/97.9 GB the GPU0 pair coexists *only* in the order it was originally brought up. **Restore/reboot order: `vllm-meromero-rp` to `healthy` FIRST, then `vllm-gen`** β€” meromero (0.52 = 49.38 GiB) is the one that cannot fit in the remainder. "First" means **observed healthy**, not a `sleep`: a 10s gap against a 2–3 min weight load cost a 7-restart crash-loop. Verify a restore against **KV-pool size** (`GPU KV cache size` / `Maximum concurrency` in the container log), not `nvidia-smi` used-MiB β€” the latter swings ~7 GB on allocator slack with identical serving capacity. Baselines: gen β‰ˆ14.36 GiB / 403k tok / 1.54Γ— (h300 build: 401,550 tok / 1.53Γ—); meromero 542,202 tok. **πŸ“Š MEASURED VRAM CENSUS 2026-08-20 23:20** (nvidia-smi PIDβ†’container, not util-fraction guesses) β€” **GPU0 92,572/97,887 MiB (94.6%), 5.2 GB free:** meromero 50,072 + gen 42,500. **GPU1 81,448/97,887 MiB (83.2%), 16.1 GB free** *(after the lfm25 + reranker cleanups below; was 95,388 / 2.4 GB free β€” **13.9 GB reclaimed in one night**)*: fablefusion-probe **43,452** + selene 16,870 + reward 9,512 + coder 6,158 + embed 3,304 + rerank-a3 2,112. **Still no room for a ~22 GB PPL probe seat β€” fablefusion is the only remaining block big enough.** ⚠ **fablefusion is the single biggest reclaimable block (43.4 GB) and is nearly idle** β€” LiteLLM spend logs show `char-rp-probe` at **4 calls, last 2026-08-19 08:52**, vs `char-rp` (meromero) at 129 calls, last 2026-08-20 15:28. - **FLEET RERANKER** = A3 (bge-reranker-v2-m3) PROD ana-ml2 GPU1 :8013. Passive watch; levers = A4 :8014 / util / 2nd replica; incumbent :8002 warm. `docs/pfi/reranker-selection-ledger.md`. -- **πŸ”΄ RERANKER FLEET AUDIT 2026-08-20 β€” the R43 cutover is only half-landed; `nevermore` is still running on the reranker Brokkr measured as HARMFUL.** Three reranker seats are up, and the usage is backwards from the design: - - `:8013` **A3 bge-reranker-v2-m3 β€” PRODUCTION**, backs the `reranker` alias. **0 calls** in the 4-day spend window to 2026-08-21. 2,314 MiB. - - `:8002` **Qwen3-Reranker-0.6B β€” the RETIRED incumbent**, kept warm only as a rollback path. **STILL SERVING LIVE TRAFFIC: 7 calls, 12-hourly at 03:00/15:00**, most recently 2026-08-21T03:00:31, all from api_key `dc0af5ba431b…`. 3,586 MiB. - - `:8014` **A4 gte-reranker-modernbert β€” "throughput fallback"**, but it has **NO LiteLLM alias at all** (config has only `qwen3-reranker` and `reranker`; the `reranker-a4-gte-modernbert` name in global `CLAUDE.md` is **STALE β€” it does not exist**). Unreachable through the gateway; 1,420 MiB serving nobody. - - **Root cause of the live-traffic finding:** `nevermore` is hard-wired to the incumbent **by name** β€” `/opt/docker/compose/nevermore/.env` has `NEVERMORE_RERANK_MODEL=qwen3-reranker` (+ `NEVERMORE_EMBED_MODEL=qwen3-embedding`, which is fine, that's the live embedder) pointed at the gateway. The R43 cutover repointed the `reranker` alias but **deliberately left `qwen3-reranker` naming the Qwen model** (correct under the no-false-aliases rule) β€” so nevermore never moved. Brokkr R43 measured that model **harming 80/90 fleet queries** (no-reranker beat it 89/90 vs 56/90), so nevermore's twice-daily rerank pass is very likely degrading its own briefing. - - **Fix is one line in nevermore's `.env`** (`NEVERMORE_RERANK_MODEL=reranker`) + a nevermore restart β€” NOT a gateway restart. Do that BEFORE retiring `:8002`, or nevermore's rerank pass breaks. **Then** `:8002` and `:8014` are both genuinely free to retire (~5.0 GB). +- **🟒 RERANKER + NEVERMORE CLEANUP β€” RESOLVED 2026-08-20 (found while answering "why do we have 3 rerankers?").** The R43 cutover was only half-landed: it repointed the `reranker` alias but never moved the one consumer that mattered. + - **β˜… THE BIG ONE β€” `nevermore`'s LLM summarization had been DEAD for 8 days and nothing noticed.** Its `.env` pinned `LLAMA_SWAP_MODEL=granite-4.1-8b`, an alias retired 2026-08-12 with the granite seat. Result: **67 consecutive failures, 0 tokens, status=failure**, twice daily, silently β€” the twice-daily briefing was rendering with no LLM pass at all. Nothing alerts on `status=failure` in the spend logs, so this was invisible until someone went looking for something else. + - **Also found:** `NEVERMORE_RERANK_MODEL=qwen3-reranker` β€” nevermore was the ONLY caller of the retired Qwen incumbent (7 calls, 12-hourly at 03:00/15:00 UTC = its 08:00/20:00 PDT cron), the model Brokkr measured **harming 80/90 fleet queries**. Its rerank calls succeeded; they were just running through the bad model. Meanwhile the production `reranker` alias had **0 calls in 4 days**. + - **β˜… THE RULE THIS EARNS: retiring a model is a TWO-SIDED operation.** Grep every consumer's config for the alias *before* deleting it. And consumers must pin stable **capability aliases** (`summarizer`, `reranker`) never model names (`granite-4.1-8b`, `qwen3-reranker`) β€” then the gateway can repoint without anyone editing a downstream `.env`. Both of nevermore's breakages are the same bug. + - **FIXED:** nevermore `.env` β†’ `LLAMA_SWAP_MODEL=summarizer` + `NEVERMORE_RERANK_MODEL=reranker` (backup `.env.bak-pre-model-repoint-20260820`), worker recreated, all three deps verified live β€” summarizer returns clean content with 0 reasoning chars at nevermore's exact call shape (`temperature 0.2`, `max_tokens 4000`), reranker scores 0.95 on-topic vs ~1e-5 off-topic, embedding returns dim-1024. ⚠ `NEVERMORE_EMBED_MODEL=qwen3-embedding` was already correct β€” left alone. ⚠ nevermore's `.env` is **server-only** (`.env` is excluded from the mirror both ways), so this fix is not in git. + - **RETIRED:** `vllm-rerank` (:8002 Qwen incumbent) + its `qwen3-reranker` alias; `vllm-rerank-a4` (:8014) + its alias; `vllm-granite` (Exited 8 days, dead service block). **A3 PROMOTED** from a throwaway `docker run` into `stacks/vllm` as service `vllm-rerank-a3` (the ledger's own open follow-up) β€” healthy in 55s, image pinned. Container name **deliberately keeps the bake-off arm name** so the ledger/memory/R43 references stay valid. + - **⚠️ CORRECTION β€” my "A4 has no alias at all" claim was WRONG.** `reranker-a4-gte-modernbert` **did** exist; I grepped `config.yaml` and concluded absence. **LiteLLM serves BOTH config-defined AND DB-defined models** β€” live was **32** models, `config.yaml` only **26**. The 6 DB-only ones: `ext-tts`, `gpt-4o-mini-tts`, `tts-1`, `tts-1-hd`, `reranker-a3-bge-v2-m3`, `reranker-a4-gte-modernbert`. **`/v1/models` (or `/model/info`, which flags `db_model: true`) is the ground truth β€” never `config.yaml` alone.** DB models delete hot via `POST /model/delete {"id": …}` with **no gateway restart**; config models need a file edit + `docker restart litellm` (~84s). A4's alias was deleted that way once its backend was gone. + - **⏳ REMAINING:** `reranker-a3-bge-v2-m3` (DB-defined, id `1f08a73e-c09c-463c-8f18-5edea51fb736`) still exists as a **duplicate of `reranker` on the same backend**, 0 calls. It's Brokkr's cutover-verification handle (prod == arm at maxdiff 0.000000), so **not removed unilaterally** β€” it is another agent's tooling, and it is redundant rather than broken. Ask Brokkr before deleting. - **EVIDENCE HOLD (partial):** WT #394 FILE half STILL STANDS β€” do NOT delete on-disk gen dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`); rex393-fiction-* + r42-gate-* KEEP. @@ -172,9 +174,9 @@ _As of 2026-08-20 23:30 β€” **the Heretic-300 session** (see the πŸ”΄ entry abov ⚠ **Never set the ZFS cachefile on one pool.** The runbook's `zpool set cachefile=… nvme` was a trap: populating a cache flips the host from import-by-scan to import-by-cache, so a cache holding only `nvme` leaves `ssd`+`tank` unimported and empties every CT 103 export. Set on all three 2026-08-18, verified in the 11,976-byte cache. ⚠ **Migrate FIRST, patch after** β€” a signed kernel would land in `/boot` on the 1.3 GB root. ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on this host and serves `hard` NFS to esh-docker-vm and esh-pve β€” quiesce both before any reboot** or you wedge esh-docker-vm into D-state. Off-box at `nh3-dev:~/backups/esh-pve-nas/`: DOM image `dom-sdq-20260818.img.zst` (2.38 GiB, crash-consistent), clean `bootchain-20260818.tar.gz`, config snapshot `…20260818T051*.tar.gz`. Runbook `docs/runbooks/esh-pve-nas-boot-migration.md`; detail β†’ `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`. -- **βšͺ LFM2.5-2.6B RETIRED PERMANENTLY 2026-08-20 (operator directive).** `vllm-lfm25` (:8021, GPU1) removed: service deleted from `stacks/vllm/compose.yaml` + pushed live (backup `compose.yaml.bak-pre-lfm25-retire-20260820`), container `docker rm -f`'d, `lfm2.5-2.6b` alias deleted from the LiteLLM config (live + canonical; backup `config.yaml.bak-pre-lfm25-retire-20260820`, 28β†’27 models). **Freed 8,721 MiB on GPU1.** Justification: it was an EVAL-ONLY bake-off seat vs `granite-4.1-8b` (brokkr R-target 2026-08-10) that never received the operator ruling it was pending; the comparator was retired from the roster 2026-08-15; it was deliberately never in any default/fallback routing chain; and spend logs showed **0 calls in the 4-day window**. Weights remain in the shared HF cache β€” nothing deleted from disk. ⚠ **The gateway restart that makes the alias-deletion take effect was HELD** so it could batch with a reranker change β€” until `docker restart litellm` runs, `lfm2.5-2.6b` is still routable in-memory and will error against a dead backend. ⚠ `vllm-granite` is **still a defined service** in `stacks/vllm/compose.yaml` though the model was retired 2026-08-12 β€” dead config, 0 VRAM (stopped), worth the same cleanup pass. +- **βšͺ LFM2.5-2.6B RETIRED PERMANENTLY 2026-08-20 (operator directive).** `vllm-lfm25` (:8021, GPU1) removed: service deleted from `stacks/vllm/compose.yaml` + pushed live (backup `compose.yaml.bak-pre-lfm25-retire-20260820`), container `docker rm -f`'d, `lfm2.5-2.6b` alias deleted from the LiteLLM config (live + canonical; backup `config.yaml.bak-pre-lfm25-retire-20260820`, 28β†’27 models). **Freed 8,721 MiB on GPU1.** Justification: it was an EVAL-ONLY bake-off seat vs `granite-4.1-8b` (brokkr R-target 2026-08-10) that never received the operator ruling it was pending; the comparator was retired from the roster 2026-08-15; it was deliberately never in any default/fallback routing chain; and spend logs showed **0 calls in the 4-day window**. Weights remain in the shared HF cache β€” nothing deleted from disk. The held gateway restart fired at 23:46 alongside the `qwen3-reranker` removal (one ~84s blip covered both); `lfm2.5-2.6b`, `qwen3-reranker` and `granite-4.1-8b` all now 400 cleanly. `vllm-granite`'s dead service block was removed in the same pass. -- **OPEN FOLLOW-UPS (parked):** repoint `nevermore` off the retired reranker (see the πŸ”΄ reranker audit) then retire `:8002`/`:8014`; delete the dead `vllm-granite` service block; move gen seat off pinned-nightly to stable once #51113 ships; Lobe one-time TTS UI pass; delete the 1.8GB litellm dump; `harden-esh-docker-vm` (park id 28, PROMOTED β€” Tier-1 done, `/mnt/books` stays hard w/ watchdog); chatterbox-fast build-context divergence; #363 research-wing ingest (no deadline); optionally attach our MTP reproducer to vllm#47087 (needs a GitHub identity β€” operator's call). +- **OPEN FOLLOW-UPS (parked):** ask Brokkr whether the duplicate `reranker-a3-bge-v2-m3` alias can go; move gen seat off pinned-nightly to stable once #51113 ships; Lobe one-time TTS UI pass; delete the 1.8GB litellm dump; `harden-esh-docker-vm` (park id 28, PROMOTED β€” Tier-1 done, `/mnt/books` stays hard w/ watchdog); chatterbox-fast build-context divergence; #363 research-wing ingest (no deadline); optionally attach our MTP reproducer to vllm#47087 (needs a GitHub identity β€” operator's call). - **althing monitor** ARMED (handle `infra-ops`). ⚠️ Re-arm ONLY after a real FIRE (rc0), never after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, never chained with `&` (orphans it β€” hit this twice 2026-08-17, `stop-monitor` reclaims). diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index ee4ab0a..982fbb9 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -8,7 +8,8 @@ # Model-name β†’ upstream mapping: # phi4-mini β†’ vLLM :8004 (generative chat) # qwen3-embedding β†’ vLLM :8001 (/v1/embeddings) -# qwen3-reranker β†’ vLLM :8002 (/rerank) +# reranker β†’ vLLM :8013 (/rerank; bge-v2-m3. The old qwen3-reranker +# alias on :8002 was retired 2026-08-20.) # * (wildcard) β†’ llama-swap :9292 (the swappable generative zoo) # # The wildcard fronts llama-swap so its whole model zoo logs through the @@ -313,14 +314,14 @@ model_list: model_info: mode: embedding - # --- Qwen3 reranker (proxy /rerank route) --- - - model_name: qwen3-reranker - litellm_params: - model: hosted_vllm/Qwen/Qwen3-Reranker-0.6B - api_base: http://10.250.50.54:8002/v1 - api_key: os.environ/VLLM_API_KEY - model_info: - mode: rerank + # --- qwen3-reranker RETIRED 2026-08-20. It named Qwen3-Reranker-0.6B on :8002, + # the incumbent the R43 bake-off replaced on 2026-08-06 after measuring it + # HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). The alias + # was kept as the rollback path and, for 13 days, was the ONLY reranker + # actually receiving traffic: nevermore was pinned to it by name, so the + # cutover moved `reranker` but never moved nevermore. Fixed at the consumer + # (nevermore now pins `reranker`), then the seat and this alias were retired. + # Use `reranker` -> bge-reranker-v2-m3 :8013. --- # --- Worldtree capability aliases (roleβ†’capability gateway swaps, ADR-0012). # Stable role-named aliases so consumers bind the CAPABILITY, not a concrete @@ -337,7 +338,8 @@ model_list: top_p: 0.9 model_info: mode: chat - # reranker β†’ generic capability name for rerank (currently qwen3-reranker). + # reranker β†’ generic capability name for rerank. THE reranker alias β€” the only + # one left as of 2026-08-20. Every consumer pins this name, never a model name. - model_name: reranker litellm_params: model: hosted_vllm/BAAI/bge-reranker-v2-m3 diff --git a/stacks/vllm/.env.example b/stacks/vllm/.env.example index a14a3f8..6126433 100644 --- a/stacks/vllm/.env.example +++ b/stacks/vllm/.env.example @@ -4,12 +4,19 @@ # # edit .env with real values # docker compose up -d -# Image version β€” pin for reproducibility (`latest` for edge) -VLLM_VERSION=latest +# Image version β€” PINNED. Do not put `latest` here: every service in this file +# shares this one variable, so a bare `docker compose up -d` would silently +# upgrade the whole stack's vLLM at once. Pinned 2026-08-20 to the version all +# seats were already running (`latest` and `v0.24.0` were the same local image, +# 4091d5593f77, so the pin changed nothing at runtime). Bump deliberately. +VLLM_VERSION=v0.24.0 # Host ports (container always listens on 8000 internally) EMBED_PORT=8001 -RERANK_PORT=8002 +# 8013 β€” the reranker moved here 2026-08-20 when bge-v2-m3 (the R43 winner, which +# had been running as a throwaway `docker run` on this port) was promoted into +# this stack and the Qwen incumbent on :8002 was retired. +RERANK_PORT=8013 REWARD_PORT=8003 # GPU assignment β€” all services share this GPU @@ -18,7 +25,10 @@ GPU_ID=1 # Models β€” reference by full repo name in API requests EMBED_MODEL=Qwen/Qwen3-Embedding-0.6B -RERANK_MODEL=Qwen/Qwen3-Reranker-0.6B +# bge-reranker-v2-m3 β€” the R43 bake-off winner, replacing Qwen3-Reranker-0.6B +# (measured HARMING 80/90 fleet queries). Multilingual cross-encoder; needs no +# --hf-overrides, unlike the Qwen reranker it displaced. +RERANK_MODEL=BAAI/bge-reranker-v2-m3 # Skywork is a local-path AWQ output, not from HF Hub. Bind-mounted into the # reward container at /local-models β€” see compose.yaml. No env var here for # the model path itself since it's hard-coded in the compose command. @@ -72,62 +82,12 @@ API_KEY= # services (embed/rerank). Reward is local-path, ignores this. HF_TOKEN= -# === granite-4.1-8b (production summarizer / dreaming agent) === -# Replaced phi4-mini 2026-06-05 (Granite 4.1 8B beat phi4 on precision in -# brokkr's R15 P03 model-fitness eval). Same GPU-1 slot, reusing phi4's port. -GRANITE_PORT=8004 -# GPU 1 β€” co-located with the embed/rerank/reward trio. With phi4 retired, GPU 1 -# has ~19 GB free; granite at 56K + FP8 KV needs ~15-16 GB β†’ ~2-3 GB margin. -# NOTE: llama-swap also uses GPU 1 dynamically β€” a large swap-in could contend; -# pin llama-swap to GPU 0 for clean separation (follow-up). -GRANITE_GPU_ID=1 -# Official IBM pre-quantized FP8 (compressed-tensors) β€” calibrated, ~9.6 GB, -# loaded directly (FP8 native on Blackwell cc 12.0). Fallback to vLLM-native dynamic FP8 -# from BF16: GRANITE_MODEL=ibm-granite/granite-4.1-8b + GRANITE_QUANT=fp8. -GRANITE_MODEL=ibm-granite/granite-4.1-8b-fp8 -GRANITE_QUANT=compressed-tensors -GRANITE_SERVED_NAME=granite-4.1-8b -# 131072 ctx β€” MAXED 2026-06-13 (was 51200/50K). GPU-1 rebalance: granite shares -# the card with the trio + qwen35-vl (vision). Qwen was over-provisioned on KV -# (20x conc @ 32k), so trimming it freed room for granite's ~305k-token pool here. -# PagedAttention allocates KV per ACTUAL token, so 131072 is only a CEILING β€” a 1k -# summarize turn uses ~1k tokens, so the pool holds ~300 concurrently; the "2.33x" -# headline is worst-case (every request maxing 131k). Granite 4.1 supports 131072. -# 65536 β€” REDUCED 2026-06-14 (was 131072) to free GPU-1 room for the FP8 vision -# model (Qwen3.6-35B-A3B, stacks/qwen36-vl, ~34 GB weights). Summarizer load is -# short parallel calls, so the 64K cap is ample. -# 131072 β€” RESTORED 2026-07-16 (native max) for full-chapter summarization; GPU-1 -# freed by the image-bench evict + qwen36-vl gone, so the 128K ctx fits again. -# 16384 β€” SHRUNK 2026-07-27 (operator: granite is being phased out) to free GPU-1 -# room for vllm-coder (the Zed FIM seat). Full-chapter ctx dropped; util 0.13 holds -# 16K cleanly (32768 @ util 0.12 crash-looped: KV est-max was only 19376 tokens). -GRANITE_MAX_MODEL_LEN=16384 -# FP8 KV cache (native on Blackwell cc 12.0). At 50K β‰ˆ ~4.2 GB (vs ~8.4 GB at fp16). -GRANITE_KV_CACHE_DTYPE=fp8 -# util 0.35 (~33.6 GB) β€” tuned 2026-06-13 to leave ~3.5 GB free on GPU 1 alongside -# the trio + qwen co-tenants. On this shared card vLLM needs free >= util*total at -# startup, and START ORDER matters: trim qwen FIRST, then grow granite, else granite -# OOMs against the full card. (0.37 overshot to 1.7 GB free; 0.35 lands ~3.7 GB.) -# 0.24 β€” REBALANCED 2026-06-14 (was 0.35) for the FP8 vision cutover. GPU-1 budget: -# qwen36-vl 0.46 + granite 0.24 + reward 0.10 + embed/rerank 0.03 β‰ˆ 0.90 total, -# ~7.5 GB headroom (the OOM buffer; held under 20-concurrent load test). granite -# gets a 169K-token KV pool = 2.58x concurrency @ 64K. Bring qwen36 up LAST. -# 0.18 β€” RIGHT-SIZED 2026-07-16 (was 0.34 live; qwen36-vl no longer a GPU-1 tenant, -# image-bench evicted) to free ~10.5 GB for relocating a GPU0 model onto GPU1. KV -# 6.45 GiB = 84,528 tokens = 1.29x concurrency @ 65536 (summarizer = short parallel -# calls; ample). Effective slope on this shared card β‰ˆ 950 MiB KV per 0.01 util, and -# KV must hold >= 1x max-model-len β€” util 0.15 undershot (crash: est max-len 47184 < -# 65536), 0.18 lands the target cleanly. -# 0.27 β€” RE-GROWN 2026-07-16 (same session) after char-rp moved onto GPU-1: spend the -# leftover room on full-chapter context (max-len 131072). KV 15.0 GiB = 196,560 tokens -# = 1.50x @ 131072; GPU-1 lands ~6.7 GB headroom (char-rp 30 + granite 27 + selene 17 + trio). -# 0.13 β€” SHRUNK 2026-07-27 (was 0.27) with the max-len drop; frees ~14 GB on GPU-1 for -# vllm-coder. granite phasing out, so no longer worth the big KV pool. (0.12 was too -# small for even 16K KV; 0.13 gives ~1.4x @ 16384.) -GRANITE_GPU_MEM_UTIL=0.13 -# Concurrency cap. Was 1024 (fan-out summarizer); DROPPED to 256 on 2026-07-27 with the -# phasing-out shrink (256 is ample for the reduced summarizer load; smaller sched state). -GRANITE_MAX_NUM_SEQS=256 +# === granite-4.1-8b β€” RETIRED 2026-08-12, vars removed 2026-08-20 === +# Was the production summarizer. The `summarizer` / `classifier` gateway aliases +# were repointed at the gen seat and the container stopped; the service block and +# these tunables are now gone. See the tombstone in compose.yaml for the lesson +# this cost us (nevermore stayed pinned to the dead `granite-4.1-8b` alias and +# failed silently for 8 days). # Qwen2.5-Coder-1.5B (BASE) β€” FIM code-completion seat for Zed editor inline # edit-predictions (deep-research pick 2026-07-27, Apache-2.0). GPU-1, alongside the @@ -143,17 +103,8 @@ CODER_KV_CACHE_DTYPE=fp8 CODER_GPU_MEM_UTIL=0.06 CODER_MAX_NUM_SEQS=32 -# LFM2.5-2.6B β€” NON-PRODUCTION bake-off alias vs granite (brokkr R-target 2026-08-10). -# LiquidAI LFM Open License v1.0 (); served raw (no vLLM -# reasoning-parser) so content is non-empty. util 0.09 (~8.6GB) fits GPU1's free slack -# without touching production reservations; max-len 16384 keeps KV small (bake-off -# doesn't need the 131k ceiling). Vendor sampling lives in the LiteLLM alias. -LFM25_PORT=8021 -LFM25_GPU_ID=1 -LFM25_MODEL=LiquidAI/LFM2.5-2.6B -LFM25_SERVED_NAME=lfm2.5-2.6b -LFM25_MAX_MODEL_LEN=16384 -LFM25_KV_CACHE_DTYPE=auto -LFM25_GPU_MEM_UTIL=0.09 -LFM25_MAX_NUM_SEQS=8 +# === LFM2.5-2.6B β€” RETIRED PERMANENTLY 2026-08-20 (operator directive) === +# Eval-only bake-off seat vs granite-4.1-8b that never got its production ruling; +# its comparator was retired first, and it logged 0 calls in the 4 days before it +# came down. Container removed, service block and tunables deleted, gateway alias +# dropped. Weights remain in the shared HF cache. diff --git a/stacks/vllm/compose.yaml b/stacks/vllm/compose.yaml index eb96749..06e6f4a 100644 --- a/stacks/vllm/compose.yaml +++ b/stacks/vllm/compose.yaml @@ -83,9 +83,27 @@ services: - homepage.description=Qwen3 Embedding via vLLM (ana-ml2) - homepage.href=http://10.250.50.54:${EMBED_PORT}/docs - vllm-rerank: + # THE fleet reranker. Backs the LiteLLM `reranker` alias, which is what every + # consumer should name β€” never the model, never a bake-off arm name. + # + # It won the Brokkr R43 bake-off (docs/pfi/reranker-selection-ledger.md) and + # replaced Qwen3-Reranker-0.6B, which was measured HARMING 80/90 fleet queries + # (no-reranker beat it 89/90 vs 56/90). The R42 v13 acceptance gate went + # 56/90 -> 90/90 on the cutover, its first-ever PASS. + # + # Promoted from a throwaway `docker run` to this service 2026-08-20 (the + # ledger's own open follow-up). It carried the bake-off's arm name from the + # start and KEEPS it: the ledger, persistent-memory and the R43 record all say + # `vllm-rerank-a3`, and renaming for tidiness would orphan every one of those + # references. The name carries its provenance. + # + # Retired alongside this promotion: `vllm-rerank` (Qwen3-Reranker-0.6B, :8002 β€” + # the rollback path, kept warm 13 days) and `vllm-rerank-a4` + # (gte-reranker-modernbert, :8014 β€” a documented throughput fallback that was + # never given a gateway alias, so it was unreachable the whole time). + vllm-rerank-a3: image: vllm/vllm-openai:${VLLM_VERSION} - container_name: vllm-rerank + container_name: vllm-rerank-a3 restart: unless-stopped ipc: host ports: @@ -103,8 +121,9 @@ services: - ${RERANK_MODEL} - --runner - pooling - - --hf-overrides - - '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}' + # No --hf-overrides here. The Qwen reranker needed one to be coerced into a + # sequence-classification head; bge-reranker-v2-m3 is natively a + # cross-encoder and vLLM resolves it directly. - --host - 0.0.0.0 - --port @@ -134,9 +153,9 @@ services: - tnet labels: - homepage.group=AI - Eval & Retrieval - - homepage.name=vLLM Rerank (Qwen3) + - homepage.name=vLLM Rerank (bge-v2-m3) - homepage.icon=mdi-sort-variant - - homepage.description=Qwen3 Reranker via vLLM (ana-ml2) + - homepage.description=BAAI bge-reranker-v2-m3 β€” the fleet reranker, backs the `reranker` alias (ana-ml2) - homepage.href=http://10.250.50.54:${RERANK_PORT}/docs vllm-reward: @@ -197,86 +216,24 @@ services: - homepage.description=Skywork-Reward-V2 8B classifier via vLLM (ana-ml2) - homepage.href=http://10.250.50.54:${REWARD_PORT}/docs - # Granite 4.1 8B (FP8) β€” production summarizer (replaced phi4-mini - # 2026-06-05, which had superseded the llama-swap granite-4-small pin). - # Generative chat model (OpenAI /v1/chat/completions), so NO --runner - # pooling. FP8 on RTX PRO 6000 Blackwell (cc 12.0): near-lossless, ~1.2x, ~6 GB. - vllm-granite: - image: vllm/vllm-openai:${VLLM_VERSION} - container_name: vllm-granite - restart: unless-stopped - ipc: host - ports: - - "${GRANITE_PORT}:8000" - volumes: - - /tank/aimodels/huggingface:/hfcache - environment: - - HF_HOME=/hfcache - - HF_HUB_CACHE=/hfcache/hub - - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-} - - VLLM_API_KEY=${API_KEY:-} - command: - # Production summarizer (replaced phi4-mini 2026-06-05). Default = official - # IBM pre-quantized FP8 (compressed-tensors), loaded directly; FP8 is native - # on the RTX PRO 6000 Blackwell (cc 12.0). Fallback to vLLM-native dynamic FP8 from - # BF16: GRANITE_MODEL=ibm-granite/granite-4.1-8b + GRANITE_QUANT=fp8. - - ${GRANITE_MODEL} - - --served-model-name - - ${GRANITE_SERVED_NAME} - - --quantization - - ${GRANITE_QUANT} - - --host - - 0.0.0.0 - - --port - - "8000" - - --gpu-memory-utilization - - ${GRANITE_GPU_MEM_UTIL} - - --max-model-len - - ${GRANITE_MAX_MODEL_LEN} - # Very high so the KV pool (not the seq cap) is the only concurrency bound β€” - # granite is the fleet fan-out summarizer/classifier (many concurrent SHORT - # calls). vLLM's default resolves to 128, capping below the KV bound - # (~192 @ 1K-tok); 1024 unblocks it (VRAM-neutral β€” KV pool is util-bound). - - --max-num-seqs - - ${GRANITE_MAX_NUM_SEQS} - - --dtype - - auto - # CUDA graphs ENABLED (no --enforce-eager) for decode throughput. Made - # room 2026-06-05 by right-sizing the embed/rerank/reward trio's KV pools - # (they were over-provisioned at 5.9x/2.0x/3.9x concurrency); GPU 1 now has - # ~17 GB free after granite, so graph-capture buffers fit. If the trio - # ever grows back, granite may need --enforce-eager again on this card. - # FP8 KV cache β€” halves KV memory; near-lossless on Blackwell (cc 12.0). - - --kv-cache-dtype - - ${GRANITE_KV_CACHE_DTYPE} - # Prefix caching pinned EXPLICIT (vLLM v1 defaults it on, but pin so a - # version flip can't silently disable it). Benched 2026-06-13: ~6.5x faster - # TTFT (45ms vs 292ms) on a shared ~4.5k-token summarizer template; soft/ - # evictable KV, neutral when prefixes don't repeat β€” pure win for granite. - - --enable-prefix-caching - deploy: - resources: - reservations: - devices: - - driver: nvidia - device_ids: - - "${GRANITE_GPU_ID}" - capabilities: - - gpu - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8000/health"] - interval: 30s - timeout: 10s - retries: 3 - start_period: 180s - networks: - - tnet - labels: - - homepage.group=AI - Inference - - homepage.name=vLLM Granite 4.1 8B (summarizer) - - homepage.icon=mdi-text-box-outline - - homepage.description=Granite 4.1 8B FP8 via vLLM (ana-ml2) - - homepage.href=http://10.250.50.54:${GRANITE_PORT}/docs + # vllm-granite (ibm-granite/granite-4.1-8b-fp8, :8004) β€” RETIRED 2026-08-12, + # service block removed 2026-08-20. It was the fleet summarizer until the + # `summarizer` and `classifier` aliases were repointed at the gen seat; the + # container was stopped then and sat Exited for 8 days while this block still + # claimed it as production. + # + # ⚠️ Retiring the SEAT did not retire its CONSUMERS, and nobody checked. The + # `granite-4.1-8b` gateway alias was deleted at the same time, but `nevermore` + # was still pinned to that alias by name β€” so its LLM summarization pass failed + # 67 consecutive times over 8 days, 0 tokens, entirely silently, because a + # failed gateway call still returns 200-shaped spend-log rows and nothing + # alerts on status=failure. Found 2026-08-20 only because someone asked an + # unrelated question about reranker VRAM. + # + # The rule this earns: RETIRING A MODEL IS A TWO-SIDED OPERATION. Grep every + # consumer's config for the alias BEFORE deleting it, and prefer that consumers + # pin stable ALIASES (`summarizer`) over model names (`granite-4.1-8b`) so the + # gateway can repoint them without anyone editing a downstream .env. vllm-coder: image: vllm/vllm-openai:${VLLM_VERSION}