diff --git a/persistent-memory.md b/persistent-memory.md index f5c7963..4d4f150 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -132,7 +132,7 @@ _As of 2026-08-20 — continued from the 08-19 infra session (DNS/.internal, wat - **🟢 LITELLM — upgraded v1.91.0→v1.97.0, spend-log DB purged 6GB→16MB + CAPPED (2026-08-17).** `store_prompts_in_spend_logs:false` + `maximum_spend_logs_retention_period:7d`. ⚠ **1.8GB pre-upgrade pg_dump still on ana-docker `/opt/docker/compose/litellm/` — deletable now the upgrade is proven** (operator was going to call it). Commit `01b5ad9`. -- **⚠️ GPU zero-sum (both cards ~94–95/97.9 GB).** GPU0: gen + meromero. GPU1: fablefusion + utility cluster. Any util bump on either seat of a shared card must be checked against the co-tenant (starved meromero into a crash-loop once at 0.45). +- **⚠️ GPU zero-sum (both cards ~94–95/97.9 GB).** GPU0: gen + meromero. GPU1: fablefusion + utility cluster. Any util bump on either seat of a shared card must be checked against the co-tenant (starved meromero into a crash-loop once at 0.45). **⚠️ BOOT ORDER IS PART OF THE STATE (2026-08-20).** `--gpu-memory-utilization` sets the target as a fraction of **TOTAL** VRAM, but vLLM **refuses to start unless that whole target is FREE right now** — so at ~96.4/97.9 GB the GPU0 pair coexists *only* in the order it was originally brought up. **Restore/reboot order: `vllm-meromero-rp` to `healthy` FIRST, then `vllm-gen`** — meromero (0.52 = 49.38 GiB) is the one that cannot fit in the remainder. "First" means **observed healthy**, not a `sleep`: a 10s gap against a 2–3 min weight load cost a 7-restart crash-loop. Verify a restore against **KV-pool size** (`GPU KV cache size` / `Maximum concurrency` in the container log), not `nvidia-smi` used-MiB — the latter swings ~7 GB on allocator slack with identical serving capacity. Baselines: gen ≈14.36 GiB / 403k tok / 1.54×; meromero 542,202 tok. - **FLEET RERANKER** = A3 (bge-reranker-v2-m3) PROD ana-ml2 GPU1 :8013. Passive watch; levers = A4 :8014 / util / 2nd replica; incumbent :8002 warm. `docs/pfi/reranker-selection-ledger.md`.