diff --git a/persistent-memory.md b/persistent-memory.md index 32c1bcb..2711d3a 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -119,7 +119,7 @@ _As of 2026-07-15 — a long fleet-ops + infra session. Everything below either **PARKED (operator-directed holds):** - **image-bench (qwen-image-bench, GPU1) eviction — DONE 2026-07-15 (operator-directed, same-day as the switch).** Stopped `vllm-qwen-image-bench` on ana-ml2 (Exited(0), revert via `docker compose start`) → GPU1 freed ~32 GB (95→63 GB used). LiteLLM `image-judge` + `qwen-image-bench` aliases repointed to gen (:8015, qwen3.6-35b-a3b-heretic), deterministic judge sampling (temp0/top_k1/rep_pen1.05) + `enable_thinking:false` kept; verified live with :8014 down. comfy-dev pinged as regression watcher (NSFW-anatomy + tight-grounding; thread 01KXMRDJ). Backstop = the stopped stack (revert ~90 s) + live config backup `config.yaml.bak-pre-imagejudge-20260715`. Full record: auto-memory `project_arbo_gen_switch_imagebench_evict`. -- **granite right-size — un-started GPU1 win.** granite (GPU1) is ~29 GB (util 0.34 -> ~16.5 GB oversized KV for an 8B summarizer). Right-sizing util 0.34 -> ~0.13 frees ~16 GB WHILE keeping the summarizer (fleet's dedicated summarizer/classifier; aliases granite-4.1-8b/summarizer/classifier). Independent of image-bench; operator was freeing GPU1 for a TBD "something else". +- **granite right-size — DONE 2026-07-16 (~10.5 GB freed).** util 0.34→0.18 + max-model-len 131072→65536 on `vllm-granite` (GPU1, `/opt/docker/compose/vllm` .env — recreate `vllm-granite` ONLY, it's a shared stack w/ embed/rerank/reward). GPU1 62,641→51,897 MiB (~45 GB free now); KV 6.45 GiB / 84,528 tok / 1.29x@65536, summarizer healthy. Effective slope ~950 MiB KV per 0.01 util + KV must hold ≥1× max-len; util 0.15 undershot (crash-loop, ~2-3 min summarizer outage) before 0.18 landed. Frees GPU1 to relocate a GPU0 model — operator's next call: which (gen :8015 / char-rp :8016 / char-rp-reasoning :8018). Backup `.env.bak-pre-granite-rightsize-20260716`. **Open loops for the fresh session:** - **`stacks/heretic2-charrp-reasoning/` is UNTRACKED in git** (`?? stacks/heretic2-charrp-reasoning/`) — the char-rp-reasoning compose stack is deployed but never committed; commit it (canonical mirror of the live seat). @@ -128,6 +128,8 @@ _As of 2026-07-15 — a long fleet-ops + infra session. Everything below either ## Recent decisions +- `[2026-07-16]` **granite right-sized → ~10.5 GB freed on GPU1** (util 0.34→0.18 + max-len 131072→65536; KV 6.45 GiB / 1.29x@65536, summarizer healthy). GPU1 now ~45 GB free to relocate a GPU0 model. LESSON: ~950 MiB KV per 0.01 util here + KV must hold ≥1× max-len — util 0.15 crash-looped (est max-len 47184<65536, ~2-3 min summarizer blip) before 0.18 landed. `.env`-only, recreate `vllm-granite` alone (shared stack). + - `[2026-07-15]` **image-bench eviction DONE (parked item closed).** Stopped vllm-qwen-image-bench (ana-ml2 GPU1, ~32 GB freed); LiteLLM `image-judge`+`qwen-image-bench` → gen :8015 (judge samplers + thinking-off), verified with :8014 down; comfy-dev pinged; also backfilled the canonical char-rp-reasoning litellm block (was lagging live). Revert ~90 s. auto-memory `project_arbo_gen_switch_imagebench_evict`. - `[2026-07-15]` arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake → `persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md` diff --git a/stacks/vllm/.env.example b/stacks/vllm/.env.example index ce1d18e..313e890 100644 --- a/stacks/vllm/.env.example +++ b/stacks/vllm/.env.example @@ -107,4 +107,10 @@ GRANITE_KV_CACHE_DTYPE=fp8 # qwen36-vl 0.46 + granite 0.24 + reward 0.10 + embed/rerank 0.03 ≈ 0.90 total, # ~7.5 GB headroom (the OOM buffer; held under 20-concurrent load test). granite # gets a 169K-token KV pool = 2.58x concurrency @ 64K. Bring qwen36 up LAST. -GRANITE_GPU_MEM_UTIL=0.24 +# 0.18 — RIGHT-SIZED 2026-07-16 (was 0.34 live; qwen36-vl no longer a GPU-1 tenant, +# image-bench evicted) to free ~10.5 GB for relocating a GPU0 model onto GPU1. KV +# 6.45 GiB = 84,528 tokens = 1.29x concurrency @ 65536 (summarizer = short parallel +# calls; ample). Effective slope on this shared card ≈ 950 MiB KV per 0.01 util, and +# KV must hold >= 1x max-model-len — util 0.15 undershot (crash: est max-len 47184 < +# 65536), 0.18 lands the target cleanly. +GRANITE_GPU_MEM_UTIL=0.18