esh-pfi-infrastructure/persistent-memory.d/2026-10-05-fv-ml1-kv-pins-and-rebalance.md

[2026-10-05] fv-ml1: vLLM KV pinned in bytes; GPUs 0/1 rebalanced by usage; contexts to published maximums; gen-small output cap

As recorded in the in-flight section at the time

• fv-ml1 GPU 0/1 REBALANCED 2026-10-05 2226–2249 (Prime: "accept all that" + "context up to published maximums if at all possible"; playbooks/fv-ml1-rebalance-20261005.yaml, elway all green, gateway smoke tests pass). It follows the 2131 KV pins (meromero 22 GiB, coder pinned, gen-small 5.5 GiB). Why: on a shared card util-sized KV is a co-boot lottery, and GPU 0 creeps to ~60 MiB free after a boot. Rule: KV ≥ 1.25× the seat's max context, rounded up to 0.5 GiB. Two exceptions: meromero and gen-small keep Prime's numbers, and voices gets 1.07× (short prose traffic). Usage basis: 7 days of LiteLLM logs (gen-small 22,396 calls; sec 400; char-rp-fast 55; char-rp 14) plus the engines' own KV logs. Layout: GPU 0 = cyberprev, erp-seat, voices, coder, parakeet-nemo (4,036 MiB free right after; ~2.3 GB spare at parakeet's cap; cyberprev may still creep). GPU 1 = gen-small, meromero-rp, intern-decision (10,864 free; ~3.3 GB spare at the 32k peak once gen-small settles). Contexts raised: coder 16,384 → 32,768 (card max), voices 8,192 → 40,960 (native max). YaRN extensions were NOT enabled, and Prime confirmed 2026-10-06: leave YaRN off for gen-small (gen-small/cyberprev to ~1M, voices to 131k). They don't fit, and Qwen's own cards warn static YaRN hurts short prompts. intern-decision stays at 32,768 (card default 8,192; base config 262k; ~0.21 MiB/token of VRAM). gen-small default output cap 32,768 (--override-generation-config; an explicit max_tokens still wins, verified in vLLM get_max_tokens). It answers 9 runaway ~260k-token replies in 7 days (albok-nemi ×6, all-agents-local ×3, all from nh3-dev); albok-dev told 2250 and closed it 10-06 0000. Nemi now sends max_tokens 8192 (albok-nemi 0.1.2, tagging.max_tokens), and its nh3-dev harness configs moved off the shared all-agents-local key onto vault albok/litellm-key-nemi + albok/litellm-key. The loops came from gen-small at temperature 0 on some memory entries; a capped reply degrades to Nemi's rule-based tagger. ⚠ Verify both cards' plateaus ~10-07/08 from Beszel (stats.g.<n>.mu × 1.024). .env backups *.env.bak-20261005-rebal. Earlier the same night: KV pins 2131–2139 (playbooks/fv-ml1-kv-pin-20261005.yaml; diagnosis in commit 269c80f). Park 75 dropped by Prime 2115.

The diagnosis that led to it (2026-10-05 ~2120, read-only)

• GPU 1 lost ~2.5 GB in one step at the 0234 Docker-move restart (Beszel 2 h windows: flat 88,566–88,779 for 4 days, then flat 91,082; Beszel mu = MiB / 1.024). The cause: vllm-meromero-rp sized KV from --gpu-memory-utilization 0.52, and vLLM's profile counts co-booting neighbours as its own non-torch memory. Non-torch read 8.11 → 1.74 → −0.56 GiB over three boots, and KV came out 20.37 → 26.74 → 29.04 GiB. intern-decision's 32k-call budget went ~2.4 GB short.
• GPU 0 had no step; it creeps for 1–2 days after each boot and plateaued at ~60 MiB free (10-03 → 10-05). The 10-01 "Free ~1.05 GB at rest" was read just after a restart.
• Fix 1 (2131–2139, playbooks/fv-ml1-kv-pin-20261005.yaml): meromero pinned 22 GiB (Prime), coder pinned at its usual 77,056 tokens, gen-small 8 → 5.5 GiB (Prime: leave parakeet on GPU 0; GPU 3 stays clear).
• Fix 2 (2226–2249, playbooks/fv-ml1-rebalance-20261005.yaml): usage-based sizing plus Prime's "contexts up to published maximums". The planned voices↔coder swap no longer fit once voices went to 40,960, so the partition became gen-small → GPU 1, erp + coder → GPU 0 (brute-forced over every split for the best worst-case spare).
• Usage basis (7 days of LiteLLM spend logs plus the engines' own KV logs): gen-small 22,396 calls (p99 prompt 4.5k, peak in-flight ~24.5k tokens excluding runaways); sec 400; char-rp-fast 55; char-rp 14.
• YaRN: Prime confirmed 2026-10-06, leave it off for gen-small (cyberprev ~1M would need ~45 GiB; voices 131k ~18 GiB bf16; gen-small fits only by evicting, and Qwen's cards warn static YaRN degrades short prompts, 99% of its traffic).
• Lessons kept as auto-memory: reference_vllm_util_kv_coboot_race, feedback_vllm_headroom_at_plateau_not_boot. vLLM's --kv-cache-memory exists since v0.10.2 (PR #21489); ⚠ open vLLM PR #56698 would re-introduce profiling for pinned sizes (check it before any image bump).