16 lines
4.4 KiB
Markdown
16 lines
4.4 KiB
Markdown
# `[2026-10-05]` fv-ml1: vLLM KV pinned in bytes; GPUs 0/1 rebalanced by usage; contexts to published maximums; gen-small output cap
|
||
|
||
## As recorded in the in-flight section at the time
|
||
|
||
- **fv-ml1 GPU 0/1 REBALANCED 2026-10-05 2226–2249 (Prime: "accept all that" + "context up to published maximums if at all possible"; `playbooks/fv-ml1-rebalance-20261005.yaml`, elway all green, gateway smoke tests pass).** It follows the 2131 KV pins (meromero 22 GiB, coder pinned, gen-small 5.5 GiB). Why: on a shared card util-sized KV is a co-boot lottery, and GPU 0 creeps to ~60 MiB free after a boot. Rule: KV ≥ 1.25× the seat's max context, rounded up to 0.5 GiB. Two exceptions: meromero and gen-small keep Prime's numbers, and voices gets 1.07× (short prose traffic). Usage basis: 7 days of LiteLLM logs (gen-small 22,396 calls; sec 400; char-rp-fast 55; char-rp 14) plus the engines' own KV logs. **Layout:** GPU 0 = cyberprev, erp-seat, voices, coder, parakeet-nemo (4,036 MiB free right after; ~2.3 GB spare at parakeet's cap; cyberprev may still creep). GPU 1 = gen-small, meromero-rp, intern-decision (10,864 free; ~3.3 GB spare at the 32k peak once gen-small settles). Contexts raised: coder 16,384 → **32,768** (card max), voices 8,192 → **40,960** (native max). YaRN extensions were NOT enabled, and **Prime confirmed 2026-10-06: leave YaRN off for gen-small** (gen-small/cyberprev to ~1M, voices to 131k). They don't fit, and Qwen's own cards warn static YaRN hurts short prompts. intern-decision stays at 32,768 (card default 8,192; base config 262k; ~0.21 MiB/token of VRAM). **gen-small default output cap 32,768** (`--override-generation-config`; an explicit max_tokens still wins, verified in vLLM `get_max_tokens`). It answers 9 runaway ~260k-token replies in 7 days (albok-nemi ×6, all-agents-local ×3, all from nh3-dev); albok-dev told 2250 and closed it 10-06 0000. Nemi now sends max_tokens 8192 (albok-nemi 0.1.2, `tagging.max_tokens`), and its nh3-dev harness configs moved off the shared all-agents-local key onto vault `albok/litellm-key-nemi` + `albok/litellm-key`. The loops came from gen-small at temperature 0 on some memory entries; a capped reply degrades to Nemi's rule-based tagger. ⚠ **Verify both cards' plateaus ~10-07/08** from Beszel (`stats.g.<n>.mu` × 1.024). .env backups `*.env.bak-20261005-rebal`. Earlier the same night: KV pins 2131–2139 (`playbooks/fv-ml1-kv-pin-20261005.yaml`; diagnosis in commit 269c80f). Park 75 dropped by Prime 2115.
|
||
|
||
## The diagnosis that led to it (2026-10-05 ~2120, read-only)
|
||
|
||
- GPU 1 lost ~2.5 GB in one step at the 0234 Docker-move restart (Beszel 2 h windows: flat 88,566–88,779 for 4 days, then flat 91,082; Beszel `mu` = MiB / 1.024). The cause: `vllm-meromero-rp` sized KV from `--gpu-memory-utilization` 0.52, and vLLM's profile counts co-booting neighbours as its own non-torch memory. Non-torch read 8.11 → 1.74 → −0.56 GiB over three boots, and KV came out 20.37 → 26.74 → 29.04 GiB. intern-decision's 32k-call budget went ~2.4 GB short.
|
||
- GPU 0 had no step; it creeps for 1–2 days after each boot and plateaued at ~60 MiB free (10-03 → 10-05). The 10-01 "Free ~1.05 GB at rest" was read just after a restart.
|
||
- Fix 1 (2131–2139, `playbooks/fv-ml1-kv-pin-20261005.yaml`): meromero pinned 22 GiB (Prime), coder pinned at its usual 77,056 tokens, gen-small 8 → 5.5 GiB (Prime: leave parakeet on GPU 0; GPU 3 stays clear).
|
||
- Fix 2 (2226–2249, `playbooks/fv-ml1-rebalance-20261005.yaml`): usage-based sizing plus Prime's "contexts up to published maximums". The planned voices↔coder swap no longer fit once voices went to 40,960, so the partition became gen-small → GPU 1, erp + coder → GPU 0 (brute-forced over every split for the best worst-case spare).
|
||
- Usage basis (7 days of LiteLLM spend logs plus the engines' own KV logs): gen-small 22,396 calls (p99 prompt 4.5k, peak in-flight ~24.5k tokens excluding runaways); sec 400; char-rp-fast 55; char-rp 14.
|
||
- YaRN: Prime confirmed 2026-10-06, leave it off for gen-small (cyberprev ~1M would need ~45 GiB; voices 131k ~18 GiB bf16; gen-small fits only by evicting, and Qwen's cards warn static YaRN degrades short prompts, 99% of its traffic).
|
||
- Lessons kept as auto-memory: `reference_vllm_util_kv_coboot_race`, `feedback_vllm_headroom_at_plateau_not_boot`. vLLM's `--kv-cache-memory` exists since v0.10.2 (PR #21489); ⚠ open vLLM PR #56698 would re-introduce profiling for pinned sizes (check it before any image bump).
|