fix(fv-ml1): pin vLLM KV in bytes on meromero (22 GiB) and coder; gen-small KV 8 -> 5.5 GiB

meromero-rp sized its KV from --gpu-memory-utilization, and vLLM's profile
counts co-booting neighbours as its own non-torch memory. Three boots gave
20.37 / 26.74 / 29.04 GiB of KV, and the 2026-10-05 Docker-move restart cut
GPU 1 free memory to 3,981 MiB, below intern-decision's 32k-token budget.
Coder read negative KV on every co-boot. Both are now pinned, so util only
gates the boot check (meromero util 0.52 -> 0.47, just above its footprint).

GPU 0 plateaus at ~60 MiB free 1-2 days after a boot (Beszel), so
gen-small's KV drops to 5.5 GiB (461,215 tokens, 1.76x) and parakeet-nemo
stays on GPU 0 with room for its 3,840 MiB cap. GPU 3 stays clear.

Applied with playbooks/fv-ml1-kv-pin-20261005.yaml: GPU 1 free 3,981 ->
11,801 MiB, GPU 0 495 -> 5,039 MiB right after boot; all nine seats healthy.
This commit is contained in:
vh
2026-10-05 21:40:28 -07:00
parent 269c80f3ca
commit 6851ecbd0b
9 changed files with 229 additions and 9 deletions
+9
View File
@@ -26,6 +26,15 @@
# stopped fitting on the 24th": nothing about this seat changed, gen crept up
# underneath it. Headroom arithmetic done against a long-running gen is
# measuring a moving number.
#
# 2026-10-05: the seat lives on fv-ml1 GPU 1 (MEROMERO_GPU_ID=1 on the host),
# shared with erp-seat, coder and intern-decision. KV is now PINNED in bytes
# (Prime: 22 GiB, ~346k tokens, ~1.32x at 262k). Util-sized KV was a co-boot
# lottery (20.37 / 26.74 / 29.04 GiB on three boots) and the 29.04 boot ate
# intern-decision's 32k headroom. Util now only gates the boot check, so it
# sits just above the real footprint (~43.6 GiB). Before ANY restart:
# util x 95.0 GiB <= measured Free + this seat's own resident memory.
MEROMERO_GPU_MEM_UTIL=0.47
MEROMERO_KV_CACHE_MEMORY=23622320128
MEROMERO_MAX_MODEL_LEN=262144
MEROMERO_GPU_ID=0
+8 -1
View File
@@ -64,7 +64,14 @@ services:
- --max-num-seqs
- "${MEROMERO_MAX_NUM_SEQS:-32}"
- --gpu-memory-utilization
- "${MEROMERO_GPU_MEM_UTIL:-0.52}"
- "${MEROMERO_GPU_MEM_UTIL:-0.47}"
# KV pinned in bytes (2026-10-05, Prime: 22 GiB). Util-sized KV was a lottery on
# this shared card: vLLM's profile counts co-booting neighbours as its own non-torch
# memory, so three boots gave 20.37 / 26.74 / 29.04 GiB of KV. With this set, vLLM
# skips profiling and util only gates the boot check (Free at start >= util x total),
# so util stays just above the real footprint (~43.6 GiB).
- --kv-cache-memory
- "${MEROMERO_KV_CACHE_MEMORY:-23622320128}"
- --kv-cache-dtype
- fp8
- --trust-remote-code