vh
6851ecbd0b
meromero-rp sized its KV from --gpu-memory-utilization, and vLLM's profile counts co-booting neighbours as its own non-torch memory. Three boots gave 20.37 / 26.74 / 29.04 GiB of KV, and the 2026-10-05 Docker-move restart cut GPU 1 free memory to 3,981 MiB, below intern-decision's 32k-token budget. Coder read negative KV on every co-boot. Both are now pinned, so util only gates the boot check (meromero util 0.52 -> 0.47, just above its footprint). GPU 0 plateaus at ~60 MiB free 1-2 days after a boot (Beszel), so gen-small's KV drops to 5.5 GiB (461,215 tokens, 1.76x) and parakeet-nemo stays on GPU 0 with room for its 3,840 MiB cap. GPU 3 stays clear. Applied with playbooks/fv-ml1-kv-pin-20261005.yaml: GPU 1 free 3,981 -> 11,801 MiB, GPU 0 495 -> 5,039 MiB right after boot; all nine seats healthy.
41 lines
2.4 KiB
Bash
41 lines
2.4 KiB
Bash
# ana-ml2 GPU0 char-rp prose seat (MeroMero-v2). Real .env lives on the host.
|
|
#
|
|
# ⚠ GPU0 IS SHARED WITH `vllm-gen` AND THE BUDGETS MUST SUM UNDER ~0.92.
|
|
# gen runs at --gpu-memory-utilization 0.43 but actually holds ~45.6 GiB of the
|
|
# 94.97 GiB card, because the utilization figure sizes the KV cache and does not
|
|
# cover CUDA context, graphs and non-torch overhead. This seat was at 0.51:
|
|
# 0.43 + 0.51 = 0.94, which left ~0.6 GiB of real headroom and worked right up
|
|
# until it did not. On 2026-08-24 it stopped fitting and the seat entered a
|
|
# crash-loop — 13 restarts, `torch.OutOfMemoryError: ... 195.19 MiB is free`,
|
|
# both the `char-rp` and `char-rp-reasoning` gateway aliases returning 500s
|
|
# (they share this one seat; see the LiteLLM routes for :8016).
|
|
#
|
|
# 0.47 restores a real margin (~4.8 GiB free on the card) and costs nothing you
|
|
# can use: KV cache goes 27.36 GiB -> 23.56 GiB, which is 430,825 -> 371,023
|
|
# tokens against a max-model-len of 262,144. The pool still holds 1.4x a
|
|
# full-length sequence; what you lose is concurrent long requests, not context.
|
|
#
|
|
# Raising this again means lowering gen's in the same change. Check the real
|
|
# numbers, not the flags: `nvidia-smi --query-compute-apps=pid,used_memory`.
|
|
#
|
|
# ⚠ AND CHECK THEM ON A FRESHLY-RESTARTED gen. Measured 2026-08-24: gen held
|
|
# 46,726 MiB (45.6 GiB) after ~3 days of uptime, and 39,424 MiB (38.5 GiB)
|
|
# immediately after a restart — the same container, the same
|
|
# `--gpu-memory-utilization 0.43`, ~7 GiB apart. Its footprint GROWS WITH
|
|
# UPTIME, which is the missing half of why this seat "fit on the 21st and
|
|
# stopped fitting on the 24th": nothing about this seat changed, gen crept up
|
|
# underneath it. Headroom arithmetic done against a long-running gen is
|
|
# measuring a moving number.
|
|
#
|
|
# 2026-10-05: the seat lives on fv-ml1 GPU 1 (MEROMERO_GPU_ID=1 on the host),
|
|
# shared with erp-seat, coder and intern-decision. KV is now PINNED in bytes
|
|
# (Prime: 22 GiB, ~346k tokens, ~1.32x at 262k). Util-sized KV was a co-boot
|
|
# lottery (20.37 / 26.74 / 29.04 GiB on three boots) and the 29.04 boot ate
|
|
# intern-decision's 32k headroom. Util now only gates the boot check, so it
|
|
# sits just above the real footprint (~43.6 GiB). Before ANY restart:
|
|
# util x 95.0 GiB <= measured Free + this seat's own resident memory.
|
|
MEROMERO_GPU_MEM_UTIL=0.47
|
|
MEROMERO_KV_CACHE_MEMORY=23622320128
|
|
MEROMERO_MAX_MODEL_LEN=262144
|
|
MEROMERO_GPU_ID=0
|