diff --git a/stacks/meromero-charrp/.env.example b/stacks/meromero-charrp/.env.example index 5ac8ce2..f6844ad 100644 --- a/stacks/meromero-charrp/.env.example +++ b/stacks/meromero-charrp/.env.example @@ -1,4 +1,22 @@ # ana-ml2 GPU0 char-rp prose seat (MeroMero-v2). Real .env lives on the host. -MEROMERO_GPU_MEM_UTIL=0.52 +# +# ⚠ GPU0 IS SHARED WITH `vllm-gen` AND THE BUDGETS MUST SUM UNDER ~0.92. +# gen runs at --gpu-memory-utilization 0.43 but actually holds ~45.6 GiB of the +# 94.97 GiB card, because the utilization figure sizes the KV cache and does not +# cover CUDA context, graphs and non-torch overhead. This seat was at 0.51: +# 0.43 + 0.51 = 0.94, which left ~0.6 GiB of real headroom and worked right up +# until it did not. On 2026-08-24 it stopped fitting and the seat entered a +# crash-loop — 13 restarts, `torch.OutOfMemoryError: ... 195.19 MiB is free`, +# both the `char-rp` and `char-rp-reasoning` gateway aliases returning 500s +# (they share this one seat; see the LiteLLM routes for :8016). +# +# 0.47 restores a real margin (~4.8 GiB free on the card) and costs nothing you +# can use: KV cache goes 27.36 GiB -> 23.56 GiB, which is 430,825 -> 371,023 +# tokens against a max-model-len of 262,144. The pool still holds 1.4x a +# full-length sequence; what you lose is concurrent long requests, not context. +# +# Raising this again means lowering gen's in the same change. Check the real +# numbers, not the flags: `nvidia-smi --query-compute-apps=pid,used_memory`. +MEROMERO_GPU_MEM_UTIL=0.47 MEROMERO_MAX_MODEL_LEN=262144 MEROMERO_GPU_ID=0