From 850e0c3351aa39253d01894d8bb838877c73ca59 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 24 Aug 2026 11:42:36 -0700 Subject: [PATCH] =?UTF-8?q?fix(meromero-charrp):=20drop=20GPU0=20budget=20?= =?UTF-8?q?to=200.47=20=E2=80=94=20the=20seat=20was=20OOM=20crash-looping?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `vllm-meromero-rp` had been restarting since 2026-08-24 18:2x, 13 times by the time it was looked at, taking both the `char-rp` and `char-rp-reasoning` gateway aliases down with it (they resolve to the same seat on :8016 — hosted_vllm/char-rp and hosted_vllm/char-rp-thinking). Root cause is CUDA OOM on ana-ml2 GPU0, which the startup logs hide well: the engine gets through weights, torch.compile and CUDA-graph capture looking entirely healthy, then dies at KV-cache allocation with `torch.OutOfMemoryError: ... 195.19 MiB is free`. GPU0 is shared with `vllm-gen`. gen is configured at 0.43 but actually holds ~45.6 GiB of the 94.97 GiB card, because --gpu-memory-utilization sizes the KV cache and does not account for CUDA context, graphs and non-torch overhead. This seat was at 0.51, so the pair was committed to 0.94 of the card with about 0.6 GiB of real headroom. That fit on 08-21 and stopped fitting today. 0.47 restores ~4.8 GiB of margin and costs nothing usable: KV cache 27.36 -> 23.56 GiB, 430,825 -> 371,023 tokens against a max-model-len of 262,144, so the pool still holds 1.4x a full-length sequence. What is lost is concurrent long requests, not context. Verified through the gateway rather than at the container: char-rp returns 200 with content, char-rp-reasoning returns 200 with both content and reasoning_content populated. Seat is healthy with RestartCount 0. The arithmetic and the "check used_memory, not the flag" warning are written into the env template, because the next person to raise either budget needs to lower the other in the same change. --- stacks/meromero-charrp/.env.example | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/stacks/meromero-charrp/.env.example b/stacks/meromero-charrp/.env.example index 5ac8ce2..f6844ad 100644 --- a/stacks/meromero-charrp/.env.example +++ b/stacks/meromero-charrp/.env.example @@ -1,4 +1,22 @@ # ana-ml2 GPU0 char-rp prose seat (MeroMero-v2). Real .env lives on the host. -MEROMERO_GPU_MEM_UTIL=0.52 +# +# ⚠ GPU0 IS SHARED WITH `vllm-gen` AND THE BUDGETS MUST SUM UNDER ~0.92. +# gen runs at --gpu-memory-utilization 0.43 but actually holds ~45.6 GiB of the +# 94.97 GiB card, because the utilization figure sizes the KV cache and does not +# cover CUDA context, graphs and non-torch overhead. This seat was at 0.51: +# 0.43 + 0.51 = 0.94, which left ~0.6 GiB of real headroom and worked right up +# until it did not. On 2026-08-24 it stopped fitting and the seat entered a +# crash-loop — 13 restarts, `torch.OutOfMemoryError: ... 195.19 MiB is free`, +# both the `char-rp` and `char-rp-reasoning` gateway aliases returning 500s +# (they share this one seat; see the LiteLLM routes for :8016). +# +# 0.47 restores a real margin (~4.8 GiB free on the card) and costs nothing you +# can use: KV cache goes 27.36 GiB -> 23.56 GiB, which is 430,825 -> 371,023 +# tokens against a max-model-len of 262,144. The pool still holds 1.4x a +# full-length sequence; what you lose is concurrent long requests, not context. +# +# Raising this again means lowering gen's in the same change. Check the real +# numbers, not the flags: `nvidia-smi --query-compute-apps=pid,used_memory`. +MEROMERO_GPU_MEM_UTIL=0.47 MEROMERO_MAX_MODEL_LEN=262144 MEROMERO_GPU_ID=0