vh
850e0c3351
`vllm-meromero-rp` had been restarting since 2026-08-24 18:2x, 13 times by the time it was looked at, taking both the `char-rp` and `char-rp-reasoning` gateway aliases down with it (they resolve to the same seat on :8016 — hosted_vllm/char-rp and hosted_vllm/char-rp-thinking). Root cause is CUDA OOM on ana-ml2 GPU0, which the startup logs hide well: the engine gets through weights, torch.compile and CUDA-graph capture looking entirely healthy, then dies at KV-cache allocation with `torch.OutOfMemoryError: ... 195.19 MiB is free`. GPU0 is shared with `vllm-gen`. gen is configured at 0.43 but actually holds ~45.6 GiB of the 94.97 GiB card, because --gpu-memory-utilization sizes the KV cache and does not account for CUDA context, graphs and non-torch overhead. This seat was at 0.51, so the pair was committed to 0.94 of the card with about 0.6 GiB of real headroom. That fit on 08-21 and stopped fitting today. 0.47 restores ~4.8 GiB of margin and costs nothing usable: KV cache 27.36 -> 23.56 GiB, 430,825 -> 371,023 tokens against a max-model-len of 262,144, so the pool still holds 1.4x a full-length sequence. What is lost is concurrent long requests, not context. Verified through the gateway rather than at the container: char-rp returns 200 with content, char-rp-reasoning returns 200 with both content and reasoning_content populated. Seat is healthy with RestartCount 0. The arithmetic and the "check used_memory, not the flag" warning are written into the env template, because the next person to raise either budget needs to lower the other in the same change.
23 lines
1.3 KiB
Bash
23 lines
1.3 KiB
Bash
# ana-ml2 GPU0 char-rp prose seat (MeroMero-v2). Real .env lives on the host.
|
|
#
|
|
# ⚠ GPU0 IS SHARED WITH `vllm-gen` AND THE BUDGETS MUST SUM UNDER ~0.92.
|
|
# gen runs at --gpu-memory-utilization 0.43 but actually holds ~45.6 GiB of the
|
|
# 94.97 GiB card, because the utilization figure sizes the KV cache and does not
|
|
# cover CUDA context, graphs and non-torch overhead. This seat was at 0.51:
|
|
# 0.43 + 0.51 = 0.94, which left ~0.6 GiB of real headroom and worked right up
|
|
# until it did not. On 2026-08-24 it stopped fitting and the seat entered a
|
|
# crash-loop — 13 restarts, `torch.OutOfMemoryError: ... 195.19 MiB is free`,
|
|
# both the `char-rp` and `char-rp-reasoning` gateway aliases returning 500s
|
|
# (they share this one seat; see the LiteLLM routes for :8016).
|
|
#
|
|
# 0.47 restores a real margin (~4.8 GiB free on the card) and costs nothing you
|
|
# can use: KV cache goes 27.36 GiB -> 23.56 GiB, which is 430,825 -> 371,023
|
|
# tokens against a max-model-len of 262,144. The pool still holds 1.4x a
|
|
# full-length sequence; what you lose is concurrent long requests, not context.
|
|
#
|
|
# Raising this again means lowering gen's in the same change. Check the real
|
|
# numbers, not the flags: `nvidia-smi --query-compute-apps=pid,used_memory`.
|
|
MEROMERO_GPU_MEM_UTIL=0.47
|
|
MEROMERO_MAX_MODEL_LEN=262144
|
|
MEROMERO_GPU_ID=0
|