42 lines
2.4 KiB
Markdown
42 lines
2.4 KiB
Markdown
# `[2026-09-04]` `gen` moved to ana-ml2 GPU0 — and I sized it against the wrong number, twice in one hour
|
|
|
|
`vllm-embed` had OOM-crashed **7 times** (`RestartCount 7`): GPU1 was at **216 MiB free** of
|
|
97,887, and a ~3.4 GB tenant sharing a card with five other seats dies when it cannot get another
|
|
96 MiB mid-inference. Root cause of the `qwen3-embed` 500s brokkr saw — **not** the LiteLLM
|
|
gateway restart he attributed them to (different host, different component, 50 min earlier, and
|
|
six of the seven crashes predate it).
|
|
|
|
**Operator ruling:** move `gen` to GPU0. *"we were keeping it free for training, but we don't have
|
|
the power for sustained load."* GPU0 was never free — `mog-sec` has been there at util 0.52
|
|
(55,126 MiB) since the August move.
|
|
|
|
## ⚠ THE SIZING ERROR, WHICH IS THE POINT OF THIS ENTRY
|
|
|
|
gen at util 0.43 wants 42,091 MiB GPU0 free 42,113 margin 22 MiB — would not start
|
|
gen at util 0.41 wants 40,134 MiB margin 1,979 MiB — I called this safe. IT WAS NOT.
|
|
result 1,548 MiB free, 56 FlashInfer autotuner OOM-fallbacks per 3 min
|
|
gen at util 0.38 34,558 MiB actual 4,466-7,548 MiB free ZERO OOM events over 6 min
|
|
|
|
**`gpu-memory-utilization` governs vLLM's declared weights+KV budget. It does not cover what the
|
|
process actually needs at runtime** — FlashInfer JIT autotuner workspace, CUDA graphs, expandable
|
|
segments all allocate on top. My "1,979 MiB margin" was against the declared budget, so the seat
|
|
came up and then silently fell back to slower kernels for want of 20 MB chunks.
|
|
|
|
⚠ **Same class of error as the DAC plateau the same day: checked a proxy, reported it as the
|
|
thing.** The fix was to *measure* — count OOM-fallback events before and after — not to re-derive.
|
|
|
|
## Final state and what it cost
|
|
|
|
GPU0 mog-sec 55,126 + gen 34,558 = 89,701 / 97,887
|
|
GPU1 50,507 / 97,887 — 46.7 GB free, was 216 MiB
|
|
|
|
`gen` KV cache is now **348,515 → 268,205 tokens, 1.02x concurrency at its 262,144 max-model-len**
|
|
— it holds exactly one full-context request. Short/medium requests still batch; long-context
|
|
throughput effectively serialises. The lever to reclaim it is `MOG_GPU_MEM_UTIL 0.52`, untouched.
|
|
|
|
.env.bak-preGPU0-20260904-164032 the GPU move
|
|
.env.bak-preShrink-165133 the utilization
|
|
|
|
⚠ `/opt/docker/compose/*/.env` is root-owned — `docker compose restart` needs **sudo** on
|
|
ana-docker/ana-ml2 or it fails with a bare `permission denied` on the .env.
|