From e91278799dcfc9a8e66dc4085331824e9caf55a1 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 1 Sep 2026 18:08:14 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20irv-ml1=20GPU?= =?UTF-8?q?=20resident=20map;=20dots-tts=20at=202.4x=20its=20recorded=20VR?= =?UTF-8?q?AM?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit tts-dev asked for an A6000 window for an approved TTS bench and flagged a 3090 VRAM delta. Probed the box and mapped PID to container rather than taking the reported figures. - The 18.5 GB process they attributed to the 3090 is comfyui, on the A6000. And it is 18.5 GB rather than the ~11.8 GB they budgeted, so stopping it gives ~44.4 GB free, not the tight margin they expected. - Their "~4 GB unaccounted" on the 3090 is two things: parakeet is a third tenant the doc figure never counted, and dots-tts alone is holding 14,430 MiB against a burn-in figure of ~6 GB. The second is the larger finding and it is theirs to act on; handed over with a caching-allocator hypothesis and a one-restart discriminating test. - Restated the GPU ordering foot-gun: device_ids ["1"] is the A6000 in a container, but a bare native CUDA_VISIBLE_DEVICES=1 gets the 3090. Window not granted unilaterally — comfyui is comfy-dev's and they are mid-migration, so the request went to them directly and infra-ops relays. Ruled that the bench runs as a plain container under lkraven rather than under /opt/docker/compose/, which is for deployed stacks and would leave a canonical entry reporting as drift until deleted. Read-only probes; nothing on the box was changed. Memory-only; no version bump per the SemVer SKIP list. --- .../2026-09-01-irv-ml1-gpu-residents.md | 70 +++++++++++++++++++ persistent-memory.md | 9 +++ 2 files changed, 79 insertions(+) create mode 100644 persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md diff --git a/persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md b/persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md new file mode 100644 index 0000000..424bf04 --- /dev/null +++ b/persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md @@ -0,0 +1,70 @@ +# `[2026-09-01]` irv-ml1 GPU resident map — and dots-tts is holding 2.4x its recorded VRAM + +Measured **2026-09-01T18:05:58-07:00**, PID -> container mapped via +`/proc//cgroup` + `docker inspect .State.Pid`, not inferred. + + GPU 0 RTX 3090 18,727 / 24,576 MiB (~5.8 GB free, 76% used) + 1852662 14,430 MiB dots-tts up 17d21h + 3739643 3,686 MiB omnivoice-ref up 19d02h + 116892 582 MiB parakeet up 49d01h + + GPU 1 RTX A6000 23,253 / 49,140 MiB (~25.9 GB free) + 4012952 18,500 MiB comfyui up 8d09h + 3166988 4,722 MiB studio-gate up 19d09h + +## ⚠ dots-tts is 14,430 MiB; the burn-in recorded ~6 GB + +2.4x the figure in [[reference_omnivoice_streaming_tts]] / the 2026-08-09→10 dots burn-in. +**Hypothesis, not a finding: PyTorch's caching allocator never returns memory to the driver, +so the process holds its high-water mark for the life of the container.** dots runs +`optimize=True` (torch.compile + CUDA-graph capture, which allocates aggressively) and this +process has been up 17 days — one unusual request weeks ago explains the whole delta and it +would never come back down. + +**Cheap discriminating test (tts-dev's domain, handed to them):** restart `dots-tts` and read +the resident figure. Drops to ~6 GB = allocator high-water mark, mitigate with a bound or a +periodic recycle, not a leak hunt. Stays near 14 GB = genuinely resident and the ~6 GB doc +figure is wrong for the deployed config. + +⚠ **Do NOT reach for `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` reflexively** — our own +notes record it **conflicts with `optimize=True`** on dots (CUDA-graph `curr_block` error). + +## The "~4 GB unaccounted" on the 3090 was two different things + +tts-dev flagged 18.7 GB against a ~14.6 GB doc figure for "dots + omnivoice-ref". Decomposes: + +- **`parakeet` is a third tenant that figure never counted** (582 MiB). Not creep — an + uncounted tenant. The doc is wrong, not the card. +- **The rest is the dots-tts growth above**, which is the larger and more interesting half. + +They also mis-located the 18.5 GB process: it is `comfyui` on the **A6000**, not the 3090. + +## ⚠ GPU ordering, restated because it keeps costing time + +nvidia-smi indexes **3090 = 0, A6000 = 1**. Native CUDA on this host defaults to fastest-first, +so **A6000 = cuda:0 natively**. `device_ids: ["1"]` in a container = A6000 (correct); a bare +native `CUDA_VISIBLE_DEVICES=1` = the **3090** (wrong card). Native path needs +`CUDA_DEVICE_ORDER=PCI_BUS_ID`. → [[reference_irv_ml1_gpu_r14]] + +## Open — A6000 window request, brokered not decided + +tts-dev has operator approval for a two-engine TTS bench (FireRedAudio, then Breeze-TTS-2) and +needs `comfyui` stopped for the duration; FireRedAudio wants ~26 GB (21.2 GB backbone + 8.4 GB +decoder) against ~25.9 GB free, which is a coin flip. Stopping comfyui gives **~44.4 GB**. +Their build is a day or two out; nothing blocked today. + +**Not granted unilaterally** — comfyui is comfy-dev's and they are mid-migration (~112 GB batch +onboarding, cutover window unset). Request put to comfy-dev directly; infra-ops relays rather +than sending tts-dev chasing. → [[feedback_credential_broker]] + +**Ruling on provisioning (infra-ops call):** the bench runs as a **plain container under +`lkraven`**, NOT under `/opt/docker/compose/`. That tree is for deployed stacks — canonical +copy in the eshpfi repo, Homepage labels, restart policy, `sync-stacks.sh` drift coverage. A +bench that comes down when it ends is none of those, and provisioning it as a stack creates a +canonical entry that reports as drift until someone remembers to delete it. + +⚠ **This may dissolve rather than need scheduling:** comfyui is leaving irv-ml1 for the R750xa, +after which the A6000 is permanently free. Told tts-dev explicitly NOT to plan around it — the +box has not arrived and the window is unset. → [[2026-09-01-ada-migration-branch-a]] + +Threads: tts-dev `01M1FTB0TEP4T7P21B511F0CS3`. diff --git a/persistent-memory.md b/persistent-memory.md index d260d3e..39b1d8f 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -135,6 +135,15 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2 without deciding that first. Exactly TWO 3c launches, only one died. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md` +- **⏸ A6000 WINDOW REQUEST — brokered with comfy-dev, awaiting their timing.** tts-dev needs + `comfyui` stopped on irv-ml1's A6000 for an operator-approved TTS bench (FireRedAudio, then + Breeze-TTS-2); ~26 GB needed against ~25.9 GB free = a coin flip, and stopping comfyui gives + ~44.4 GB. Their build is a day or two out. **Not granted unilaterally** — comfyui is + comfy-dev's and they are mid-migration. Bench provisioning ruled: **plain container under + `lkraven`, not `/opt/docker/compose/`**. ⚠ Also measured: **`dots-tts` is holding 14,430 MiB + against a recorded ~6 GB** (2.4x — likely a PyTorch caching-allocator high-water mark after + 17 days up), and the 3090 is at 76% behind a warn-only watchdog. + → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md` - **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a) `zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving. Branch (b) rejected on the trade (comfy-dev corrected the *reason*: only comfyui mounts