memory: snapshot — irv-ml1 GPU resident map; dots-tts at 2.4x its recorded VRAM

tts-dev asked for an A6000 window for an approved TTS bench and flagged a
3090 VRAM delta. Probed the box and mapped PID to container rather than
taking the reported figures.

- The 18.5 GB process they attributed to the 3090 is comfyui, on the
  A6000. And it is 18.5 GB rather than the ~11.8 GB they budgeted, so
  stopping it gives ~44.4 GB free, not the tight margin they expected.
- Their "~4 GB unaccounted" on the 3090 is two things: parakeet is a
  third tenant the doc figure never counted, and dots-tts alone is
  holding 14,430 MiB against a burn-in figure of ~6 GB. The second is
  the larger finding and it is theirs to act on; handed over with a
  caching-allocator hypothesis and a one-restart discriminating test.
- Restated the GPU ordering foot-gun: device_ids ["1"] is the A6000 in a
  container, but a bare native CUDA_VISIBLE_DEVICES=1 gets the 3090.

Window not granted unilaterally — comfyui is comfy-dev's and they are
mid-migration, so the request went to them directly and infra-ops relays.
Ruled that the bench runs as a plain container under lkraven rather than
under /opt/docker/compose/, which is for deployed stacks and would leave
a canonical entry reporting as drift until deleted.

Read-only probes; nothing on the box was changed.

Memory-only; no version bump per the SemVer SKIP list.

persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md

@@ -0,0 +1,70 @@
+# `[2026-09-01]` irv-ml1 GPU resident map — and dots-tts is holding 2.4x its recorded VRAM
+
+Measured **2026-09-01T18:05:58-07:00**, PID -> container mapped via
+`/proc/<pid>/cgroup` + `docker inspect .State.Pid`, not inferred.
+
+    GPU 0  RTX 3090   18,727 / 24,576 MiB   (~5.8 GB free, 76% used)
+      1852662   14,430 MiB   dots-tts          up 17d21h
+      3739643    3,686 MiB   omnivoice-ref     up 19d02h
+       116892      582 MiB   parakeet          up 49d01h
+
+    GPU 1  RTX A6000  23,253 / 49,140 MiB   (~25.9 GB free)
+      4012952   18,500 MiB   comfyui           up  8d09h
+      3166988    4,722 MiB   studio-gate       up 19d09h
+
+## ⚠ dots-tts is 14,430 MiB; the burn-in recorded ~6 GB
+
+2.4x the figure in [[reference_omnivoice_streaming_tts]] / the 2026-08-09→10 dots burn-in.
+**Hypothesis, not a finding: PyTorch's caching allocator never returns memory to the driver,
+so the process holds its high-water mark for the life of the container.** dots runs
+`optimize=True` (torch.compile + CUDA-graph capture, which allocates aggressively) and this
+process has been up 17 days — one unusual request weeks ago explains the whole delta and it
+would never come back down.
+
+**Cheap discriminating test (tts-dev's domain, handed to them):** restart `dots-tts` and read
+the resident figure. Drops to ~6 GB = allocator high-water mark, mitigate with a bound or a
+periodic recycle, not a leak hunt. Stays near 14 GB = genuinely resident and the ~6 GB doc
+figure is wrong for the deployed config.
+
+⚠ **Do NOT reach for `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` reflexively** — our own
+notes record it **conflicts with `optimize=True`** on dots (CUDA-graph `curr_block` error).
+
+## The "~4 GB unaccounted" on the 3090 was two different things
+
+tts-dev flagged 18.7 GB against a ~14.6 GB doc figure for "dots + omnivoice-ref". Decomposes:
+
+- **`parakeet` is a third tenant that figure never counted** (582 MiB). Not creep — an
+  uncounted tenant. The doc is wrong, not the card.
+- **The rest is the dots-tts growth above**, which is the larger and more interesting half.
+
+They also mis-located the 18.5 GB process: it is `comfyui` on the **A6000**, not the 3090.
+
+## ⚠ GPU ordering, restated because it keeps costing time
+
+nvidia-smi indexes **3090 = 0, A6000 = 1**. Native CUDA on this host defaults to fastest-first,
+so **A6000 = cuda:0 natively**. `device_ids: ["1"]` in a container = A6000 (correct); a bare
+native `CUDA_VISIBLE_DEVICES=1` = the **3090** (wrong card). Native path needs
+`CUDA_DEVICE_ORDER=PCI_BUS_ID`. → [[reference_irv_ml1_gpu_r14]]
+
+## Open — A6000 window request, brokered not decided
+
+tts-dev has operator approval for a two-engine TTS bench (FireRedAudio, then Breeze-TTS-2) and
+needs `comfyui` stopped for the duration; FireRedAudio wants ~26 GB (21.2 GB backbone + 8.4 GB
+decoder) against ~25.9 GB free, which is a coin flip. Stopping comfyui gives **~44.4 GB**.
+Their build is a day or two out; nothing blocked today.
+
+**Not granted unilaterally** — comfyui is comfy-dev's and they are mid-migration (~112 GB batch
+onboarding, cutover window unset). Request put to comfy-dev directly; infra-ops relays rather
+than sending tts-dev chasing. → [[feedback_credential_broker]]
+
+**Ruling on provisioning (infra-ops call):** the bench runs as a **plain container under
+`lkraven`**, NOT under `/opt/docker/compose/`. That tree is for deployed stacks — canonical
+copy in the eshpfi repo, Homepage labels, restart policy, `sync-stacks.sh` drift coverage. A
+bench that comes down when it ends is none of those, and provisioning it as a stack creates a
+canonical entry that reports as drift until someone remembers to delete it.
+
+⚠ **This may dissolve rather than need scheduling:** comfyui is leaving irv-ml1 for the R750xa,
+after which the A6000 is permanently free. Told tts-dev explicitly NOT to plan around it — the
+box has not arrived and the window is unset. → [[2026-09-01-ada-migration-branch-a]]
+
+Threads: tts-dev `01M1FTB0TEP4T7P21B511F0CS3`.