diff --git a/persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md b/persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md index 424bf04..afe0370 100644 --- a/persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md +++ b/persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md @@ -12,22 +12,21 @@ Measured **2026-09-01T18:05:58-07:00**, PID -> container mapped via 4012952 18,500 MiB comfyui up 8d09h 3166988 4,722 MiB studio-gate up 19d09h -## ⚠ dots-tts is 14,430 MiB; the burn-in recorded ~6 GB +## ⚠ dots-tts is 14,430 MiB; the burn-in recorded ~6 GB — CAUSE IDENTIFIED BY tts-dev -2.4x the figure in [[reference_omnivoice_streaming_tts]] / the 2026-08-09→10 dots burn-in. -**Hypothesis, not a finding: PyTorch's caching allocator never returns memory to the driver, -so the process holds its high-water mark for the life of the container.** dots runs -`optimize=True` (torch.compile + CUDA-graph capture, which allocates aggressively) and this -process has been up 17 days — one unusual request weeks ago explains the whole delta and it -would never come back down. +2.4x the figure in the 2026-08-09→10 dots burn-in. -**Cheap discriminating test (tts-dev's domain, handed to them):** restart `dots-tts` and read -the resident figure. Drops to ~6 GB = allocator high-water mark, mitigate with a bound or a -periodic recycle, not a leak hunt. Stays near 14 GB = genuinely resident and the ~6 GB doc -figure is wrong for the deployed config. +**Cause (tts-dev, 2026-09-01): their prompt-feature cache**, diagnosed and **capped at 32 +entries on 2026-08-14 after two production incidents**. The cap is holding in the sense that +nothing has crashed, but 14.43 GB says the ceiling they chose still permits a long way of +growth. Theirs to own; no infra-ops action. -⚠ **Do NOT reach for `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` reflexively** — our own -notes record it **conflicts with `optimize=True`** on dots (CUDA-graph `curr_block` error). +⚠ **infra-ops' hypothesis here was WRONG and is withdrawn** — I proposed a PyTorch +caching-allocator high-water mark from 17 days of uptime, with a restart as the discriminating +test. Plausible, and not what it is. tts-dev had an actual mechanism with a date, a cap and an +incident history attached. **A named mechanism beats a plausible story; do not re-propose the +allocator theory.** (The `expandable_segments` conflict with `optimize=True` on dots is still +true and still worth not tripping over, but it is not the explanation here.) ## The "~4 GB unaccounted" on the 3090 was two different things @@ -46,7 +45,29 @@ so **A6000 = cuda:0 natively**. `device_ids: ["1"]` in a container = A6000 (corr native `CUDA_VISIBLE_DEVICES=1` = the **3090** (wrong card). Native path needs `CUDA_DEVICE_ORDER=PCI_BUS_ID`. → [[reference_irv_ml1_gpu_r14]] -## Open — A6000 window request, brokered not decided +## ✅ CLOSED — A6000 window WITHDRAWN 2026-09-01T18:09 (operator freed the VRAM instead) + +The operator freed ComfyUI's VRAM directly, unblocking tts-dev without anyone stopping +anything. Request retracted with comfy-dev; nothing scheduled. + +⚠ **It was a MODEL UNLOAD, not a stop.** Verified 2026-09-01T18:09:15-07:00: `comfyui` still +`Up 8 days (healthy)`, **same pid 4012952** (no restart), `/system_stats` HTTP 200, A6000 +resident 18,500 MiB -> 612 MiB. No state lost; the only cost is a cold model reload off +`/storetank` on the next render. Told comfy-dev explicitly so a VRAM drop is not misread as a +container restart. + +⚠ **THE 43.8 GB IS A SNAPSHOT, NOT A FLOOR — the coordination shrank, it did not vanish.** +ComfyUI is live and reloads its ~18.5 GB on the next render. Worst case during the bench: + + A6000 free (idle) ~43.8 GB + ComfyUI reloads 18.5 GB + = actual floor ~25.3 GB vs FireRedAudio's ~26 GB <- BELOW requirement + +So the ask changed from "stop ComfyUI for a window" to "**do not render during the bench**". +Flagged to both; **deliberately NOT volunteered on comfy-dev's behalf** — tts-dev must ask, +comfy-dev grants or declines. Open on their side, not infra-ops'. + +## Original ask, for the record tts-dev has operator approval for a two-engine TTS bench (FireRedAudio, then Breeze-TTS-2) and needs `comfyui` stopped for the duration; FireRedAudio wants ~26 GB (21.2 GB backbone + 8.4 GB diff --git a/persistent-memory.md b/persistent-memory.md index 39b1d8f..d87ff4e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -135,15 +135,15 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2 without deciding that first. Exactly TWO 3c launches, only one died. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md` -- **⏸ A6000 WINDOW REQUEST — brokered with comfy-dev, awaiting their timing.** tts-dev needs - `comfyui` stopped on irv-ml1's A6000 for an operator-approved TTS bench (FireRedAudio, then - Breeze-TTS-2); ~26 GB needed against ~25.9 GB free = a coin flip, and stopping comfyui gives - ~44.4 GB. Their build is a day or two out. **Not granted unilaterally** — comfyui is - comfy-dev's and they are mid-migration. Bench provisioning ruled: **plain container under - `lkraven`, not `/opt/docker/compose/`**. ⚠ Also measured: **`dots-tts` is holding 14,430 MiB - against a recorded ~6 GB** (2.4x — likely a PyTorch caching-allocator high-water mark after - 17 days up), and the 3090 is at 76% behind a warn-only watchdog. - → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md` +- **✅ A6000 window CLOSED 2026-09-01 — operator freed ComfyUI's VRAM instead of anyone + stopping it.** ⚠ It was a **model unload, not a stop**: `comfyui` still up 8 days, same pid, + HTTP 200, 18,500 -> 612 MiB. ⚠ **The resulting ~43.8 GB free is a snapshot, not a floor** — + ComfyUI reloads ~18.5 GB on the next render, putting the real floor at ~25.3 GB against + FireRedAudio's ~26 GB. Coordination shrank to "don't render during the bench"; tts-dev's ask + to make, comfy-dev's to grant. Also measured: **`dots-tts` holds 14,430 MiB vs a ~6 GB + baseline** — tts-dev's **prompt-feature cache**, capped at 32 entries 2026-08-14 after two + incidents (infra-ops' caching-allocator hypothesis was wrong, withdrawn). 3090 at 76% behind + a warn-only watchdog. → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md` - **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a) `zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving. Branch (b) rejected on the trade (comfy-dev corrected the *reason*: only comfyui mounts