memory: snapshot — A6000 window closed; my dots-tts hypothesis was wrong

Operator freed ComfyUI's VRAM directly, so tts-dev is unblocked and the
window request is withdrawn with comfy-dev.

- Verified it was a model unload, not a stop: comfyui still up 8 days,
  same pid, HTTP 200, 18,500 -> 612 MiB. Told comfy-dev explicitly so a
  VRAM drop is not misread as a restart of their service.
- The resulting 43.8 GB free is a snapshot, not a floor. ComfyUI is live
  and reloads ~18.5 GB on the next render, which puts the real floor at
  ~25.3 GB against FireRedAudio's ~26 GB requirement. The coordination
  shrank from "stop ComfyUI" to "don't render during the bench" rather
  than disappearing. Flagged to both; not volunteered on comfy-dev's
  behalf.
- Withdrew my caching-allocator hypothesis for the dots-tts VRAM. tts-dev
  identified it as their prompt-feature cache, capped at 32 entries on
  2026-08-14 after two production incidents. A named mechanism with an
  incident history beats a plausible story, and the useful finding is
  that 14.43 GB sits inside a cap they deliberately chose.

Read-only probes; nothing on the box was changed.

Memory-only; no version bump per the SemVer SKIP list.

persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md

@@ -12,22 +12,21 @@ Measured **2026-09-01T18:05:58-07:00**, PID -> container mapped via
       4012952   18,500 MiB   comfyui           up  8d09h
       3166988    4,722 MiB   studio-gate       up 19d09h
 
-## ⚠ dots-tts is 14,430 MiB; the burn-in recorded ~6 GB
+## ⚠ dots-tts is 14,430 MiB; the burn-in recorded ~6 GB — CAUSE IDENTIFIED BY tts-dev
 
-2.4x the figure in [[reference_omnivoice_streaming_tts]] / the 2026-08-09→10 dots burn-in.
+2.4x the figure in the 2026-08-09→10 dots burn-in.
-**Hypothesis, not a finding: PyTorch's caching allocator never returns memory to the driver,
-so the process holds its high-water mark for the life of the container.** dots runs
-`optimize=True` (torch.compile + CUDA-graph capture, which allocates aggressively) and this
-process has been up 17 days — one unusual request weeks ago explains the whole delta and it
-would never come back down.
 
-**Cheap discriminating test (tts-dev's domain, handed to them):** restart `dots-tts` and read
+**Cause (tts-dev, 2026-09-01): their prompt-feature cache**, diagnosed and **capped at 32
-the resident figure. Drops to ~6 GB = allocator high-water mark, mitigate with a bound or a
+entries on 2026-08-14 after two production incidents**. The cap is holding in the sense that
-periodic recycle, not a leak hunt. Stays near 14 GB = genuinely resident and the ~6 GB doc
+nothing has crashed, but 14.43 GB says the ceiling they chose still permits a long way of
-figure is wrong for the deployed config.
+growth. Theirs to own; no infra-ops action.
 
-⚠ **Do NOT reach for `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` reflexively** — our own
+⚠ **infra-ops' hypothesis here was WRONG and is withdrawn** — I proposed a PyTorch
-notes record it **conflicts with `optimize=True`** on dots (CUDA-graph `curr_block` error).
+caching-allocator high-water mark from 17 days of uptime, with a restart as the discriminating
+test. Plausible, and not what it is. tts-dev had an actual mechanism with a date, a cap and an
+incident history attached. **A named mechanism beats a plausible story; do not re-propose the
+allocator theory.** (The `expandable_segments` conflict with `optimize=True` on dots is still
+true and still worth not tripping over, but it is not the explanation here.)
 
 ## The "~4 GB unaccounted" on the 3090 was two different things
 
@@ -46,7 +45,29 @@ so **A6000 = cuda:0 natively**. `device_ids: ["1"]` in a container = A6000 (corr
 native `CUDA_VISIBLE_DEVICES=1` = the **3090** (wrong card). Native path needs
 `CUDA_DEVICE_ORDER=PCI_BUS_ID`. → [[reference_irv_ml1_gpu_r14]]
 
-## Open — A6000 window request, brokered not decided
+## ✅ CLOSED — A6000 window WITHDRAWN 2026-09-01T18:09 (operator freed the VRAM instead)
+
+The operator freed ComfyUI's VRAM directly, unblocking tts-dev without anyone stopping
+anything. Request retracted with comfy-dev; nothing scheduled.
+
+⚠ **It was a MODEL UNLOAD, not a stop.** Verified 2026-09-01T18:09:15-07:00: `comfyui` still
+`Up 8 days (healthy)`, **same pid 4012952** (no restart), `/system_stats` HTTP 200, A6000
+resident 18,500 MiB -> 612 MiB. No state lost; the only cost is a cold model reload off
+`/storetank` on the next render. Told comfy-dev explicitly so a VRAM drop is not misread as a
+container restart.
+
+⚠ **THE 43.8 GB IS A SNAPSHOT, NOT A FLOOR — the coordination shrank, it did not vanish.**
+ComfyUI is live and reloads its ~18.5 GB on the next render. Worst case during the bench:
+
+    A6000 free (idle)     ~43.8 GB
+    ComfyUI reloads        18.5 GB
+    = actual floor        ~25.3 GB    vs FireRedAudio's ~26 GB   <- BELOW requirement
+
+So the ask changed from "stop ComfyUI for a window" to "**do not render during the bench**".
+Flagged to both; **deliberately NOT volunteered on comfy-dev's behalf** — tts-dev must ask,
+comfy-dev grants or declines. Open on their side, not infra-ops'.
+
+## Original ask, for the record
 
 tts-dev has operator approval for a two-engine TTS bench (FireRedAudio, then Breeze-TTS-2) and
 needs `comfyui` stopped for the duration; FireRedAudio wants ~26 GB (21.2 GB backbone + 8.4 GB

persistent-memory.md

@@ -135,15 +135,15 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
   instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2
   without deciding that first. Exactly TWO 3c launches, only one died.
   → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
-- **⏸ A6000 WINDOW REQUEST — brokered with comfy-dev, awaiting their timing.** tts-dev needs
+- **✅ A6000 window CLOSED 2026-09-01 — operator freed ComfyUI's VRAM instead of anyone
-  `comfyui` stopped on irv-ml1's A6000 for an operator-approved TTS bench (FireRedAudio, then
+  stopping it.** ⚠ It was a **model unload, not a stop**: `comfyui` still up 8 days, same pid,
-  Breeze-TTS-2); ~26 GB needed against ~25.9 GB free = a coin flip, and stopping comfyui gives
+  HTTP 200, 18,500 -> 612 MiB. ⚠ **The resulting ~43.8 GB free is a snapshot, not a floor** —
-  ~44.4 GB. Their build is a day or two out. **Not granted unilaterally** — comfyui is
+  ComfyUI reloads ~18.5 GB on the next render, putting the real floor at ~25.3 GB against
-  comfy-dev's and they are mid-migration. Bench provisioning ruled: **plain container under
+  FireRedAudio's ~26 GB. Coordination shrank to "don't render during the bench"; tts-dev's ask
-  `lkraven`, not `/opt/docker/compose/`**. ⚠ Also measured: **`dots-tts` is holding 14,430 MiB
+  to make, comfy-dev's to grant. Also measured: **`dots-tts` holds 14,430 MiB vs a ~6 GB
-  against a recorded ~6 GB** (2.4x — likely a PyTorch caching-allocator high-water mark after
+  baseline** — tts-dev's **prompt-feature cache**, capped at 32 entries 2026-08-14 after two
-  17 days up), and the 3090 is at 76% behind a warn-only watchdog.
+  incidents (infra-ops' caching-allocator hypothesis was wrong, withdrawn). 3090 at 76% behind
-  → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md`
+  a warn-only watchdog. → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md`
 - **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a)
   `zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving.
   Branch (b) rejected on the trade (comfy-dev corrected the *reason*: only comfyui mounts