memory: snapshot — A6000 window closed; my dots-tts hypothesis was wrong

Operator freed ComfyUI's VRAM directly, so tts-dev is unblocked and the
window request is withdrawn with comfy-dev.

- Verified it was a model unload, not a stop: comfyui still up 8 days,
  same pid, HTTP 200, 18,500 -> 612 MiB. Told comfy-dev explicitly so a
  VRAM drop is not misread as a restart of their service.
- The resulting 43.8 GB free is a snapshot, not a floor. ComfyUI is live
  and reloads ~18.5 GB on the next render, which puts the real floor at
  ~25.3 GB against FireRedAudio's ~26 GB requirement. The coordination
  shrank from "stop ComfyUI" to "don't render during the bench" rather
  than disappearing. Flagged to both; not volunteered on comfy-dev's
  behalf.
- Withdrew my caching-allocator hypothesis for the dots-tts VRAM. tts-dev
  identified it as their prompt-feature cache, capped at 32 entries on
  2026-08-14 after two production incidents. A named mechanism with an
  incident history beats a plausible story, and the useful finding is
  that 14.43 GB sits inside a cap they deliberately chose.

Read-only probes; nothing on the box was changed.

Memory-only; no version bump per the SemVer SKIP list.
This commit is contained in:
vh
2026-09-01 18:10:34 -07:00
parent e91278799d
commit 46a3c63706
2 changed files with 44 additions and 23 deletions
@@ -12,22 +12,21 @@ Measured **2026-09-01T18:05:58-07:00**, PID -> container mapped via
4012952 18,500 MiB comfyui up 8d09h
3166988 4,722 MiB studio-gate up 19d09h
## ⚠ dots-tts is 14,430 MiB; the burn-in recorded ~6 GB
## ⚠ dots-tts is 14,430 MiB; the burn-in recorded ~6 GB — CAUSE IDENTIFIED BY tts-dev
2.4x the figure in [[reference_omnivoice_streaming_tts]] / the 2026-08-09→10 dots burn-in.
**Hypothesis, not a finding: PyTorch's caching allocator never returns memory to the driver,
so the process holds its high-water mark for the life of the container.** dots runs
`optimize=True` (torch.compile + CUDA-graph capture, which allocates aggressively) and this
process has been up 17 days — one unusual request weeks ago explains the whole delta and it
would never come back down.
2.4x the figure in the 2026-08-09→10 dots burn-in.
**Cheap discriminating test (tts-dev's domain, handed to them):** restart `dots-tts` and read
the resident figure. Drops to ~6 GB = allocator high-water mark, mitigate with a bound or a
periodic recycle, not a leak hunt. Stays near 14 GB = genuinely resident and the ~6 GB doc
figure is wrong for the deployed config.
**Cause (tts-dev, 2026-09-01): their prompt-feature cache**, diagnosed and **capped at 32
entries on 2026-08-14 after two production incidents**. The cap is holding in the sense that
nothing has crashed, but 14.43 GB says the ceiling they chose still permits a long way of
growth. Theirs to own; no infra-ops action.
⚠ **Do NOT reach for `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` reflexively** — our own
notes record it **conflicts with `optimize=True`** on dots (CUDA-graph `curr_block` error).
⚠ **infra-ops' hypothesis here was WRONG and is withdrawn** — I proposed a PyTorch
caching-allocator high-water mark from 17 days of uptime, with a restart as the discriminating
test. Plausible, and not what it is. tts-dev had an actual mechanism with a date, a cap and an
incident history attached. **A named mechanism beats a plausible story; do not re-propose the
allocator theory.** (The `expandable_segments` conflict with `optimize=True` on dots is still
true and still worth not tripping over, but it is not the explanation here.)
## The "~4 GB unaccounted" on the 3090 was two different things
@@ -46,7 +45,29 @@ so **A6000 = cuda:0 natively**. `device_ids: ["1"]` in a container = A6000 (corr
native `CUDA_VISIBLE_DEVICES=1` = the **3090** (wrong card). Native path needs
`CUDA_DEVICE_ORDER=PCI_BUS_ID`. → [[reference_irv_ml1_gpu_r14]]
## Open — A6000 window request, brokered not decided
## ✅ CLOSED — A6000 window WITHDRAWN 2026-09-01T18:09 (operator freed the VRAM instead)
The operator freed ComfyUI's VRAM directly, unblocking tts-dev without anyone stopping
anything. Request retracted with comfy-dev; nothing scheduled.
⚠ **It was a MODEL UNLOAD, not a stop.** Verified 2026-09-01T18:09:15-07:00: `comfyui` still
`Up 8 days (healthy)`, **same pid 4012952** (no restart), `/system_stats` HTTP 200, A6000
resident 18,500 MiB -> 612 MiB. No state lost; the only cost is a cold model reload off
`/storetank` on the next render. Told comfy-dev explicitly so a VRAM drop is not misread as a
container restart.
⚠ **THE 43.8 GB IS A SNAPSHOT, NOT A FLOOR — the coordination shrank, it did not vanish.**
ComfyUI is live and reloads its ~18.5 GB on the next render. Worst case during the bench:
A6000 free (idle) ~43.8 GB
ComfyUI reloads 18.5 GB
= actual floor ~25.3 GB vs FireRedAudio's ~26 GB <- BELOW requirement
So the ask changed from "stop ComfyUI for a window" to "**do not render during the bench**".
Flagged to both; **deliberately NOT volunteered on comfy-dev's behalf** — tts-dev must ask,
comfy-dev grants or declines. Open on their side, not infra-ops'.
## Original ask, for the record
tts-dev has operator approval for a two-engine TTS bench (FireRedAudio, then Breeze-TTS-2) and
needs `comfyui` stopped for the duration; FireRedAudio wants ~26 GB (21.2 GB backbone + 8.4 GB