# `[2026-09-01]` irv-ml1 GPU resident map — and dots-tts is holding 2.4x its recorded VRAM

Measured **2026-09-01T18:05:58-07:00**, PID -> container mapped via
`/proc/<pid>/cgroup` + `docker inspect .State.Pid`, not inferred.

    GPU 0  RTX 3090   18,727 / 24,576 MiB   (~5.8 GB free, 76% used)
      1852662   14,430 MiB   dots-tts          up 17d21h
      3739643    3,686 MiB   omnivoice-ref     up 19d02h
       116892      582 MiB   parakeet          up 49d01h

    GPU 1  RTX A6000  23,253 / 49,140 MiB   (~25.9 GB free)
      4012952   18,500 MiB   comfyui           up  8d09h
      3166988    4,722 MiB   studio-gate       up 19d09h

## ⚠ dots-tts is 14,430 MiB; the burn-in recorded ~6 GB — CAUSE IDENTIFIED BY tts-dev

2.4x the figure in the 2026-08-09→10 dots burn-in.

**Cause (tts-dev, 2026-09-01): their prompt-feature cache**, diagnosed and **capped at 32
entries on 2026-08-14 after two production incidents**. The cap is holding in the sense that
nothing has crashed, but 14.43 GB says the ceiling they chose still permits a long way of
growth. Theirs to own; no infra-ops action.

⚠ **infra-ops' hypothesis here was WRONG and is withdrawn** — I proposed a PyTorch
caching-allocator high-water mark from 17 days of uptime, with a restart as the discriminating
test. Plausible, and not what it is. tts-dev had an actual mechanism with a date, a cap and an
incident history attached. **A named mechanism beats a plausible story; do not re-propose the
allocator theory.** (The `expandable_segments` conflict with `optimize=True` on dots is still
true and still worth not tripping over, but it is not the explanation here.)

## The "~4 GB unaccounted" on the 3090 was two different things

tts-dev flagged 18.7 GB against a ~14.6 GB doc figure for "dots + omnivoice-ref". Decomposes:

- **`parakeet` is a third tenant that figure never counted** (582 MiB). Not creep — an
  uncounted tenant. The doc is wrong, not the card.
- **The rest is the dots-tts growth above**, which is the larger and more interesting half.

They also mis-located the 18.5 GB process: it is `comfyui` on the **A6000**, not the 3090.

## ⚠ GPU ordering, restated because it keeps costing time

nvidia-smi indexes **3090 = 0, A6000 = 1**. Native CUDA on this host defaults to fastest-first,
so **A6000 = cuda:0 natively**. `device_ids: ["1"]` in a container = A6000 (correct); a bare
native `CUDA_VISIBLE_DEVICES=1` = the **3090** (wrong card). Native path needs
`CUDA_DEVICE_ORDER=PCI_BUS_ID`. → [[reference_irv_ml1_gpu_r14]]

## ✅ CLOSED — A6000 window WITHDRAWN 2026-09-01T18:09 (operator freed the VRAM instead)

The operator freed ComfyUI's VRAM directly, unblocking tts-dev without anyone stopping
anything. Request retracted with comfy-dev; nothing scheduled.

⚠ **It was a MODEL UNLOAD, not a stop.** Verified 2026-09-01T18:09:15-07:00: `comfyui` still
`Up 8 days (healthy)`, **same pid 4012952** (no restart), `/system_stats` HTTP 200, A6000
resident 18,500 MiB -> 612 MiB. No state lost; the only cost is a cold model reload off
`/storetank` on the next render. Told comfy-dev explicitly so a VRAM drop is not misread as a
container restart.

⚠ **THE 43.8 GB IS A SNAPSHOT, NOT A FLOOR — the coordination shrank, it did not vanish.**
ComfyUI is live and reloads its ~18.5 GB on the next render. Worst case during the bench:

    A6000 free (idle)     ~43.8 GB
    ComfyUI reloads        18.5 GB
    = actual floor        ~25.3 GB    vs FireRedAudio's ~26 GB   <- BELOW requirement

So the ask changed from "stop ComfyUI for a window" to "**do not render during the bench**".
Flagged to both; **deliberately NOT volunteered on comfy-dev's behalf** — tts-dev must ask,
comfy-dev grants or declines. Open on their side, not infra-ops'.

## Original ask, for the record

tts-dev has operator approval for a two-engine TTS bench (FireRedAudio, then Breeze-TTS-2) and
needs `comfyui` stopped for the duration; FireRedAudio wants ~26 GB (21.2 GB backbone + 8.4 GB
decoder) against ~25.9 GB free, which is a coin flip. Stopping comfyui gives **~44.4 GB**.
Their build is a day or two out; nothing blocked today.

**Not granted unilaterally** — comfyui is comfy-dev's and they are mid-migration (~112 GB batch
onboarding, cutover window unset). Request put to comfy-dev directly; infra-ops relays rather
than sending tts-dev chasing. → [[feedback_credential_broker]]

**Ruling on provisioning (infra-ops call):** the bench runs as a **plain container under
`lkraven`**, NOT under `/opt/docker/compose/`. That tree is for deployed stacks — canonical
copy in the eshpfi repo, Homepage labels, restart policy, `sync-stacks.sh` drift coverage. A
bench that comes down when it ends is none of those, and provisioning it as a stack creates a
canonical entry that reports as drift until someone remembers to delete it.

⚠ **This may dissolve rather than need scheduling:** comfyui is leaving irv-ml1 for the R750xa,
after which the A6000 is permanently free. Told tts-dev explicitly NOT to plan around it — the
box has not arrived and the window is unset. → [[2026-09-01-ada-migration-branch-a]]

Threads: tts-dev `01M1FTB0TEP4T7P21B511F0CS3`.