vh
46a3c63706
Operator freed ComfyUI's VRAM directly, so tts-dev is unblocked and the window request is withdrawn with comfy-dev. - Verified it was a model unload, not a stop: comfyui still up 8 days, same pid, HTTP 200, 18,500 -> 612 MiB. Told comfy-dev explicitly so a VRAM drop is not misread as a restart of their service. - The resulting 43.8 GB free is a snapshot, not a floor. ComfyUI is live and reloads ~18.5 GB on the next render, which puts the real floor at ~25.3 GB against FireRedAudio's ~26 GB requirement. The coordination shrank from "stop ComfyUI" to "don't render during the bench" rather than disappearing. Flagged to both; not volunteered on comfy-dev's behalf. - Withdrew my caching-allocator hypothesis for the dots-tts VRAM. tts-dev identified it as their prompt-feature cache, capped at 32 entries on 2026-08-14 after two production incidents. A named mechanism with an incident history beats a plausible story, and the useful finding is that 14.43 GB sits inside a cap they deliberately chose. Read-only probes; nothing on the box was changed. Memory-only; no version bump per the SemVer SKIP list.
92 lines
5.0 KiB
Markdown
92 lines
5.0 KiB
Markdown
# `[2026-09-01]` irv-ml1 GPU resident map — and dots-tts is holding 2.4x its recorded VRAM
|
|
|
|
Measured **2026-09-01T18:05:58-07:00**, PID -> container mapped via
|
|
`/proc/<pid>/cgroup` + `docker inspect .State.Pid`, not inferred.
|
|
|
|
GPU 0 RTX 3090 18,727 / 24,576 MiB (~5.8 GB free, 76% used)
|
|
1852662 14,430 MiB dots-tts up 17d21h
|
|
3739643 3,686 MiB omnivoice-ref up 19d02h
|
|
116892 582 MiB parakeet up 49d01h
|
|
|
|
GPU 1 RTX A6000 23,253 / 49,140 MiB (~25.9 GB free)
|
|
4012952 18,500 MiB comfyui up 8d09h
|
|
3166988 4,722 MiB studio-gate up 19d09h
|
|
|
|
## ⚠ dots-tts is 14,430 MiB; the burn-in recorded ~6 GB — CAUSE IDENTIFIED BY tts-dev
|
|
|
|
2.4x the figure in the 2026-08-09→10 dots burn-in.
|
|
|
|
**Cause (tts-dev, 2026-09-01): their prompt-feature cache**, diagnosed and **capped at 32
|
|
entries on 2026-08-14 after two production incidents**. The cap is holding in the sense that
|
|
nothing has crashed, but 14.43 GB says the ceiling they chose still permits a long way of
|
|
growth. Theirs to own; no infra-ops action.
|
|
|
|
⚠ **infra-ops' hypothesis here was WRONG and is withdrawn** — I proposed a PyTorch
|
|
caching-allocator high-water mark from 17 days of uptime, with a restart as the discriminating
|
|
test. Plausible, and not what it is. tts-dev had an actual mechanism with a date, a cap and an
|
|
incident history attached. **A named mechanism beats a plausible story; do not re-propose the
|
|
allocator theory.** (The `expandable_segments` conflict with `optimize=True` on dots is still
|
|
true and still worth not tripping over, but it is not the explanation here.)
|
|
|
|
## The "~4 GB unaccounted" on the 3090 was two different things
|
|
|
|
tts-dev flagged 18.7 GB against a ~14.6 GB doc figure for "dots + omnivoice-ref". Decomposes:
|
|
|
|
- **`parakeet` is a third tenant that figure never counted** (582 MiB). Not creep — an
|
|
uncounted tenant. The doc is wrong, not the card.
|
|
- **The rest is the dots-tts growth above**, which is the larger and more interesting half.
|
|
|
|
They also mis-located the 18.5 GB process: it is `comfyui` on the **A6000**, not the 3090.
|
|
|
|
## ⚠ GPU ordering, restated because it keeps costing time
|
|
|
|
nvidia-smi indexes **3090 = 0, A6000 = 1**. Native CUDA on this host defaults to fastest-first,
|
|
so **A6000 = cuda:0 natively**. `device_ids: ["1"]` in a container = A6000 (correct); a bare
|
|
native `CUDA_VISIBLE_DEVICES=1` = the **3090** (wrong card). Native path needs
|
|
`CUDA_DEVICE_ORDER=PCI_BUS_ID`. → [[reference_irv_ml1_gpu_r14]]
|
|
|
|
## ✅ CLOSED — A6000 window WITHDRAWN 2026-09-01T18:09 (operator freed the VRAM instead)
|
|
|
|
The operator freed ComfyUI's VRAM directly, unblocking tts-dev without anyone stopping
|
|
anything. Request retracted with comfy-dev; nothing scheduled.
|
|
|
|
⚠ **It was a MODEL UNLOAD, not a stop.** Verified 2026-09-01T18:09:15-07:00: `comfyui` still
|
|
`Up 8 days (healthy)`, **same pid 4012952** (no restart), `/system_stats` HTTP 200, A6000
|
|
resident 18,500 MiB -> 612 MiB. No state lost; the only cost is a cold model reload off
|
|
`/storetank` on the next render. Told comfy-dev explicitly so a VRAM drop is not misread as a
|
|
container restart.
|
|
|
|
⚠ **THE 43.8 GB IS A SNAPSHOT, NOT A FLOOR — the coordination shrank, it did not vanish.**
|
|
ComfyUI is live and reloads its ~18.5 GB on the next render. Worst case during the bench:
|
|
|
|
A6000 free (idle) ~43.8 GB
|
|
ComfyUI reloads 18.5 GB
|
|
= actual floor ~25.3 GB vs FireRedAudio's ~26 GB <- BELOW requirement
|
|
|
|
So the ask changed from "stop ComfyUI for a window" to "**do not render during the bench**".
|
|
Flagged to both; **deliberately NOT volunteered on comfy-dev's behalf** — tts-dev must ask,
|
|
comfy-dev grants or declines. Open on their side, not infra-ops'.
|
|
|
|
## Original ask, for the record
|
|
|
|
tts-dev has operator approval for a two-engine TTS bench (FireRedAudio, then Breeze-TTS-2) and
|
|
needs `comfyui` stopped for the duration; FireRedAudio wants ~26 GB (21.2 GB backbone + 8.4 GB
|
|
decoder) against ~25.9 GB free, which is a coin flip. Stopping comfyui gives **~44.4 GB**.
|
|
Their build is a day or two out; nothing blocked today.
|
|
|
|
**Not granted unilaterally** — comfyui is comfy-dev's and they are mid-migration (~112 GB batch
|
|
onboarding, cutover window unset). Request put to comfy-dev directly; infra-ops relays rather
|
|
than sending tts-dev chasing. → [[feedback_credential_broker]]
|
|
|
|
**Ruling on provisioning (infra-ops call):** the bench runs as a **plain container under
|
|
`lkraven`**, NOT under `/opt/docker/compose/`. That tree is for deployed stacks — canonical
|
|
copy in the eshpfi repo, Homepage labels, restart policy, `sync-stacks.sh` drift coverage. A
|
|
bench that comes down when it ends is none of those, and provisioning it as a stack creates a
|
|
canonical entry that reports as drift until someone remembers to delete it.
|
|
|
|
⚠ **This may dissolve rather than need scheduling:** comfyui is leaving irv-ml1 for the R750xa,
|
|
after which the A6000 is permanently free. Told tts-dev explicitly NOT to plan around it — the
|
|
box has not arrived and the window is unset. → [[2026-09-01-ada-migration-branch-a]]
|
|
|
|
Threads: tts-dev `01M1FTB0TEP4T7P21B511F0CS3`.
|