vh
46a3c63706
Operator freed ComfyUI's VRAM directly, so tts-dev is unblocked and the window request is withdrawn with comfy-dev. - Verified it was a model unload, not a stop: comfyui still up 8 days, same pid, HTTP 200, 18,500 -> 612 MiB. Told comfy-dev explicitly so a VRAM drop is not misread as a restart of their service. - The resulting 43.8 GB free is a snapshot, not a floor. ComfyUI is live and reloads ~18.5 GB on the next render, which puts the real floor at ~25.3 GB against FireRedAudio's ~26 GB requirement. The coordination shrank from "stop ComfyUI" to "don't render during the bench" rather than disappearing. Flagged to both; not volunteered on comfy-dev's behalf. - Withdrew my caching-allocator hypothesis for the dots-tts VRAM. tts-dev identified it as their prompt-feature cache, capped at 32 entries on 2026-08-14 after two production incidents. A named mechanism with an incident history beats a plausible story, and the useful finding is that 14.43 GB sits inside a cap they deliberately chose. Read-only probes; nothing on the box was changed. Memory-only; no version bump per the SemVer SKIP list.
5.0 KiB
[2026-09-01] irv-ml1 GPU resident map — and dots-tts is holding 2.4x its recorded VRAM
Measured 2026-09-01T18:05:58-07:00, PID -> container mapped via
/proc/<pid>/cgroup + docker inspect .State.Pid, not inferred.
GPU 0 RTX 3090 18,727 / 24,576 MiB (~5.8 GB free, 76% used)
1852662 14,430 MiB dots-tts up 17d21h
3739643 3,686 MiB omnivoice-ref up 19d02h
116892 582 MiB parakeet up 49d01h
GPU 1 RTX A6000 23,253 / 49,140 MiB (~25.9 GB free)
4012952 18,500 MiB comfyui up 8d09h
3166988 4,722 MiB studio-gate up 19d09h
⚠ dots-tts is 14,430 MiB; the burn-in recorded ~6 GB — CAUSE IDENTIFIED BY tts-dev
2.4x the figure in the 2026-08-09→10 dots burn-in.
Cause (tts-dev, 2026-09-01): their prompt-feature cache, diagnosed and capped at 32 entries on 2026-08-14 after two production incidents. The cap is holding in the sense that nothing has crashed, but 14.43 GB says the ceiling they chose still permits a long way of growth. Theirs to own; no infra-ops action.
⚠ infra-ops' hypothesis here was WRONG and is withdrawn — I proposed a PyTorch
caching-allocator high-water mark from 17 days of uptime, with a restart as the discriminating
test. Plausible, and not what it is. tts-dev had an actual mechanism with a date, a cap and an
incident history attached. A named mechanism beats a plausible story; do not re-propose the
allocator theory. (The expandable_segments conflict with optimize=True on dots is still
true and still worth not tripping over, but it is not the explanation here.)
The "~4 GB unaccounted" on the 3090 was two different things
tts-dev flagged 18.7 GB against a ~14.6 GB doc figure for "dots + omnivoice-ref". Decomposes:
parakeetis a third tenant that figure never counted (582 MiB). Not creep — an uncounted tenant. The doc is wrong, not the card.- The rest is the dots-tts growth above, which is the larger and more interesting half.
They also mis-located the 18.5 GB process: it is comfyui on the A6000, not the 3090.
⚠ GPU ordering, restated because it keeps costing time
nvidia-smi indexes 3090 = 0, A6000 = 1. Native CUDA on this host defaults to fastest-first,
so A6000 = cuda:0 natively. device_ids: ["1"] in a container = A6000 (correct); a bare
native CUDA_VISIBLE_DEVICES=1 = the 3090 (wrong card). Native path needs
CUDA_DEVICE_ORDER=PCI_BUS_ID. → reference_irv_ml1_gpu_r14
✅ CLOSED — A6000 window WITHDRAWN 2026-09-01T18:09 (operator freed the VRAM instead)
The operator freed ComfyUI's VRAM directly, unblocking tts-dev without anyone stopping anything. Request retracted with comfy-dev; nothing scheduled.
⚠ It was a MODEL UNLOAD, not a stop. Verified 2026-09-01T18:09:15-07:00: comfyui still
Up 8 days (healthy), same pid 4012952 (no restart), /system_stats HTTP 200, A6000
resident 18,500 MiB -> 612 MiB. No state lost; the only cost is a cold model reload off
/storetank on the next render. Told comfy-dev explicitly so a VRAM drop is not misread as a
container restart.
⚠ THE 43.8 GB IS A SNAPSHOT, NOT A FLOOR — the coordination shrank, it did not vanish. ComfyUI is live and reloads its ~18.5 GB on the next render. Worst case during the bench:
A6000 free (idle) ~43.8 GB
ComfyUI reloads 18.5 GB
= actual floor ~25.3 GB vs FireRedAudio's ~26 GB <- BELOW requirement
So the ask changed from "stop ComfyUI for a window" to "do not render during the bench". Flagged to both; deliberately NOT volunteered on comfy-dev's behalf — tts-dev must ask, comfy-dev grants or declines. Open on their side, not infra-ops'.
Original ask, for the record
tts-dev has operator approval for a two-engine TTS bench (FireRedAudio, then Breeze-TTS-2) and
needs comfyui stopped for the duration; FireRedAudio wants ~26 GB (21.2 GB backbone + 8.4 GB
decoder) against ~25.9 GB free, which is a coin flip. Stopping comfyui gives ~44.4 GB.
Their build is a day or two out; nothing blocked today.
Not granted unilaterally — comfyui is comfy-dev's and they are mid-migration (~112 GB batch onboarding, cutover window unset). Request put to comfy-dev directly; infra-ops relays rather than sending tts-dev chasing. → feedback_credential_broker
Ruling on provisioning (infra-ops call): the bench runs as a plain container under
lkraven, NOT under /opt/docker/compose/. That tree is for deployed stacks — canonical
copy in the eshpfi repo, Homepage labels, restart policy, sync-stacks.sh drift coverage. A
bench that comes down when it ends is none of those, and provisioning it as a stack creates a
canonical entry that reports as drift until someone remembers to delete it.
⚠ This may dissolve rather than need scheduling: comfyui is leaving irv-ml1 for the R750xa, after which the A6000 is permanently free. Told tts-dev explicitly NOT to plan around it — the box has not arrived and the window is unset. → 2026-09-01-ada-migration-branch-a
Threads: tts-dev 01M1FTB0TEP4T7P21B511F0CS3.