memory: snapshot — irv-ml1 GPU resident map; dots-tts at 2.4x its recorded VRAM
tts-dev asked for an A6000 window for an approved TTS bench and flagged a 3090 VRAM delta. Probed the box and mapped PID to container rather than taking the reported figures. - The 18.5 GB process they attributed to the 3090 is comfyui, on the A6000. And it is 18.5 GB rather than the ~11.8 GB they budgeted, so stopping it gives ~44.4 GB free, not the tight margin they expected. - Their "~4 GB unaccounted" on the 3090 is two things: parakeet is a third tenant the doc figure never counted, and dots-tts alone is holding 14,430 MiB against a burn-in figure of ~6 GB. The second is the larger finding and it is theirs to act on; handed over with a caching-allocator hypothesis and a one-restart discriminating test. - Restated the GPU ordering foot-gun: device_ids ["1"] is the A6000 in a container, but a bare native CUDA_VISIBLE_DEVICES=1 gets the 3090. Window not granted unilaterally — comfyui is comfy-dev's and they are mid-migration, so the request went to them directly and infra-ops relays. Ruled that the bench runs as a plain container under lkraven rather than under /opt/docker/compose/, which is for deployed stacks and would leave a canonical entry reporting as drift until deleted. Read-only probes; nothing on the box was changed. Memory-only; no version bump per the SemVer SKIP list.
This commit is contained in:
@@ -135,6 +135,15 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
|
||||
instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2
|
||||
without deciding that first. Exactly TWO 3c launches, only one died.
|
||||
→ `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
|
||||
- **⏸ A6000 WINDOW REQUEST — brokered with comfy-dev, awaiting their timing.** tts-dev needs
|
||||
`comfyui` stopped on irv-ml1's A6000 for an operator-approved TTS bench (FireRedAudio, then
|
||||
Breeze-TTS-2); ~26 GB needed against ~25.9 GB free = a coin flip, and stopping comfyui gives
|
||||
~44.4 GB. Their build is a day or two out. **Not granted unilaterally** — comfyui is
|
||||
comfy-dev's and they are mid-migration. Bench provisioning ruled: **plain container under
|
||||
`lkraven`, not `/opt/docker/compose/`**. ⚠ Also measured: **`dots-tts` is holding 14,430 MiB
|
||||
against a recorded ~6 GB** (2.4x — likely a PyTorch caching-allocator high-water mark after
|
||||
17 days up), and the 3090 is at 76% behind a warn-only watchdog.
|
||||
→ `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md`
|
||||
- **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a)
|
||||
`zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving.
|
||||
Branch (b) rejected on the trade (comfy-dev corrected the *reason*: only comfyui mounts
|
||||
|
||||
Reference in New Issue
Block a user