memory: snapshot — irv-ml1 GPU resident map; dots-tts at 2.4x its recorded VRAM

tts-dev asked for an A6000 window for an approved TTS bench and flagged a
3090 VRAM delta. Probed the box and mapped PID to container rather than
taking the reported figures.

- The 18.5 GB process they attributed to the 3090 is comfyui, on the
  A6000. And it is 18.5 GB rather than the ~11.8 GB they budgeted, so
  stopping it gives ~44.4 GB free, not the tight margin they expected.
- Their "~4 GB unaccounted" on the 3090 is two things: parakeet is a
  third tenant the doc figure never counted, and dots-tts alone is
  holding 14,430 MiB against a burn-in figure of ~6 GB. The second is
  the larger finding and it is theirs to act on; handed over with a
  caching-allocator hypothesis and a one-restart discriminating test.
- Restated the GPU ordering foot-gun: device_ids ["1"] is the A6000 in a
  container, but a bare native CUDA_VISIBLE_DEVICES=1 gets the 3090.

Window not granted unilaterally — comfyui is comfy-dev's and they are
mid-migration, so the request went to them directly and infra-ops relays.
Ruled that the bench runs as a plain container under lkraven rather than
under /opt/docker/compose/, which is for deployed stacks and would leave
a canonical entry reporting as drift until deleted.

Read-only probes; nothing on the box was changed.

Memory-only; no version bump per the SemVer SKIP list.
This commit is contained in:
vh
2026-09-01 18:08:14 -07:00
parent 2cf73fd556
commit e91278799d
2 changed files with 79 additions and 0 deletions
+9
View File
@@ -135,6 +135,15 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2
without deciding that first. Exactly TWO 3c launches, only one died.
→ `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
- **⏸ A6000 WINDOW REQUEST — brokered with comfy-dev, awaiting their timing.** tts-dev needs
`comfyui` stopped on irv-ml1's A6000 for an operator-approved TTS bench (FireRedAudio, then
Breeze-TTS-2); ~26 GB needed against ~25.9 GB free = a coin flip, and stopping comfyui gives
~44.4 GB. Their build is a day or two out. **Not granted unilaterally** — comfyui is
comfy-dev's and they are mid-migration. Bench provisioning ruled: **plain container under
`lkraven`, not `/opt/docker/compose/`**. ⚠ Also measured: **`dots-tts` is holding 14,430 MiB
against a recorded ~6 GB** (2.4x — likely a PyTorch caching-allocator high-water mark after
17 days up), and the 3090 is at 76% behind a warn-only watchdog.
→ `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md`
- **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a)
`zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving.
Branch (b) rejected on the trade (comfy-dev corrected the *reason*: only comfyui mounts