memory: snapshot — A6000 window closed; my dots-tts hypothesis was wrong
Operator freed ComfyUI's VRAM directly, so tts-dev is unblocked and the window request is withdrawn with comfy-dev. - Verified it was a model unload, not a stop: comfyui still up 8 days, same pid, HTTP 200, 18,500 -> 612 MiB. Told comfy-dev explicitly so a VRAM drop is not misread as a restart of their service. - The resulting 43.8 GB free is a snapshot, not a floor. ComfyUI is live and reloads ~18.5 GB on the next render, which puts the real floor at ~25.3 GB against FireRedAudio's ~26 GB requirement. The coordination shrank from "stop ComfyUI" to "don't render during the bench" rather than disappearing. Flagged to both; not volunteered on comfy-dev's behalf. - Withdrew my caching-allocator hypothesis for the dots-tts VRAM. tts-dev identified it as their prompt-feature cache, capped at 32 entries on 2026-08-14 after two production incidents. A named mechanism with an incident history beats a plausible story, and the useful finding is that 14.43 GB sits inside a cap they deliberately chose. Read-only probes; nothing on the box was changed. Memory-only; no version bump per the SemVer SKIP list.
This commit is contained in:
@@ -135,15 +135,15 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
|
||||
instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2
|
||||
without deciding that first. Exactly TWO 3c launches, only one died.
|
||||
→ `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
|
||||
- **⏸ A6000 WINDOW REQUEST — brokered with comfy-dev, awaiting their timing.** tts-dev needs
|
||||
`comfyui` stopped on irv-ml1's A6000 for an operator-approved TTS bench (FireRedAudio, then
|
||||
Breeze-TTS-2); ~26 GB needed against ~25.9 GB free = a coin flip, and stopping comfyui gives
|
||||
~44.4 GB. Their build is a day or two out. **Not granted unilaterally** — comfyui is
|
||||
comfy-dev's and they are mid-migration. Bench provisioning ruled: **plain container under
|
||||
`lkraven`, not `/opt/docker/compose/`**. ⚠ Also measured: **`dots-tts` is holding 14,430 MiB
|
||||
against a recorded ~6 GB** (2.4x — likely a PyTorch caching-allocator high-water mark after
|
||||
17 days up), and the 3090 is at 76% behind a warn-only watchdog.
|
||||
→ `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md`
|
||||
- **✅ A6000 window CLOSED 2026-09-01 — operator freed ComfyUI's VRAM instead of anyone
|
||||
stopping it.** ⚠ It was a **model unload, not a stop**: `comfyui` still up 8 days, same pid,
|
||||
HTTP 200, 18,500 -> 612 MiB. ⚠ **The resulting ~43.8 GB free is a snapshot, not a floor** —
|
||||
ComfyUI reloads ~18.5 GB on the next render, putting the real floor at ~25.3 GB against
|
||||
FireRedAudio's ~26 GB. Coordination shrank to "don't render during the bench"; tts-dev's ask
|
||||
to make, comfy-dev's to grant. Also measured: **`dots-tts` holds 14,430 MiB vs a ~6 GB
|
||||
baseline** — tts-dev's **prompt-feature cache**, capped at 32 entries 2026-08-14 after two
|
||||
incidents (infra-ops' caching-allocator hypothesis was wrong, withdrawn). 3090 at 76% behind
|
||||
a warn-only watchdog. → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md`
|
||||
- **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a)
|
||||
`zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving.
|
||||
Branch (b) rejected on the trade (comfy-dev corrected the *reason*: only comfyui mounts
|
||||
|
||||
Reference in New Issue
Block a user