memory: snapshot — A6000 window closed; my dots-tts hypothesis was wrong

Operator freed ComfyUI's VRAM directly, so tts-dev is unblocked and the
window request is withdrawn with comfy-dev.

- Verified it was a model unload, not a stop: comfyui still up 8 days,
  same pid, HTTP 200, 18,500 -> 612 MiB. Told comfy-dev explicitly so a
  VRAM drop is not misread as a restart of their service.
- The resulting 43.8 GB free is a snapshot, not a floor. ComfyUI is live
  and reloads ~18.5 GB on the next render, which puts the real floor at
  ~25.3 GB against FireRedAudio's ~26 GB requirement. The coordination
  shrank from "stop ComfyUI" to "don't render during the bench" rather
  than disappearing. Flagged to both; not volunteered on comfy-dev's
  behalf.
- Withdrew my caching-allocator hypothesis for the dots-tts VRAM. tts-dev
  identified it as their prompt-feature cache, capped at 32 entries on
  2026-08-14 after two production incidents. A named mechanism with an
  incident history beats a plausible story, and the useful finding is
  that 14.43 GB sits inside a cap they deliberately chose.

Read-only probes; nothing on the box was changed.

Memory-only; no version bump per the SemVer SKIP list.
This commit is contained in:
vh
2026-09-01 18:10:34 -07:00
parent e91278799d
commit 46a3c63706
2 changed files with 44 additions and 23 deletions
+9 -9
View File
@@ -135,15 +135,15 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2
without deciding that first. Exactly TWO 3c launches, only one died.
→ `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
- **⏸ A6000 WINDOW REQUEST — brokered with comfy-dev, awaiting their timing.** tts-dev needs
`comfyui` stopped on irv-ml1's A6000 for an operator-approved TTS bench (FireRedAudio, then
Breeze-TTS-2); ~26 GB needed against ~25.9 GB free = a coin flip, and stopping comfyui gives
~44.4 GB. Their build is a day or two out. **Not granted unilaterally** — comfyui is
comfy-dev's and they are mid-migration. Bench provisioning ruled: **plain container under
`lkraven`, not `/opt/docker/compose/`**. ⚠ Also measured: **`dots-tts` is holding 14,430 MiB
against a recorded ~6 GB** (2.4x — likely a PyTorch caching-allocator high-water mark after
17 days up), and the 3090 is at 76% behind a warn-only watchdog.
→ `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md`
- **✅ A6000 window CLOSED 2026-09-01 — operator freed ComfyUI's VRAM instead of anyone
stopping it.** ⚠ It was a **model unload, not a stop**: `comfyui` still up 8 days, same pid,
HTTP 200, 18,500 -> 612 MiB. ⚠ **The resulting ~43.8 GB free is a snapshot, not a floor** —
ComfyUI reloads ~18.5 GB on the next render, putting the real floor at ~25.3 GB against
FireRedAudio's ~26 GB. Coordination shrank to "don't render during the bench"; tts-dev's ask
to make, comfy-dev's to grant. Also measured: **`dots-tts` holds 14,430 MiB vs a ~6 GB
baseline** — tts-dev's **prompt-feature cache**, capped at 32 entries 2026-08-14 after two
incidents (infra-ops' caching-allocator hypothesis was wrong, withdrawn). 3090 at 76% behind
a warn-only watchdog. → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md`
- **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a)
`zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving.
Branch (b) rejected on the trade (comfy-dev corrected the *reason*: only comfyui mounts