- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`, 0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` = `/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch 2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention), re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at `/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):** comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) → rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1), lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved.