STATUS: 🟥 block — irv-ml1 stalled after CUDA driver upgrade attempt

Driver bump 570.124.06 → 595.58.03 attempted via
scripts/upgrade-irv-ml1-cuda.sh on 2026-04-25; host did not come up
cleanly after the post-install reboot.

Captures, in one place, everything currently in-flight against irv-ml1
so it can be revisited as a unit once the machine is healthy:

  * Recovery options ordered by invasiveness (console diagnose →
    rollback script → manual driver pin → try a different driver
    series).
  * Three deferred TTS deploys (Kokoro GPU flip, VibeVoice 1.5B,
    Chatterbox Turbo) with the exact playbook commands to re-run.
  * Smoke-test reminder for the 5 existing GPU stacks (comfyui,
    cosyvoice, qwen3-tts, index-tts, parakeet) which were healthy
    before the upgrade but unverified after.
  * Explicit 'don't touch irv-ml1' guard rail until resolved.

Sits in 🟥 Blocked — surfaces above the existing 'Quick wins'
section so future sessions see it first.

STATUS.md

@@ -87,6 +87,78 @@ significant work lands — don't let it drift quietly.
 
 ## Open issues
 
+### 🟥 Blocked — irv-ml1 stalled after CUDA driver upgrade (2026-04-25)
+
+Driver bump on irv-ml1 (570.124.06 → 595.58.03 via NVIDIA's CUDA APT
+repo) was attempted to unblock Kokoro's GPU image (which requires
+CUDA ≥ 12.9; old driver capped at 12.8). The host did not come up
+cleanly after the post-install reboot. **Stop touching irv-ml1 until
+this is resolved.**
+
+**What was attempted**: `scripts/upgrade-irv-ml1-cuda.sh phase1`
+on 2026-04-25. Phase 1 is supposed to: snapshot current nvidia
+packages → stop GPU containers → `apt install cuda-drivers` →
+DKMS rebuild → reboot. Whether all those steps actually completed
+before the unclean boot is unknown — the snapshot file at
+`/tmp/nvidia-pre-upgrade.txt` on irv-ml1 (if it survived) will say.
+
+**Recovery options** (in increasing invasiveness):
+
+1. **Console diagnose first.** SSH may or may not work depending on
+   the failure mode. Check via console (irv-ml1 is bare-metal, has
+   physical access via Lenovo IPMI). Look at `journalctl -b -1` from
+   the previous boot to see what failed: DKMS build failure, kernel
+   module load error, nvidia-container-toolkit init regression, etc.
+
+2. **Rollback the driver** if the host is reachable but GPU is broken:
+   ```
+   scripts/upgrade-irv-ml1-cuda.sh rollback
+   ```
+   This reinstalls the snapshot from `/tmp/nvidia-pre-upgrade.txt`
+   with `--allow-downgrades` and reboots. Restores 570.124.06.
+
+3. **Manual driver pin** if the script can't recover:
+   ```
+   ssh irv-ml1 sudo apt install --allow-downgrades \
+     nvidia-driver=570.124.06-1 nvidia-driver-cuda=570.124.06-1 \
+     nvidia-driver-libs=570.124.06-1 nvidia-kernel-open-dkms=570.124.06-1
+   ssh irv-ml1 sudo apt-mark hold nvidia-driver nvidia-driver-cuda \
+     nvidia-driver-libs nvidia-kernel-open-dkms
+   ssh irv-ml1 sudo systemctl reboot
+   ```
+
+4. **Try a different driver version** if 595 is genuinely incompatible
+   with kernel 6.1.0-37: 580 series may build cleanly. Check via
+   `apt-cache madison cuda-drivers` after recovery.
+
+**Once irv-ml1 is healthy again**, deferred work to revisit:
+
+- **Kokoro GPU variant**: deployed CPU works as a fallback. Switch
+  `.env` to `KOKORO_VARIANT=gpu` / `KOKORO_USE_GPU=true` /
+  `KOKORO_GPU_DEVICES=0` and redeploy via
+  `scripts/elway irv-ml1 --playbook playbooks/deploy-kokoro.yaml`.
+- **VibeVoice 1.5B deploy** (long-form / multi-speaker dialogue):
+  `scripts/elway irv-ml1 --playbook playbooks/deploy-vibevoice.yaml`.
+  Stack files at `stacks/vibevoice/`. ~12 min build + ~7 GB model.
+- **Chatterbox Turbo deploy** (low-latency English voice cloning):
+  `scripts/elway irv-ml1 --playbook playbooks/deploy-chatterbox.yaml`.
+  Stack files at `stacks/chatterbox/`. ~10 min build + ~6 GB model.
+- **Verify the existing 5 GPU stacks still work** post-upgrade:
+  comfyui, cosyvoice, qwen3-tts, index-tts, parakeet. They were all
+  Up + healthy before the driver attempt. Smoke-test each.
+- **Re-evaluate Kokoro variant default** after recovery. If a 595+
+  series driver doesn't end up working, the `.env.example` default
+  of `KOKORO_VARIANT=cpu` is correct as-is. If a working GPU driver
+  lands, flip the default to `gpu` for future deploys.
+
+**Don't run on irv-ml1 until this is closed**:
+- Any of the three deploy playbooks above
+- Any host-state-mutating playbook against irv-ml1
+- The driver upgrade script itself unless intentionally retrying
+
+Once recovered, update this section with what fixed it and remove the
+🟥 marker.
+
 ### 🟥 Quick wins (do next)
 
 1. ~~**Fix Backrest's `esh-docker-vm` URI**~~ — done.