diff --git a/STATUS.md b/STATUS.md index a203d6a..43e95ab 100644 --- a/STATUS.md +++ b/STATUS.md @@ -87,6 +87,78 @@ significant work lands — don't let it drift quietly. ## Open issues +### 🟥 Blocked — irv-ml1 stalled after CUDA driver upgrade (2026-04-25) + +Driver bump on irv-ml1 (570.124.06 → 595.58.03 via NVIDIA's CUDA APT +repo) was attempted to unblock Kokoro's GPU image (which requires +CUDA ≥ 12.9; old driver capped at 12.8). The host did not come up +cleanly after the post-install reboot. **Stop touching irv-ml1 until +this is resolved.** + +**What was attempted**: `scripts/upgrade-irv-ml1-cuda.sh phase1` +on 2026-04-25. Phase 1 is supposed to: snapshot current nvidia +packages → stop GPU containers → `apt install cuda-drivers` → +DKMS rebuild → reboot. Whether all those steps actually completed +before the unclean boot is unknown — the snapshot file at +`/tmp/nvidia-pre-upgrade.txt` on irv-ml1 (if it survived) will say. + +**Recovery options** (in increasing invasiveness): + +1. **Console diagnose first.** SSH may or may not work depending on + the failure mode. Check via console (irv-ml1 is bare-metal, has + physical access via Lenovo IPMI). Look at `journalctl -b -1` from + the previous boot to see what failed: DKMS build failure, kernel + module load error, nvidia-container-toolkit init regression, etc. + +2. **Rollback the driver** if the host is reachable but GPU is broken: + ``` + scripts/upgrade-irv-ml1-cuda.sh rollback + ``` + This reinstalls the snapshot from `/tmp/nvidia-pre-upgrade.txt` + with `--allow-downgrades` and reboots. Restores 570.124.06. + +3. **Manual driver pin** if the script can't recover: + ``` + ssh irv-ml1 sudo apt install --allow-downgrades \ + nvidia-driver=570.124.06-1 nvidia-driver-cuda=570.124.06-1 \ + nvidia-driver-libs=570.124.06-1 nvidia-kernel-open-dkms=570.124.06-1 + ssh irv-ml1 sudo apt-mark hold nvidia-driver nvidia-driver-cuda \ + nvidia-driver-libs nvidia-kernel-open-dkms + ssh irv-ml1 sudo systemctl reboot + ``` + +4. **Try a different driver version** if 595 is genuinely incompatible + with kernel 6.1.0-37: 580 series may build cleanly. Check via + `apt-cache madison cuda-drivers` after recovery. + +**Once irv-ml1 is healthy again**, deferred work to revisit: + +- **Kokoro GPU variant**: deployed CPU works as a fallback. Switch + `.env` to `KOKORO_VARIANT=gpu` / `KOKORO_USE_GPU=true` / + `KOKORO_GPU_DEVICES=0` and redeploy via + `scripts/elway irv-ml1 --playbook playbooks/deploy-kokoro.yaml`. +- **VibeVoice 1.5B deploy** (long-form / multi-speaker dialogue): + `scripts/elway irv-ml1 --playbook playbooks/deploy-vibevoice.yaml`. + Stack files at `stacks/vibevoice/`. ~12 min build + ~7 GB model. +- **Chatterbox Turbo deploy** (low-latency English voice cloning): + `scripts/elway irv-ml1 --playbook playbooks/deploy-chatterbox.yaml`. + Stack files at `stacks/chatterbox/`. ~10 min build + ~6 GB model. +- **Verify the existing 5 GPU stacks still work** post-upgrade: + comfyui, cosyvoice, qwen3-tts, index-tts, parakeet. They were all + Up + healthy before the driver attempt. Smoke-test each. +- **Re-evaluate Kokoro variant default** after recovery. If a 595+ + series driver doesn't end up working, the `.env.example` default + of `KOKORO_VARIANT=cpu` is correct as-is. If a working GPU driver + lands, flip the default to `gpu` for future deploys. + +**Don't run on irv-ml1 until this is closed**: +- Any of the three deploy playbooks above +- Any host-state-mutating playbook against irv-ml1 +- The driver upgrade script itself unless intentionally retrying + +Once recovered, update this section with what fixed it and remove the +🟥 marker. + ### 🟥 Quick wins (do next) 1. ~~**Fix Backrest's `esh-docker-vm` URI**~~ — done.