From bd749ae7475462bfd1fc5c0199e7468334edc5f5 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 27 Apr 2026 18:24:20 -0700 Subject: [PATCH] STATUS: irv-ml1 recovered, three new GPU stacks deployed, llama-swap pin cleanup, backups verified MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Big update for 2026-04-27. Sections added: * Marked the "🟥 Blocked — irv-ml1 stalled" header as RECOVERED with resolution notes (driver 595.58.03 / CUDA 13.2 IS working, both GPUs detected; the original "stall" must have been a one-shot post-install hiccup that resolved on a later boot). * New "Session milestones — 2026-04-27" section covering: - irv-ml1 unstall + 5 pre-existing GPU stacks restored - Kokoro GPU variant deployed (irv-ml1:8193) with the .env.example default flipped to gpu now that the driver works - VibeVoice 1.5B deployed (irv-ml1:8194) after fixing two bugs: full 40-char SHA required by buildx + verify regex didn't match the OpenAI list-format response shape - Chatterbox Turbo deployed (irv-ml1:8196) after fixing three: upstream moved Dockerfile path (docker/Dockerfile.gpu → Dockerfile.cu128 at root), pinned to current SHA instead of `main`, /health doesn't exist (switched all probes to /api/model-info which is the wrapper's own ready-after-loaded signal) - llama-swap qwen3.6 ttl removal across non-pinned variants; qwen3.6-35-a3b unpinned (was OOM'ing other loads via the pinned group's persistent: true flag); granite-4-small added to the pinned group to stop it swapping with qwen3.6-27b - Backup verification: all three layers green (per-host restic, PBS-ANA, PBS-NH3 mirror — 2026-04-27 snapshots everywhere). Noted that backrest's empty dashboard is expected (no plans configured; the actual orchestration is the per-host resticprofile timers). --- STATUS.md | 106 ++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 100 insertions(+), 6 deletions(-) diff --git a/STATUS.md b/STATUS.md index 3bc6b28..a3b993e 100644 --- a/STATUS.md +++ b/STATUS.md @@ -87,13 +87,21 @@ significant work lands — don't let it drift quietly. ## Open issues -### 🟥 Blocked — irv-ml1 stalled after CUDA driver upgrade (2026-04-25) +### ~~🟥 Blocked — irv-ml1 stalled after CUDA driver upgrade (2026-04-25)~~ — RECOVERED 2026-04-27 -Driver bump on irv-ml1 (570.124.06 → 595.58.03 via NVIDIA's CUDA APT -repo) was attempted to unblock Kokoro's GPU image (which requires -CUDA ≥ 12.9; old driver capped at 12.8). The host did not come up -cleanly after the post-install reboot. **Stop touching irv-ml1 until -this is resolved.** +**Resolution**: the host eventually came back on its own — driver +595.58.03 (CUDA 13.2) IS working, both GPUs detected (RTX 3090 + RTX +A6000), kernel modules loaded cleanly. The original "stall" symptoms +must have been a one-shot post-install issue that resolved on a +later boot. All 5 pre-existing GPU stacks (comfyui, cosyvoice, +qwen3-tts, index-tts, parakeet) restarted clean and report healthy. +Homepage's irv-ml1-docker provider re-enabled via `configs/homepage/ +docker.yaml`. Remaining post-recovery work was the deferred deploys +(Kokoro GPU, VibeVoice, Chatterbox) — see "Session milestones — +2026-04-27" below. + +The original stall context is preserved below for the next time we +do a CUDA driver bump on a bare-metal host: **What was attempted**: `scripts/upgrade-irv-ml1-cuda.sh phase1` on 2026-04-25. Phase 1 is supposed to: snapshot current nvidia @@ -493,6 +501,92 @@ Side fix: moved `stacks/llama-swap/config.yaml` → the server layout (`stacks//conf/` → `/opt/docker/conf//`). +## Session milestones — 2026-04-27 (irv-ml1 recovery + new GPU deploys) + +### irv-ml1 fully recovered +- Host alive, driver 595.58.03 / CUDA 13.2, both GPUs detected, + kernel modules loaded. +- 5 pre-existing GPU stacks (comfyui, cosyvoice, qwen3-tts, + index-tts, parakeet) restarted clean and report healthy. +- Homepage `irv-ml1-docker` provider re-enabled in + `configs/homepage/docker.yaml`, pushed via the rsync workflow. + +### Three new GPU stacks deployed on irv-ml1 + +- **Kokoro GPU variant** — first try succeeded once the `.env.example` + default was flipped from `cpu` → `gpu` (matching the now-working + driver). `KOKORO_VARIANT=gpu` / `KOKORO_USE_GPU=true` / + `KOKORO_GPU_DEVICES=0` (pinned to RTX 3090; ~1 GB VRAM, doesn't + need the A6000). Serving on irv-ml1:8193. + +- **VibeVoice 1.5B** — needed two iterations: + - `VIBEVOICE_SHA=7614c469a145` (12-char short) made docker buildx's + git source resolver error with "repository does not contain ref + 7614c469a145" — same commit IS still HEAD of main, but buildx + requires the FULL 40-char SHA even when the short is unambiguous. + Now `7614c469a145dafc30b1fd5f6a6046ea831dbd6a`. + - The verify step greped `/v1/audio/voices` for `voices`/`voice`/ + `alloy`/`Carter` literals, but VibeVoice's actual response shape + is OpenAI list-format `{"object":"list","data":[]}`. Switched to + JSON-parse + assert `object="list"` and `data` is a list. Robust + against an empty voices set on a fresh install. + - Serving on irv-ml1:8194. + +- **Chatterbox Turbo** (devnen wrapper) — needed three iterations: + - Old playbook used `dockerfile: docker/Dockerfile.gpu`; devnen + restructured the repo to put Dockerfiles at root, renamed by CUDA + version (`Dockerfile.cu128`, `.cpu`, `.rocm`). Switched to + `Dockerfile.cu128` (GPU build for CUDA 12.8 toolkit; works with + irv-ml1's 595.58.03 driver via backward-compat). + - Pinned `CHATTERBOX_SHA` from `main` to the current SHA + `68213e75c248f4f4bbd185f7c20ec02722fdc79b` so future upstream + churn doesn't break us without warning. + - Wait + verify steps probed `/health`, but devnen's wrapper + doesn't expose that route (404). Switched to `/api/model-info` + which returns `{"loaded":true,...}` only after the model finishes + loading — doubles as liveness + readiness. Compose healthcheck + + verify both updated. Verify of `/v1/audio/voices` also fixed + to parse the wrapper's actual response shape + `{"status":"ok","voices":[...]}` (note: NOT the same OpenAI list + shape vibevoice uses). + - Serving on irv-ml1:8196. + +### llama-swap pin/unload cleanup + +- Removed `ttl: 600` from `qwen3.6-35-a3b-abliterated`, + `qwen3.6-35-a3b-heretic`, `qwen3.6-27b` so they stay loaded across + idle (still evictable when another non-coexistent model is + requested). +- `qwen3.6-35-a3b` removed from the `pinned` group — its + `persistent: true` flag was preventing eviction under memory + pressure, OOM-ing other model loads. Now sits with `ttl: 0` + outside any group: never idle-unloads but is evictable. +- `granite-4-small` added to the `pinned` group alongside + `qwen3.5-9b` so it stops swapping with `qwen3.6-27b` when both are + used concurrently. ~5-6 GB at Q4_K_M + 120K KV ≈ comfortable inside + the existing pin budget. + +### Backup verification post-recovery + +All three layers green as of 2026-04-27: +- **Per-host restic** (`resticprofile-backup@profile-default.timer`): + all 6 hosts (ana-docker, esh-docker-vm, nh3-docker, esh-vm-db, + ana-ml2, irv-ml1) ran successfully at 01:00 today. The migration + to docker-ce on three of these hosts didn't disrupt the timers. + irv-ml1's timer fired right after recovery. +- **PBS-ANA**: every namespace (`ana-pve`, `esh-pve`, `nh3-pve`, + `sfsrv-pve`) has fresh 2026-04-27 snapshots for every VM/CT. +- **PBS-NH3 (DR mirror)**: synced — 2026-04-27 backups present in + /mnt/pbs-datastore/ns/. NH3 NAS at 27TB / 42TB used (64%, healthy + headroom). + +Note: backrest's UI shows `recentBackups: {}` for all repos because +backrest in this setup is a viewer with **zero plans configured** — +the actual orchestration is the per-host resticprofile timers. +Backrest's only configured operations are `_unassociated_` (manual +one-off pushes from the UI) and `_system_` housekeeping. Don't be +alarmed by the empty dashboard — check the per-host timers instead. + ## Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm) Morning / early afternoon — the original tooling day: