STATUS: irv-ml1 recovered, three new GPU stacks deployed, llama-swap pin cleanup, backups verified

Big update for 2026-04-27. Sections added:

* Marked the "🟥 Blocked — irv-ml1 stalled" header as RECOVERED with
  resolution notes (driver 595.58.03 / CUDA 13.2 IS working, both GPUs
  detected; the original "stall" must have been a one-shot
  post-install hiccup that resolved on a later boot).

* New "Session milestones — 2026-04-27" section covering:
  - irv-ml1 unstall + 5 pre-existing GPU stacks restored
  - Kokoro GPU variant deployed (irv-ml1:8193) with the .env.example
    default flipped to gpu now that the driver works
  - VibeVoice 1.5B deployed (irv-ml1:8194) after fixing two bugs:
    full 40-char SHA required by buildx + verify regex didn't match
    the OpenAI list-format response shape
  - Chatterbox Turbo deployed (irv-ml1:8196) after fixing three:
    upstream moved Dockerfile path (docker/Dockerfile.gpu →
    Dockerfile.cu128 at root), pinned to current SHA instead of `main`,
    /health doesn't exist (switched all probes to /api/model-info
    which is the wrapper's own ready-after-loaded signal)
  - llama-swap qwen3.6 ttl removal across non-pinned variants;
    qwen3.6-35-a3b unpinned (was OOM'ing other loads via the pinned
    group's persistent: true flag); granite-4-small added to the
    pinned group to stop it swapping with qwen3.6-27b
  - Backup verification: all three layers green (per-host restic,
    PBS-ANA, PBS-NH3 mirror — 2026-04-27 snapshots everywhere). Noted
    that backrest's empty dashboard is expected (no plans configured;
    the actual orchestration is the per-host resticprofile timers).

STATUS.md

@@ -87,13 +87,21 @@ significant work lands — don't let it drift quietly.
 
 ## Open issues
 
-### 🟥 Blocked — irv-ml1 stalled after CUDA driver upgrade (2026-04-25)
+### ~~🟥 Blocked — irv-ml1 stalled after CUDA driver upgrade (2026-04-25)~~ — RECOVERED 2026-04-27
 
-Driver bump on irv-ml1 (570.124.06 → 595.58.03 via NVIDIA's CUDA APT
+**Resolution**: the host eventually came back on its own — driver
-repo) was attempted to unblock Kokoro's GPU image (which requires
+595.58.03 (CUDA 13.2) IS working, both GPUs detected (RTX 3090 + RTX
-CUDA ≥ 12.9; old driver capped at 12.8). The host did not come up
+A6000), kernel modules loaded cleanly. The original "stall" symptoms
-cleanly after the post-install reboot. **Stop touching irv-ml1 until
+must have been a one-shot post-install issue that resolved on a
-this is resolved.**
+later boot. All 5 pre-existing GPU stacks (comfyui, cosyvoice,
+qwen3-tts, index-tts, parakeet) restarted clean and report healthy.
+Homepage's irv-ml1-docker provider re-enabled via `configs/homepage/
+docker.yaml`. Remaining post-recovery work was the deferred deploys
+(Kokoro GPU, VibeVoice, Chatterbox) — see "Session milestones —
+2026-04-27" below.
+
+The original stall context is preserved below for the next time we
+do a CUDA driver bump on a bare-metal host:
 
 **What was attempted**: `scripts/upgrade-irv-ml1-cuda.sh phase1`
 on 2026-04-25. Phase 1 is supposed to: snapshot current nvidia
@@ -493,6 +501,92 @@ Side fix: moved `stacks/llama-swap/config.yaml` →
 the server layout (`stacks/<stack>/conf/<file>` →
 `/opt/docker/conf/<stack>/<file>`).
 
+## Session milestones — 2026-04-27 (irv-ml1 recovery + new GPU deploys)
+
+### irv-ml1 fully recovered
+- Host alive, driver 595.58.03 / CUDA 13.2, both GPUs detected,
+  kernel modules loaded.
+- 5 pre-existing GPU stacks (comfyui, cosyvoice, qwen3-tts,
+  index-tts, parakeet) restarted clean and report healthy.
+- Homepage `irv-ml1-docker` provider re-enabled in
+  `configs/homepage/docker.yaml`, pushed via the rsync workflow.
+
+### Three new GPU stacks deployed on irv-ml1
+
+- **Kokoro GPU variant** — first try succeeded once the `.env.example`
+  default was flipped from `cpu` → `gpu` (matching the now-working
+  driver). `KOKORO_VARIANT=gpu` / `KOKORO_USE_GPU=true` /
+  `KOKORO_GPU_DEVICES=0` (pinned to RTX 3090; ~1 GB VRAM, doesn't
+  need the A6000). Serving on irv-ml1:8193.
+
+- **VibeVoice 1.5B** — needed two iterations:
+  - `VIBEVOICE_SHA=7614c469a145` (12-char short) made docker buildx's
+    git source resolver error with "repository does not contain ref
+    7614c469a145" — same commit IS still HEAD of main, but buildx
+    requires the FULL 40-char SHA even when the short is unambiguous.
+    Now `7614c469a145dafc30b1fd5f6a6046ea831dbd6a`.
+  - The verify step greped `/v1/audio/voices` for `voices`/`voice`/
+    `alloy`/`Carter` literals, but VibeVoice's actual response shape
+    is OpenAI list-format `{"object":"list","data":[]}`. Switched to
+    JSON-parse + assert `object="list"` and `data` is a list. Robust
+    against an empty voices set on a fresh install.
+  - Serving on irv-ml1:8194.
+
+- **Chatterbox Turbo** (devnen wrapper) — needed three iterations:
+  - Old playbook used `dockerfile: docker/Dockerfile.gpu`; devnen
+    restructured the repo to put Dockerfiles at root, renamed by CUDA
+    version (`Dockerfile.cu128`, `.cpu`, `.rocm`). Switched to
+    `Dockerfile.cu128` (GPU build for CUDA 12.8 toolkit; works with
+    irv-ml1's 595.58.03 driver via backward-compat).
+  - Pinned `CHATTERBOX_SHA` from `main` to the current SHA
+    `68213e75c248f4f4bbd185f7c20ec02722fdc79b` so future upstream
+    churn doesn't break us without warning.
+  - Wait + verify steps probed `/health`, but devnen's wrapper
+    doesn't expose that route (404). Switched to `/api/model-info`
+    which returns `{"loaded":true,...}` only after the model finishes
+    loading — doubles as liveness + readiness. Compose healthcheck
+    + verify both updated. Verify of `/v1/audio/voices` also fixed
+    to parse the wrapper's actual response shape
+    `{"status":"ok","voices":[...]}` (note: NOT the same OpenAI list
+    shape vibevoice uses).
+  - Serving on irv-ml1:8196.
+
+### llama-swap pin/unload cleanup
+
+- Removed `ttl: 600` from `qwen3.6-35-a3b-abliterated`,
+  `qwen3.6-35-a3b-heretic`, `qwen3.6-27b` so they stay loaded across
+  idle (still evictable when another non-coexistent model is
+  requested).
+- `qwen3.6-35-a3b` removed from the `pinned` group — its
+  `persistent: true` flag was preventing eviction under memory
+  pressure, OOM-ing other model loads. Now sits with `ttl: 0`
+  outside any group: never idle-unloads but is evictable.
+- `granite-4-small` added to the `pinned` group alongside
+  `qwen3.5-9b` so it stops swapping with `qwen3.6-27b` when both are
+  used concurrently. ~5-6 GB at Q4_K_M + 120K KV ≈ comfortable inside
+  the existing pin budget.
+
+### Backup verification post-recovery
+
+All three layers green as of 2026-04-27:
+- **Per-host restic** (`resticprofile-backup@profile-default.timer`):
+  all 6 hosts (ana-docker, esh-docker-vm, nh3-docker, esh-vm-db,
+  ana-ml2, irv-ml1) ran successfully at 01:00 today. The migration
+  to docker-ce on three of these hosts didn't disrupt the timers.
+  irv-ml1's timer fired right after recovery.
+- **PBS-ANA**: every namespace (`ana-pve`, `esh-pve`, `nh3-pve`,
+  `sfsrv-pve`) has fresh 2026-04-27 snapshots for every VM/CT.
+- **PBS-NH3 (DR mirror)**: synced — 2026-04-27 backups present in
+  /mnt/pbs-datastore/ns/. NH3 NAS at 27TB / 42TB used (64%, healthy
+  headroom).
+
+Note: backrest's UI shows `recentBackups: {}` for all repos because
+backrest in this setup is a viewer with **zero plans configured** —
+the actual orchestration is the per-host resticprofile timers.
+Backrest's only configured operations are `_unassociated_` (manual
+one-off pushes from the UI) and `_system_` housekeeping. Don't be
+alarmed by the empty dashboard — check the per-host timers instead.
+
 ## Session milestones — 2026-04-24 (the "tooling day" + housekeeping pm)
 
 Morning / early afternoon — the original tooling day: