stacks/kokoro: cpu/gpu variant toggle + tighter pull-log filter

Two fixes from the failed first deploy on irv-ml1:

1. CPU/GPU variant. Kokoro's GPU image needs CUDA >= 12.9; irv-ml1's
   driver 570.124.06 caps at 12.8 so the gpu variant fails with
   "nvidia-container-cli: requirement error: unsatisfied condition:
   cuda>=12.9". Make the variant a knob:

     KOKORO_VARIANT=cpu         (default — works anywhere)
     KOKORO_VARIANT=gpu         (after driver bump)
     KOKORO_USE_GPU=false|true  (matches the variant)

   Kokoro is tiny (82M params) so CPU is workable: TTFA ~1s vs ~300ms
   on GPU. Acceptable while the driver bump gets scheduled. compose.yaml
   no longer hard-codes `runtime: nvidia` — relies on the daemon's
   default-runtime + NVIDIA_VISIBLE_DEVICES gating, same as how the
   wrapper's USE_GPU flag selects the inference path inside the
   container. Toggling between variants is now a `.env` edit + restart.

2. Tighter pull-log filter. --quiet on `docker compose pull` only
   suppresses the pull command's stdout; the docker daemon still
   emits per-layer extraction events on stderr ("ffbfd7a09415
   Extracting 64.06MB" repeated dozens of times per layer). Drop those
   too via grep on the SHA-prefixed pattern. set -o pipefail keeps a
   real pull failure visible.

For existing deployments: removing /opt/docker/compose/kokoro/.env
on the host and rerunning the playbook re-seeds with the new schema.
This commit is contained in:
vh
2026-04-25 16:31:01 -07:00
parent ca16db73e0
commit 83e5e941d8
3 changed files with 36 additions and 11 deletions
+9 -3
View File
@@ -58,9 +58,15 @@ steps:
# ── pull + bring up ─────────────────────────────────────────────────
- name: "docker compose pull (first run: ~6.5 GB from GHCR)"
# --quiet drops the per-layer progress redraws that flood the log
# during a 6.5 GB pull. Final "X Pulled" line still prints.
shell: cd {{ compose_dir }} && docker compose pull --quiet
# --quiet suppresses pull-command progress, but the docker daemon
# still emits its own per-layer extraction progress events
# ("ffbfd7a09415 Extracting 64.06MB" etc) on stderr — these are
# what flooded the log on the previous run. Drop them with grep.
# set -o pipefail so a real pull failure isn't swallowed.
shell: |
set -o pipefail
cd {{ compose_dir }} && docker compose pull --quiet 2>&1 \
| grep -vE '^\s*[0-9a-f]{8,16}\s+(Extracting|Pull complete|Pulling fs layer|Verifying Checksum|Download complete|Waiting|Already exists|Downloading|Pulling from)'
- name: docker compose up -d
shell: cd {{ compose_dir }} && docker compose up -d