stacks/kokoro: cpu/gpu variant toggle + tighter pull-log filter
Two fixes from the failed first deploy on irv-ml1:
1. CPU/GPU variant. Kokoro's GPU image needs CUDA >= 12.9; irv-ml1's
driver 570.124.06 caps at 12.8 so the gpu variant fails with
"nvidia-container-cli: requirement error: unsatisfied condition:
cuda>=12.9". Make the variant a knob:
KOKORO_VARIANT=cpu (default — works anywhere)
KOKORO_VARIANT=gpu (after driver bump)
KOKORO_USE_GPU=false|true (matches the variant)
Kokoro is tiny (82M params) so CPU is workable: TTFA ~1s vs ~300ms
on GPU. Acceptable while the driver bump gets scheduled. compose.yaml
no longer hard-codes `runtime: nvidia` — relies on the daemon's
default-runtime + NVIDIA_VISIBLE_DEVICES gating, same as how the
wrapper's USE_GPU flag selects the inference path inside the
container. Toggling between variants is now a `.env` edit + restart.
2. Tighter pull-log filter. --quiet on `docker compose pull` only
suppresses the pull command's stdout; the docker daemon still
emits per-layer extraction events on stderr ("ffbfd7a09415
Extracting 64.06MB" repeated dozens of times per layer). Drop those
too via grep on the SHA-prefixed pattern. set -o pipefail keeps a
real pull failure visible.
For existing deployments: removing /opt/docker/compose/kokoro/.env
on the host and rerunning the playbook re-seeds with the new schema.
This commit is contained in:
@@ -58,9 +58,15 @@ steps:
|
||||
# ── pull + bring up ─────────────────────────────────────────────────
|
||||
|
||||
- name: "docker compose pull (first run: ~6.5 GB from GHCR)"
|
||||
# --quiet drops the per-layer progress redraws that flood the log
|
||||
# during a 6.5 GB pull. Final "X Pulled" line still prints.
|
||||
shell: cd {{ compose_dir }} && docker compose pull --quiet
|
||||
# --quiet suppresses pull-command progress, but the docker daemon
|
||||
# still emits its own per-layer extraction progress events
|
||||
# ("ffbfd7a09415 Extracting 64.06MB" etc) on stderr — these are
|
||||
# what flooded the log on the previous run. Drop them with grep.
|
||||
# set -o pipefail so a real pull failure isn't swallowed.
|
||||
shell: |
|
||||
set -o pipefail
|
||||
cd {{ compose_dir }} && docker compose pull --quiet 2>&1 \
|
||||
| grep -vE '^\s*[0-9a-f]{8,16}\s+(Extracting|Pull complete|Pulling fs layer|Verifying Checksum|Download complete|Waiting|Already exists|Downloading|Pulling from)'
|
||||
|
||||
- name: docker compose up -d
|
||||
shell: cd {{ compose_dir }} && docker compose up -d
|
||||
|
||||
Reference in New Issue
Block a user