vh
83e5e941d8
Two fixes from the failed first deploy on irv-ml1:
1. CPU/GPU variant. Kokoro's GPU image needs CUDA >= 12.9; irv-ml1's
driver 570.124.06 caps at 12.8 so the gpu variant fails with
"nvidia-container-cli: requirement error: unsatisfied condition:
cuda>=12.9". Make the variant a knob:
KOKORO_VARIANT=cpu (default — works anywhere)
KOKORO_VARIANT=gpu (after driver bump)
KOKORO_USE_GPU=false|true (matches the variant)
Kokoro is tiny (82M params) so CPU is workable: TTFA ~1s vs ~300ms
on GPU. Acceptable while the driver bump gets scheduled. compose.yaml
no longer hard-codes `runtime: nvidia` — relies on the daemon's
default-runtime + NVIDIA_VISIBLE_DEVICES gating, same as how the
wrapper's USE_GPU flag selects the inference path inside the
container. Toggling between variants is now a `.env` edit + restart.
2. Tighter pull-log filter. --quiet on `docker compose pull` only
suppresses the pull command's stdout; the docker daemon still
emits per-layer extraction events on stderr ("ffbfd7a09415
Extracting 64.06MB" repeated dozens of times per layer). Drop those
too via grep on the SHA-prefixed pattern. set -o pipefail keeps a
real pull failure visible.
For existing deployments: removing /opt/docker/compose/kokoro/.env
on the host and rerunning the playbook re-seeds with the new schema.
Kokoro
hexgrad/Kokoro-82M served via remsky/Kokoro-FastAPI.
Why this stack exists
Lowest-latency English TTS in the fleet by a wide margin — ~300 ms
time-to-first-audio on GPU, 35-100x realtime, ~1 GB VRAM at fp16.
Native streaming (Kokoro's KPipeline.__call__ is a per-phrase
generator) and the wrapper exposes it via OpenAI-compat stream=true
over chunked HTTP — drop-in for any OpenAI SDK client.
Complementary to the rest of the TTS slate:
| use case | |
|---|---|
| Kokoro | low-latency English, fixed voice library |
| Chatterbox Turbo | low-latency English w/ voice cloning + paralinguistic tags |
| IndexTTS-2 | English voice cloning + emotion vector / text control |
| Qwen3-TTS-1.7B-Base | high-quality English voice cloning |
| CosyVoice 3 | multilingual (Chinese-leaning) |
| VibeVoice 1.5B | long-form podcast / multi-speaker dialogue |
API
OpenAI-compat at http://10.100.79.3:8193:
# List built-in voices (~60 of them, named like af_bella, am_adam, jf_*, zf_*).
curl http://10.100.79.3:8193/v1/audio/voices
# Single-shot synthesis.
curl -fsS -X POST http://10.100.79.3:8193/v1/audio/speech \
-H 'Content-Type: application/json' \
-d '{"model":"kokoro","input":"Hello there.","voice":"af_bella","response_format":"wav"}' \
> out.wav
# Streaming — pipe straight into a player.
curl -fsS -X POST http://10.100.79.3:8193/v1/audio/speech \
-H 'Content-Type: application/json' \
-d '{"model":"kokoro","input":"long passage here…","voice":"af_bella","stream":true}' \
| mpv --no-cache -
# Voice mixing — sum voicepacks with weights.
curl -fsS -X POST http://10.100.79.3:8193/v1/audio/speech \
-H 'Content-Type: application/json' \
-d '{"model":"kokoro","input":"hello","voice":"af_bella(2)+af_heart(1)","response_format":"wav"}' \
> mix.wav
Web UI at /web (browse voices + try in-place); OpenAPI at /docs.
Supported response_format: mp3 | wav | opus | flac | pcm.
Voices
- Built-in: 60+ in 8 languages (en-US, en-GB, ja, zh, es, fr, hi, it).
Discoverable via
GET /v1/audio/voices— naming convention is<lang_code><gender_letter>_<name>(e.g.af_bella,am_adam,jf_alpha,zf_xiaobei). - Custom: drop
.ptvoicepacks into/worktank/kokoro/user_voices/on the host. Wrapper auto-discovers them on next request (no restart). Training Kokoro voices is non-trivial — consult the hexgrad community for how-to.
Deploy
scripts/elway irv-ml1 --playbook playbooks/deploy-kokoro.yaml
First deploy: ~6.5 GB image pull from GHCR (~2-5 min on a fast link). No model download on first run — Kokoro-82M weights are baked in. Subsequent starts: a few seconds.
License
Apache-2.0 for both the wrapper code (remsky/Kokoro-FastAPI) and the Kokoro-82M weights (hexgrad).