From 44c565ac77a64866a93b50f03e3ec33d60972aeb Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 11 May 2026 14:38:05 -0700 Subject: [PATCH] docs/asset-engine: kokoro v2 + new kokoro-captioned entry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per a request from the asset_engine consumer (althing thread 01KRCF4W66X3N24B01FF2Y7V3D), and verified against the live kokoro OpenAPI + exercised endpoints: * kokoro: version 1 → 2; adds three fields surfaced by the upstream schema but not previously declared: - speed (slider 0.25–4.0, default 1.0) - volume_multiplier (slider 0.5–2.0, default 1.0; UI-bounded since upstream is unbounded — noted in description) - lang_code (text, optional override of the voice-name-derived language hint) * kokoro-captioned: new service entry wrapping /dev/captioned_speech. Same model + image as kokoro proper but separate catalog entry because the response shape is structured JSON (audio inline as base64 + word-level timestamps), not raw audio bytes. Verified shape captured in reproducibility.notes so future consumers don't have to re-discover it. response.type = json (consumer renders custom: player + subtitle overlay). * reproducibility_audit: row added for kokoro-captioned. Deferred (separate from this commit): - kokoro-blend-voice. /v1/audio/voices/combine returns 403 on the default config (allow_local_voice_saving=False); even with the flag flipped it writes to a temp dir, not /worktank/kokoro/user_voices. The persistent blend mechanism in this fleet is playbooks/blend-kokoro-voice.yaml. Ad-hoc blending already works through /v1/audio/speech via the inline syntax voice="a(w)+b(w)"; consumer can surface that as a UI affordance without any catalog change. catalog_version stays at 1 (no field-type vocabulary changes). JSON Schema regeneration produced byte-identical output. --- docs/asset-engine/services.yaml | 111 +++++++++++++++++++++++++++++++- 1 file changed, 110 insertions(+), 1 deletion(-) diff --git a/docs/asset-engine/services.yaml b/docs/asset-engine/services.yaml index ae4c2ef..afa44db 100644 --- a/docs/asset-engine/services.yaml +++ b/docs/asset-engine/services.yaml @@ -26,7 +26,7 @@ services: 60+ built-in voices in 8 languages. Voice mixing via inline weights. OpenAI-compatible /v1/audio/speech. category: tts - version: 1 + version: 2 host: irv-ml1 endpoint: http://10.100.79.3:8193/v1/audio/speech method: POST @@ -54,6 +54,35 @@ services: type: select options: [wav, mp3, opus, flac, pcm] default: wav + - name: speed + type: slider + label: Speed + required: false + min: 0.25 + max: 4.0 + step: 0.05 + default: 1.0 + description: Playback speed multiplier. 1.0 is natural. + - name: volume_multiplier + type: slider + label: Volume + required: false + min: 0.5 + max: 2.0 + step: 0.05 + default: 1.0 + description: > + Output gain. Useful for level-matching against music beds. + Upstream is unbounded; UI bounds (0.5–2.0) are a sensible default — + override the slider client-side if you need extremes. + - name: lang_code + type: text + label: Language code + required: false + description: > + Optional language hint (e.g. "a" for American English). When unset, + kokoro infers from the first letter of the voice name. Set when text + contains code-switching kokoro guesses wrong on. - name: stream type: bool default: false @@ -75,6 +104,81 @@ services: notes: | Voice mixing: voice="name1(w1)+name2(w2)+..." normalizes weights. Custom voices persist at /worktank/kokoro/user_voices (bind-mounted). + v2 (2026-05-11): added speed, volume_multiplier, lang_code per + kokoro's OpenAPI surface. + + - id: kokoro-captioned + name: Kokoro Captioned Speech + description: > + Kokoro TTS with word-level timestamps returned alongside the audio. + For subtitle generation and video sync. Same model as `kokoro`; this + is a separate catalog entry because the response shape is structured + JSON (audio inline + timestamps), not raw audio bytes. + category: tts + version: 1 + host: irv-ml1 + endpoint: http://10.100.79.3:8193/dev/captioned_speech + method: POST + content_type: application/json + model: + id: hexgrad/Kokoro-82M + revision: null + image: ghcr.io/remsky/kokoro-fastapi-gpu:v0.2.4-master + fields: + - name: input + type: textarea + label: Text + required: true + max_length: 5000 + - name: voice + type: select + label: Voice + source_url: http://10.100.79.3:8193/v1/audio/voices + source_jsonpath: $.voices[*] + default: af_bella + - name: response_format + type: select + options: [wav, mp3, opus, flac, pcm] + default: wav + - name: speed + type: slider + label: Speed + required: false + min: 0.25 + max: 4.0 + step: 0.05 + default: 1.0 + - name: lang_code + type: text + label: Language code + required: false + response: + type: json + mime: application/json + reproducibility: + seedable: false + deterministic: true + notes: > + Same determinism story as kokoro proper. Response shape (verified + 2026-05-11 against live API): + { + "audio": "", + "audio_format": "audio/wav" (or matching response_format), + "timestamps": [{"word": str, "start_time": float, "end_time": float}, ...] + } + Consumer must base64-decode `audio` to play; `timestamps` drives + subtitle/karaoke UI. Audio_format string in the payload is + authoritative for the decoded bytes. + estimated_latency: + cold_start_s: 2 + warm_per_unit: "~same as kokoro proper, plus minor overhead for timestamp emission" + license: Apache-2.0 + notes: | + `return_timestamps` and `stream` upstream params are deliberately + omitted from the catalog: timestamps must be on for this endpoint + to be meaningful, and streaming + JSON-with-base64 don't compose. + `download_format` / `return_download_link` skipped — same reasoning + as kokoro proper. - id: chatterbox name: Chatterbox Turbo TTS @@ -820,6 +924,11 @@ reproducibility_audit: model_deterministic: true image_tag_mutable: true notes: ":v0.2.4-master is mutable; pin to digest for v3." + - service: kokoro-captioned + seedable: false + model_deterministic: true + image_tag_mutable: true + notes: "Same image as kokoro proper; same mutability story. Response carries timestamps." - service: chatterbox seedable: false model_deterministic: true