catalog-contract: add response-decomposition fields (audio_field, timestamps_field, audio_format_field)

asset_engine consumer needed to render kokoro-captioned, whose wire
shape is a JSON envelope carrying base64-encoded audio plus a
structured timestamps array. Modeling it as response.type=json
would force either a per-service-id renderer (forbidden by
brief §1.7) or extending the closed response-type vocabulary
(forbidden by brief §2.2 without a coordinated bump).

Resolution (per althing thread 01KRCF4W66X3): keep response.type
closed at the existing six values and decompose at the response
*field* level instead — the same flexibility seam already used by
mime / mime_from_field / output_field. Adds three optional keys:

  - audio_field: JSON key holding base64-encoded audio bytes
  - audio_format_field: JSON key holding the decoded audio MIME
  - timestamps_field: JSON key holding a structured timestamps array
    (independent of type, declared by any service emitting time-
    aligned markers)

Validators in CatalogResponse enforce sane combinations:
  - audio_field requires response.type=audio
  - audio_field forbids mime_from_field
  - audio_format_field requires audio_field

This is additive and backward-compatible — no catalog_version bump,
existing services parse unchanged. CATALOG-CONTRACT.md updated with
the new rows in the response-field table and a versioning-policy
row codifying that adding optional keys to response: doesn't bump.

kokoro-captioned re-shaped to use the new schema:
  response:
    type: audio
    audio_field: audio
    audio_format_field: audio_format
    timestamps_field: timestamps
And marked status: experimental until the asset_engine consumer's
audio-with-timestamps renderer ships.

JSON Schema regenerated to reflect the new Pydantic shape.

Pydantic-model side of this change lives in the asset_engine repo
at src/asset_engine/catalog.py — committed there separately.
This commit is contained in:
vh
2026-05-11 14:55:15 -07:00
parent 44c565ac77
commit d3faeb0314
3 changed files with 70 additions and 17 deletions
+25 -13
View File
@@ -111,11 +111,12 @@ services:
name: Kokoro Captioned Speech
description: >
Kokoro TTS with word-level timestamps returned alongside the audio.
For subtitle generation and video sync. Same model as `kokoro`; this
is a separate catalog entry because the response shape is structured
JSON (audio inline + timestamps), not raw audio bytes.
For subtitle generation and video sync. Same model as `kokoro`;
separate catalog entry because the wire shape is a JSON envelope
carrying base64-encoded audio plus a structured timestamps array.
category: tts
version: 1
status: experimental
host: irv-ml1
endpoint: http://10.100.79.3:8193/dev/captioned_speech
method: POST
@@ -153,32 +154,43 @@ services:
label: Language code
required: false
response:
type: json
mime: application/json
# Stays in the closed type vocabulary: from a renderer-dispatch
# standpoint this IS audio. The audio_field/audio_format_field/
# timestamps_field decomposition tells consumers how to extract
# those parts from the JSON envelope wire shape — added to the
# catalog schema in 2026-05 specifically to support response
# shapes like this one without extending the type vocab.
type: audio
audio_field: audio
audio_format_field: audio_format
timestamps_field: timestamps
reproducibility:
seedable: false
deterministic: true
notes: >
Same determinism story as kokoro proper. Response shape (verified
2026-05-11 against live API):
Same determinism story as kokoro proper. Verified wire shape
(2026-05-11 against live API):
{
"audio": "<base64-encoded bytes in response_format>",
"audio_format": "audio/wav" (or matching response_format),
"timestamps": [{"word": str, "start_time": float, "end_time": float}, ...]
}
Consumer must base64-decode `audio` to play; `timestamps` drives
subtitle/karaoke UI. Audio_format string in the payload is
authoritative for the decoded bytes.
Consumer base64-decodes `audio` to play; `timestamps` drives
subtitle/karaoke UI. The response decomposition fields above
encode this so the renderer doesn't need per-service-id branches.
estimated_latency:
cold_start_s: 2
warm_per_unit: "~same as kokoro proper, plus minor overhead for timestamp emission"
license: Apache-2.0
notes: |
`return_timestamps` and `stream` upstream params are deliberately
`return_timestamps` and `stream` upstream params deliberately
omitted from the catalog: timestamps must be on for this endpoint
to be meaningful, and streaming + JSON-with-base64 don't compose.
`download_format` / `return_download_link` skipped — same reasoning
as kokoro proper.
`download_format` / `return_download_link` skipped — same as kokoro
proper.
status: experimental until the consumer's audio-with-timestamps
renderer ships. Once present, flip to status: ready.
- id: chatterbox
name: Chatterbox Turbo TTS