From d3faeb0314a878353b210279b95a3c73941a1f21 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 11 May 2026 14:55:15 -0700 Subject: [PATCH] catalog-contract: add response-decomposition fields (audio_field, timestamps_field, audio_format_field) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit asset_engine consumer needed to render kokoro-captioned, whose wire shape is a JSON envelope carrying base64-encoded audio plus a structured timestamps array. Modeling it as response.type=json would force either a per-service-id renderer (forbidden by brief §1.7) or extending the closed response-type vocabulary (forbidden by brief §2.2 without a coordinated bump). Resolution (per althing thread 01KRCF4W66X3): keep response.type closed at the existing six values and decompose at the response *field* level instead — the same flexibility seam already used by mime / mime_from_field / output_field. Adds three optional keys: - audio_field: JSON key holding base64-encoded audio bytes - audio_format_field: JSON key holding the decoded audio MIME - timestamps_field: JSON key holding a structured timestamps array (independent of type, declared by any service emitting time- aligned markers) Validators in CatalogResponse enforce sane combinations: - audio_field requires response.type=audio - audio_field forbids mime_from_field - audio_format_field requires audio_field This is additive and backward-compatible — no catalog_version bump, existing services parse unchanged. CATALOG-CONTRACT.md updated with the new rows in the response-field table and a versioning-policy row codifying that adding optional keys to response: doesn't bump. kokoro-captioned re-shaped to use the new schema: response: type: audio audio_field: audio audio_format_field: audio_format timestamps_field: timestamps And marked status: experimental until the asset_engine consumer's audio-with-timestamps renderer ships. JSON Schema regenerated to reflect the new Pydantic shape. Pydantic-model side of this change lives in the asset_engine repo at src/asset_engine/catalog.py — committed there separately. --- docs/asset-engine/CATALOG-CONTRACT.md | 12 +++++--- docs/asset-engine/services.schema.json | 37 +++++++++++++++++++++++++ docs/asset-engine/services.yaml | 38 +++++++++++++++++--------- 3 files changed, 70 insertions(+), 17 deletions(-) diff --git a/docs/asset-engine/CATALOG-CONTRACT.md b/docs/asset-engine/CATALOG-CONTRACT.md index feb0993..fd9f113 100644 --- a/docs/asset-engine/CATALOG-CONTRACT.md +++ b/docs/asset-engine/CATALOG-CONTRACT.md @@ -46,10 +46,13 @@ reproducibility_audit: [Audit] # one entry per service | `model.revision` | string \| null | no | SHA when known | | `model.image` | string | yes | container image ref this is hosted from | | `fields` | list[Field] | no | request parameters; empty for catalog-deferred | -| `response.type` | enum: audio, image, video, text, json, file | yes | response renderer hint | -| `response.mime` | string | no | static response MIME | -| `response.mime_from_field` | string | no | name of a field whose value determines the MIME | -| `response.output_field` | string | no | for JSON responses, the key holding the asset | +| `response.type` | enum: audio, image, video, text, json, file | yes | renderer dispatch (closed vocabulary) | +| `response.mime` | string | no | static response MIME (raw-bytes wire) | +| `response.mime_from_field` | string | no | name of a field whose value determines the MIME (raw-bytes wire) | +| `response.output_field` | string | no | for text/json responses, JSON key holding the asset | +| `response.audio_field` | string | no | for `type: audio` with JSON-envelope wire, JSON key holding base64-encoded audio bytes | +| `response.audio_format_field` | string | no | for `type: audio` with JSON-envelope wire, JSON key holding the decoded audio MIME (e.g. "audio/wav") | +| `response.timestamps_field` | string | no | JSON key holding a structured timestamps array — independent of type, declared by any service that emits time-aligned markers alongside its primary output | | `reproducibility.seedable` | bool | yes | does the endpoint accept a seed? | | `reproducibility.deterministic` | bool | yes | same params → same bytes? | | `reproducibility.notes` | string | no | gotchas | @@ -91,6 +94,7 @@ Same change-management as field types. |----------------------------------------------|-----------------------| | Add a new service | nothing | | Add a non-required field to an existing service | service `version:` | +| Add an optional key to the `response:` schema (e.g. audio_field, timestamps_field) | nothing — additive, backward-compatible | | Change a field's type, range, or default | service `version:` | | Remove a service | service `version:` (sentinel: removed=true), then drop in next catalog_version bump | | Add a new entry to the field-type vocabulary | `catalog_version:` | diff --git a/docs/asset-engine/services.schema.json b/docs/asset-engine/services.schema.json index a905774..87a28e9 100644 --- a/docs/asset-engine/services.schema.json +++ b/docs/asset-engine/services.schema.json @@ -332,6 +332,7 @@ }, "CatalogResponse": { "additionalProperties": false, + "description": "How to interpret the inference response.\n\nThe `type` is the *renderer dispatch* \u2014 what kind of asset the user\nultimately sees (audio player, text block, etc). The other fields\ndescribe how to *extract* that asset from the wire shape:\n\n - Raw-bytes wire (e.g. /v1/audio/speech returns raw audio):\n type: audio\n mime: audio/wav (or mime_from_field for dynamic)\n\n - JSON-envelope wire with base64-encoded asset (e.g. captioned\n speech returns {audio: , audio_format: ..., timestamps: ...}):\n type: audio\n audio_field: audio # JSON key with the base64 bytes\n audio_format_field: audio_format # JSON key with the MIME\n timestamps_field: timestamps # optional structured aside\n\n - JSON-envelope wire with text payload (e.g. ASR transcript):\n type: text\n output_field: text # JSON key with the text body\n timestamps_field: segments # optional, for ASR-with-timestamps\n\nRules:\n - When audio_field is set, the renderer expects a JSON envelope and\n will base64-decode that key. The wire MIME is application/json\n regardless of `mime`; `mime_from_field` is incompatible.\n - timestamps_field is independent of type \u2014 any service that\n emits structured timestamps alongside its primary output can\n declare it; the renderer extends accordingly.", "properties": { "type": { "enum": [ @@ -380,6 +381,42 @@ ], "default": null, "title": "Output Field" + }, + "audio_field": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Audio Field" + }, + "audio_format_field": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Audio Format Field" + }, + "timestamps_field": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Timestamps Field" } }, "required": [ diff --git a/docs/asset-engine/services.yaml b/docs/asset-engine/services.yaml index afa44db..06554a7 100644 --- a/docs/asset-engine/services.yaml +++ b/docs/asset-engine/services.yaml @@ -111,11 +111,12 @@ services: name: Kokoro Captioned Speech description: > Kokoro TTS with word-level timestamps returned alongside the audio. - For subtitle generation and video sync. Same model as `kokoro`; this - is a separate catalog entry because the response shape is structured - JSON (audio inline + timestamps), not raw audio bytes. + For subtitle generation and video sync. Same model as `kokoro`; + separate catalog entry because the wire shape is a JSON envelope + carrying base64-encoded audio plus a structured timestamps array. category: tts version: 1 + status: experimental host: irv-ml1 endpoint: http://10.100.79.3:8193/dev/captioned_speech method: POST @@ -153,32 +154,43 @@ services: label: Language code required: false response: - type: json - mime: application/json + # Stays in the closed type vocabulary: from a renderer-dispatch + # standpoint this IS audio. The audio_field/audio_format_field/ + # timestamps_field decomposition tells consumers how to extract + # those parts from the JSON envelope wire shape — added to the + # catalog schema in 2026-05 specifically to support response + # shapes like this one without extending the type vocab. + type: audio + audio_field: audio + audio_format_field: audio_format + timestamps_field: timestamps reproducibility: seedable: false deterministic: true notes: > - Same determinism story as kokoro proper. Response shape (verified - 2026-05-11 against live API): + Same determinism story as kokoro proper. Verified wire shape + (2026-05-11 against live API): { "audio": "", "audio_format": "audio/wav" (or matching response_format), "timestamps": [{"word": str, "start_time": float, "end_time": float}, ...] } - Consumer must base64-decode `audio` to play; `timestamps` drives - subtitle/karaoke UI. Audio_format string in the payload is - authoritative for the decoded bytes. + Consumer base64-decodes `audio` to play; `timestamps` drives + subtitle/karaoke UI. The response decomposition fields above + encode this so the renderer doesn't need per-service-id branches. estimated_latency: cold_start_s: 2 warm_per_unit: "~same as kokoro proper, plus minor overhead for timestamp emission" license: Apache-2.0 notes: | - `return_timestamps` and `stream` upstream params are deliberately + `return_timestamps` and `stream` upstream params deliberately omitted from the catalog: timestamps must be on for this endpoint to be meaningful, and streaming + JSON-with-base64 don't compose. - `download_format` / `return_download_link` skipped — same reasoning - as kokoro proper. + `download_format` / `return_download_link` skipped — same as kokoro + proper. + + status: experimental until the consumer's audio-with-timestamps + renderer ships. Once present, flip to status: ready. - id: chatterbox name: Chatterbox Turbo TTS