esh-pfi-infrastructure/docs/asset-engine/CATALOG-CONTRACT.md

# services.yaml — contract for downstream consumers

`services.yaml` in this directory is the **canonical, versioned**
description of the inference services this fleet exposes. It is a
first-class artifact: external tools (the asset-engine UI is the
first; CLIs, monitoring dashboards, and other internal services may
follow) consume it programmatically, and changes to its shape or
contents have downstream effect.

This document is the consumer contract.

## Where the data lives

- **Canonical source**: `docs/asset-engine/services.yaml` in the
  `eshpfi-management` repo (this file's directory).
- **Schema**: `docs/asset-engine/services.schema.json` (JSON Schema
  draft 2020-12). Generated from the Pydantic model in
  `~/development/asset_engine/src/asset_engine/catalog.py` —
  regenerate via `cd ~/development/asset_engine && uv run scripts/dump_schema.py`
  whenever the model changes.
- **Change history**: git log on this directory.

## Data model — high level

```yaml
catalog_version: 1                 # bump on SCHEMA changes
services: [Service]                # the entries; closed list
reproducibility_audit: [Audit]     # one entry per service
```

### Service entry

| field             | type                                      | required | meaning |
|-------------------|-------------------------------------------|----------|---------|
| `id`              | string                                    | yes      | stable URL-safe identifier; never reused |
| `name`            | string                                    | yes      | display name |
| `description`     | string                                    | no       | one-paragraph summary |
| `category`        | enum: tts, asr, sfx, music, image         | yes      | groups in pickers |
| `version`         | int                                       | yes      | per-service schema version; bump on field changes |
| `status`          | enum: ready, catalog-deferred, experimental, down | no | default: ready |

**Status semantics:**

| value | meaning | picker behavior |
|---|---|---|
| `ready` | Deployed, exercised, current default | clickable, primary surface |
| `experimental` | Deployed, may break under unusual params or while a custom renderer is being built (e.g. kokoro-captioned pre-renderer) | clickable, surfaced with a "experimental" tag |
| `catalog-deferred` | By-design out of scope this catalog version (e.g. ComfyUI requires per-asset workflow templates the catalog can't express today) | greyed, non-clickable, "deferred" tooltip |
| `down` | *Temporarily* unreachable on the host fleet — deployment gap, scheduled maintenance, container reaped, etc. Distinct from `catalog-deferred` because the intent is to flip back to `ready` once redeployed; the catalog row still describes a real intended service. | greyed, non-clickable, "service down" tooltip |
| `host`            | string (host alias)                       | yes      | which fleet host serves this |
| `endpoint`        | URL                                       | yes      | absolute URL of the inference endpoint |
| `method`          | string                                    | yes      | HTTP method (POST common) |
| `content_type`    | MIME type                                 | yes      | request body type |
| `model.id`        | string                                    | yes      | upstream model id (HF or other) |
| `model.revision`  | string \| null                            | no       | SHA when known |
| `model.image`     | string                                    | yes      | container image ref this is hosted from |
| `fields`          | list[Field]                               | no       | request parameters; empty for catalog-deferred |
| `section_groups`  | list[SectionGroup]                        | no       | named groups for progressive disclosure; consumers render fields under tabs/accordions per group. SectionGroup = `{id, label, hint?}`. Field.section refs must resolve here. |
| `response.type`   | enum: audio, image, video, text, json, file | yes    | renderer dispatch (closed vocabulary) |
| `response.mime`   | string                                    | no       | static response MIME (raw-bytes wire) |
| `response.mime_from_field` | string                          | no       | name of a field whose value determines the MIME (raw-bytes wire) |
| `response.output_field`    | string                          | no       | for text/json responses, JSON key holding the asset |
| `response.audio_field`     | string                          | no       | for `type: audio` with JSON-envelope wire, JSON key holding base64-encoded audio bytes |
| `response.audio_format_field` | string                       | no       | for `type: audio` with JSON-envelope wire, JSON key holding the decoded audio MIME (e.g. "audio/wav") |
| `response.timestamps_field` | string                         | no       | JSON key holding a structured timestamps array — independent of type, declared by any service that emits time-aligned markers alongside its primary output |
| `reproducibility.seedable`    | bool                          | yes      | does the endpoint accept a seed? |
| `reproducibility.deterministic` | bool                        | yes      | same params → same bytes? |
| `reproducibility.notes`       | string                        | no       | gotchas |
| `estimated_latency.cold_start_s` | int \| float               | no       | informational only |
| `estimated_latency.warm_per_unit` | string                    | no       | informational only |
| `license`         | string                                    | yes      | upstream license |
| `license_warning` | string                                    | no       | non-empty → consumer MUST surface a warning before use |
| `notes`           | string                                    | no       | freeform |

### Field entry

The field-type vocabulary is **closed**. Adding a new type requires:
(a) updating the Pydantic model in `asset_engine/catalog.py`,
(b) bumping `catalog_version`, (c) coordinating with all consumers.

| `type`     | meaning                                              |
|------------|------------------------------------------------------|
| `text`     | single-line string                                   |
| `textarea` | multi-line string                                    |
| `number`   | int or float, no bounded range                       |
| `slider`   | bounded numeric — requires `min`, `max`, optional `step` |
| `select`   | enum — `options:` list OR remote `source_url:` + `source_jsonpath:` |
| `bool`     | true/false                                           |
| `file`     | file upload — requires `accepted_types:` list of MIME |
| `json`     | freeform JSON value (arrays of floats, ref objects, etc) |

Optional/required: a field is required UNLESS it has either
`required: false` or `optional: true`. (Both ways are accepted; new
entries should prefer `required: false`.)

### Response renderer vocabulary

Closed set: `audio`, `image`, `video`, `text`, `json`, `file`.
Same change-management as field types.

## Versioning policy

| Change                                       | Bump                  |
|----------------------------------------------|-----------------------|
| Add a new service                            | nothing               |
| Add a non-required field to an existing service | service `version:` |
| Add an optional key to the `response:` schema (e.g. audio_field, timestamps_field) | nothing — additive, backward-compatible |
| Add an optional key to a Field (e.g. section) or to Service (e.g. section_groups) | nothing — additive, backward-compatible |
| Extend an existing enum (StatusT, FieldTypeT, ResponseTypeT, CategoryT) with a non-conflicting new value, with the consumer updated in the same coordinated change | nothing — older consumers don't encounter the new value; updated consumers parse it correctly |
| Change a field's type, range, or default     | service `version:`    |
| Remove a service                             | service `version:` (sentinel: removed=true), then drop in next catalog_version bump |
| Add a new entry to the field-type vocabulary | `catalog_version:`    |
| Add a new entry to the response-type vocabulary | `catalog_version:` |
| Drop a field-type or response-type           | `catalog_version:` (BREAKING) |
| Pure prose edit (description, notes)         | nothing               |

Consumers should pin their tested `catalog_version`. When this repo
bumps it, downstreams should re-test before consuming the new shape.

## Sync workflow for downstream consumers

The recommended pattern is **vendor + drift-check**, not
fetch-at-runtime. Reasons: deployments stay reproducible; airgapped
or network-flaky deploys still work; schema changes surface in PRs
instead of breaking production silently.

A consumer should:

1. Vendor a copy of `services.yaml` in their repo (e.g. at
   `data/services.yaml`).
2. Record the source SHA they synced from in a `services.yaml.lock`
   alongside (path, source SHA, sync timestamp).
3. Provide a `sync_catalog` script that pulls the latest from the
   canonical path and updates both files.
4. Validate the vendored file against `services.schema.json` on
   load — refuse to start if validation fails.
5. CI: re-validate on every commit.

The asset-engine repo at `~/development/asset_engine/` is the
reference implementation of this pattern; copy the script shape from
`scripts/sync_catalog.py` over there.

## Known consumers

| Consumer                                    | Vendored at                          | Pin policy        |
|---------------------------------------------|--------------------------------------|-------------------|
| `asset_engine` (FastAPI/HTMX UI; ~/development/asset_engine/) | `data/services.yaml` + `data/services.yaml.lock` | catalog_version=1 |

When you add a consumer, add a row here in the same PR that lands
the consumer. This list is what we audit when planning a
`catalog_version` bump.

## Service authoring notes

When **adding** a service to the catalog:

- **Source-of-truth precedence.** OpenAPI is convenient but often lies
  by omission (bare `string` types where the dispatch chain enforces
  a specific enum, missing required-only-in-the-handler fields, etc).
  The actual ground truth is, in order:
    1. The Pydantic input model — for field shape, types, required-ness.
    2. The handler / pipeline code — for enum dispatches and runtime
       validation.
    3. The Gradio UI / client code (when present) — for blessed
       defaults and slider ranges, since the model authors picked
       these for human-facing UX.
    4. The README — least reliable; rots fastest.
  Every catalog change should cite which of these was read. "OpenAPI
  said X" is not enough on its own — past audits have caught three
  bugs in a single service (ace-step) that all looked correct in
  OpenAPI.
- The reproducibility audit at the bottom of `services.yaml` MUST get
  a matching entry for any new service.
- If a service can't produce reproducible output (no seed AND
  non-deterministic model), that's a **bug in the service** — fix it
  there before adding to the catalog. The asset-engine relies on
  reproducibility to power "regenerate" / "fork" affordances.
- If the service has a non-permissive license, populate
  `license_warning:` with the user-facing language. The UI is
  required to surface it before a generation; missing it is a
  contract violation.
- Image refs SHOULD be digest-pinned (`@sha256:...`) for true
  reproducibility. Tag-pinned (`:v1.2.3`) is acceptable but
  vulnerable to upstream re-pushes.

When **removing** a service: see the versioning policy table.
Removal is a two-step process; surprise removal breaks consumers'
asset libraries (regenerate-from-history breaks).