vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs/asset-engine: catalog + UI design brief

Browse Source 
services.yaml: form-generator contract for the forthcoming
asset-generation UI. 13 inference services on irv-ml1 (TTS, ASR,
SFX, music) catalogued with field schemas extracted from Pydantic
models, response types, reproducibility audit, and license
warnings. ComfyUI flagged catalog-deferred (workflow-DAG API
doesn't fit a form-based UI without a per-asset-type wrapper).

design-brief.md: the prompt to give a frontend-design agent before
any pixels. Locks in the data-model decisions whose later cost is
asymmetric (asset-as-first-class entity, content-addressed output
storage, reproducibility hard requirement, job table, auth as a
no-op DI seam, API surface ≠ UI surface, schema versioning,
tags/collections plumbed in v1 with no UI). Defines a closed
field-type vocabulary (8 types) and response-renderer vocabulary
(6 types) — agent isn't allowed to extend them. Pre-decides the
required UI surfaces; leaves IA, library-nav pattern, long-job
UX, and big-form ergonomics open for the agent to opine on.
This commit is contained in:
Vuong Hoang
2026-05-10 19:20:58 -07:00
parent a727b93b1e
commit 8d8d45b7ca
2 changed files with 1273 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/asset-engine/design-brief.md
+401
View File
@@ -0,0 +1,401 @@
# Asset Generation Engine — Design Brief
For an internal UI that fronts the inference services on `irv-ml1`
catalogued in [`services.yaml`](services.yaml). Hand this to a design
agent before any pixels.
> **What this is.** A FastAPI + HTMX + Shoelace web app that replaces
> `curl` for routine use of ~12 self-hosted inference services
> (TTS, ASR, SFX, music). Single-tenant in v1; the data model and
> code seams are designed so v2/v3 can grow into multi-user with
> auth, tagging, projects, and async job queues without
> architectural churn.
>
> **Not** a public product. **Not** multi-tenant on day 1. **Not** a
> generic "AI playground" — it's specifically a tool for *generating
> reusable assets* (voiceover takes, hold tones, music beds, SFX,
> transcripts) that get pulled into other projects.
---
## 1. Locked architectural decisions
These shape the data model and the code seams. They are not
negotiable in v1, because retrofitting them after the database has
real rows in it is expensive.
### 1.1 Asset is a first-class entity
Every successful generation writes one `Asset` row:
```
Asset
id uuid primary key
service_id text -- 'kokoro', 'sao', 'ace-step', ...
service_version int -- catalog `version:` at generation time
model_id text -- 'hexgrad/Kokoro-82M'
model_revision text? -- HF SHA when known, else null
image_ref text -- container image at generation time (tag or digest)
params json -- the request body that produced this asset
output_hash text -- sha256 of the output bytes
output_path text -- 'outputs/<sha256>.<ext>'
output_mime text -- 'audio/wav', 'image/png', 'text/plain'
duration_s float? -- for audio/video
byte_size int
created_by int -- FK -> User.id (always 1 in v1)
created_at datetime
tags json -- list[str]; empty in v1, surfaced in v2
collection_id uuid? -- FK -> Collection; null in v1
retention text -- 'ephemeral' | 'keep' | 'pinned'
source_asset_id uuid? -- FK -> Asset; non-null when this is a fork/regenerate
```
The presence of this row from day 1 is what enables the v2/v3 stories
(library, search, dedupe, regenerate, share, export). Backfilling
this from server logs after the fact is grim — don't.
### 1.2 Content-addressed output storage
Outputs live at `outputs/<sha256-of-bytes>.<ext>`. The asset row
points at the hash. This gets us, for free:
- **Dedupe**: same prompt + seed + model produces the same bytes →
same hash → one file on disk, multiple Asset rows referencing it.
- **Garbage collection**: `delete from blobs where ref_count = 0`.
- **Verification**: an Asset row whose `output_hash` doesn't match
the bytes on disk is corrupt; we know to re-run.
- **Migration safety**: blob layout is stable; you can move
`outputs/` to a different filesystem without touching the DB.
Path conventions like `outputs/2026/05/10/kokoro_<timestamp>.wav` rot
within months. Don't.
### 1.3 Reproducibility is a hard requirement
Every Asset must be regeneratable from its `params` row. This means:
- **Explicit seed** for any service that supports sampling. (SAO and
ace-step do; everything else is deterministic-without-sampling and
requires `image_ref` pinning instead.)
- **`model_revision`** when the upstream exposes a SHA. When it
doesn't, `image_ref` becomes the reproducibility anchor.
- **Service schema versioning** — when `services.yaml` changes a
field's shape, bump that service's `version:`. Old Asset rows
remember which version they were generated against.
If a service can't produce reproducible output, that's a bug in the
*service* to fix before it goes in the catalog — not a UI concern.
### 1.4 Job table from day 1
Every Asset is produced by a Job:
```
Job
id uuid primary key
asset_id uuid? -- FK -> Asset; null until done
service_id text
params json
state text -- 'queued' | 'running' | 'done' | 'failed'
error text?
worker text? -- which process picked it up; null in v1
queued_at datetime
started_at datetime?
finished_at datetime?
```
In v1 this is synchronous: insert → run inline → update → return.
In v2 it becomes a worker pool. In v3 it gains priority + per-user
quota. **The query surface stays the same throughout** — UI
templates that render "running jobs" or "job history" don't change.
### 1.5 Auth as a no-op dependency, not absent
Every request goes through a `current_user()` dependency:
```python
def current_user() -> User:
# v1: hardcoded; v2: header-injection from upstream proxy;
# v3: OIDC.
return User(id=1, name="me")
```
Downstream code uses `request.user.id` from day 1. v2's swap is a
dependency-injection change, not a refactor. **A code base that
assumes "no users" cannot be retrofitted with auth without
touching everything.**
### 1.6 API surface ≠ UI surface
Every UI action should be powered by a JSON API route. The HTMX
templates call those API routes via `hx-post` / `hx-get` and render
the JSON into HTML partials, OR the API routes serve dual content
(JSON + partial) by route prefix:
- `POST /api/v1/jobs` — JSON body in, JSON body out
- `POST /ui/jobs` — same handler called server-side, returns HTMX partial
Pick the **separate-route** pattern, not content negotiation by
`Accept:` header. It's noisier but clearer; debugging API issues
in v3 with content negotiation in the way is hell.
### 1.7 Service catalog is git-versioned and additive
[`services.yaml`](services.yaml) is the contract. Adding a service is
a YAML edit + server reload — no template authoring, no Python
changes. Each service entry carries `version:`. Schema changes bump
the version. Old assets remember which version they were generated
against (via `Asset.service_version`).
### 1.8 Tags + collections plumbed in the model from day 1
`Asset.tags: list[str]`, `Asset.collection_id: uuid?`. **No UI
required in v1** — but the columns must exist. Adding the columns
later is a migration touching every existing row; harmless to leave
empty for now.
---
## 2. The contract: `services.yaml`
The UI's form generator and response renderer are dispatched off
[`services.yaml`](services.yaml). The agent should treat this file
as the contract — **everything in the UI is parameterized by it**.
### 2.1 Field-type vocabulary
The form generator must support exactly these field types. No
others. New input shapes added to a service must reduce to these:
| `type:` | Renders as | Notes |
|-------------|-------------------------------------------|-------|
| `text` | `<sl-input>` single-line | for short strings |
| `textarea` | `<sl-textarea>` multi-line, autosize | prompts, lyrics |
| `number` | `<sl-input type=number>` | for integer/float without bounded range |
| `slider` | `<sl-range>` + numeric readout | requires `min`, `max`, optional `step` |
| `select` | `<sl-select>` | static `options:` OR remote `source_url:` |
| `bool` | `<sl-switch>` | |
| `file` | `<sl-input type=file>` + drag-drop zone | requires `accepted_types:` |
| `json` | `<sl-textarea>` with monospace + JSON parse on submit | for arrays-of-floats, refs, etc. |
Field attributes (per service entry):
- `name` (required)
- `type` (required)
- `label` (defaults to titlecased name)
- `required` (default false)
- `optional: true` (sugar for `required: false`)
- `default`
- `min` / `max` / `step` (for `number` / `slider`)
- `options: [...]` (for `select` with static options)
- `source_url: ...` + `source_jsonpath: ...` (for `select` with
dynamic options — the UI fetches once per session, caches)
- `description` (rendered as Shoelace help-text)
- `max_length`
- `accepted_types: [...]` (for `file`)
If a service's Pydantic model needs something this vocabulary can't
express, the catalog is wrong, not the vocabulary. **Do not extend
the vocabulary** — fix the catalog or fix the service.
### 2.2 Response-renderer vocabulary
| `response.type:` | Renders as |
|------------------|-----------------------------------------------|
| `audio` | inline `<audio controls>` + waveform thumbnail + download |
| `image` | inline `<img>` + click-to-zoom + download |
| `video` | inline `<video controls>` + download |
| `text` | monospace block + copy button |
| `json` | collapsible JSON tree |
| `file` | filename + size + download button |
`response.mime` may be static OR derived from a request field via
`mime_from_field:` (e.g. Kokoro's `response_format` field
determines whether the output is audio/wav, audio/mpeg, etc.).
### 2.3 License warnings
Some services have `license_warning:` (Voxtral CC BY-NC, SAO
Stability Community, Fish-S2 research-only). The UI must surface
these warnings clearly when those services are selected, and again
on the asset detail view. The user is responsible for not shipping
restricted assets to commercial products; the UI's job is to make
it impossible to do so unconsciously.
---
## 3. Tech constraints
- **Backend**: FastAPI, Python 3.11+. Not Flask, not Django.
- **Frontend**: Server-rendered Jinja2 templates + HTMX swaps.
No SPA framework. No build step. No npm. If a feature can't be
built with HTMX swap targets and Shoelace components, redesign
the feature.
- **Components**: Shoelace web components. Use them; don't
re-style them into oblivion. The point is to inherit a coherent
look without designing one.
- **Persistence**: SQLite via SQLAlchemy. WAL mode. Single file.
v3 may upgrade to Postgres; the schema should not depend on
SQLite-specific behavior (no `WITHOUT ROWID`, no `STRICT` table
options that diverge from Postgres types).
- **Output storage**: filesystem under `outputs/`. Content-addressed
per §1.2. No object storage in v1.
- **Process model**: single Python process serves both API and UI
in v1. v2 may split into web + worker; the Job model already
permits this.
- **Long jobs**: HTMX polling against `/ui/jobs/<id>` returns the
current state and renders the asset when done. No WebSockets in
v1. SSE acceptable if it simplifies a specific surface, but
default to polling.
---
## 4. Required UI surfaces
These are the *surfaces*, not the *layouts*. The agent decides the
IA — whether they're separate pages, a workbench, a tabbed panel,
etc. But the application must contain at least these:
### 4.1 Service Picker
Some way to select which service to use. The catalog has 13 entries
across 4 categories (`tts`, `asr`, `sfx`, `music`); ace-step is one
service with 27 fields, kokoro is one with 4. The picker should
make category-level discovery easy and surface license warnings at
selection time.
### 4.2 Make (form + submit)
Auto-generated form per the field vocabulary in §2.1. Submit creates
a Job. For services with progressive disclosure (ace-step), basic
fields default-visible, advanced collapsed. Form must remember the
last-used values per service in the session (so iterating on a
prompt doesn't lose the seed).
### 4.3 Live job view
While a job is running, the UI shows progress. For services where
duration is known up front (SAO, ace-step), surface elapsed / est
remaining. For services where it isn't, surface elapsed only.
On completion, swap in the asset preview inline.
### 4.4 Asset detail
A single asset's full record: the output (rendered per §2.2), all
params, the curl that would reproduce it, "regenerate" (same
params), "fork" (open the form pre-filled with these params for
editing), "delete", retention controls. The curl form is a
deliberate v1 affordance — users who want to script later get a
copy-paste handle without leaving the UI.
### 4.5 Library
All assets, sorted recency-first. Filterable by service, by output
type, by date range. **This is the primary navigation surface for
v2** — design v1's IA so the library can grow naturally into the
front door, not get bolted on later.
### 4.6 Job queue (v1: history only)
Every job, ordered by `queued_at`. State (queued/running/done/failed),
duration, link to the resulting asset (if any). v1 may render this
as just a list of past jobs — but the data model already supports
queued/running, so the layout should accommodate them when v2 adds a
worker pool.
---
## 5. Open for the agent to decide
Don't pre-decide these. The agent earning their keep depends on
bringing opinions on:
- **IA**: separate pages? unified workbench with service-switcher?
dashboard-of-tiles? Each has trade-offs given the 13-service
catalog and the v2 multi-user direction.
- **Library navigation pattern**: timeline? grid? tag cloud?
tree-by-collection? The v1 library has no tags and no
collections, but the design should be obviously extensible.
- **Long-job surfacing**: toast? inline progress card? dedicated
jobs panel that's always visible? Polling cadence?
- **"Make new" flow when the form is huge** (ace-step): single
scrolling form with section headers? wizard with steps?
basic-vs-advanced toggle? presets-as-templates?
- **Asset preview ergonomics**: inline play/scrub for audio
(waveform UI)? Modal for full-screen images? Asset-comparison
view (A/B same prompt different services)?
- **License-warning UX**: hard block with confirm? amber banner?
per-asset watermark? Where in the flow does the warning live so
it's noticed but not dismissed reflexively?
- **Curl-rebuilder UX**: button? always-visible expandable section?
hidden in a "developer" panel? Be opinionated about how visible
this should be.
- **Service picker affordances**: search? category tabs?
recently-used? favorites? Considering 13 services and growing.
---
## 6. Out of scope (v1)
- **Auth UI** — the seam is in the code; no login screen needed.
- **Multi-user UI** — even though the data model supports it, v1
is single-user. No "shared with you" lanes, no per-user dashboards.
- **Mobile-responsive** — desktop-only. v2 may revisit.
- **Theming / dark mode toggle** — pick one, ship it. Shoelace's
default theming applies.
- **Real-time collaboration** — no presence, no co-editing.
- **Account management, billing, quotas** — not relevant.
- **Deployment/CI** — outside the design scope; assume a single
Python process.
- **The ComfyUI integration** — `services.yaml` flags ComfyUI as
`status: catalog-deferred`. Either link out to it externally OR
design a placeholder spot for it; **do not try to fit it into the
form-based catalog without the workflow-template wrapper**.
---
## 7. Deliverables
In this order, with checkpoints between:
1. **Architecture sketch** (one page): data flow from
`services.yaml` → form-generator → request → Job → Asset →
renderer. Identify the seams. ~30 minutes of thought, not a
formal diagram.
2. **IA proposal** (one page or one wireframe): how the surfaces in
§4 fit together. Annotated with rationale. **Pause here for
review** before pixels.
3. **HTML/Jinja mockups** for the core surfaces. Real Shoelace
components, real catalog entries (use `kokoro`, `sao`, and
`ace-step` as the worked examples — they span the field-count
range from 4 to 27 and cover all three response types you'll
see in v1: audio, audio-with-waveform, audio-with-license-warning).
No backend logic; mockups are static HTML driven by hand-edited
`services.yaml` substitution.
4. **Field-type renderer reference**: for each entry in §2.1,
demonstrate the rendering with a small example. This becomes
the design system contract for v2 service additions.
5. **Response-renderer reference**: same for §2.2.
6. **Open-questions log**: a list of things the agent flagged as
intentionally undecided + the reasoning. Useful when v2 starts.
**Not in scope as deliverables**: working backend code, database
migrations, deployment artifacts. Those follow once the IA is
agreed.
---
## 8. Constraints on the design agent
- **Don't extend the field-type vocabulary in §2.1**. If a service
needs a date picker, that's because the catalog spec is wrong;
fix the catalog or push back on the service.
- **Don't design auth flows**. The seam is in the code; the UI
has no login.
- **Don't add features beyond §4**. If you find yourself wanting a
"share asset" button or a "public gallery", that's v2.
- **Don't choose a JS framework**. HTMX + Shoelace is the rule. If
a feature feels impossible without React, redesign the feature.
- **Don't theme/restyle Shoelace into a custom design system**. The
whole point of pulling in a component library is to *not* do that.
Use the defaults; pick a single accent color; move on.
docs/asset-engine/services.yaml
+872
View File
@@ -0,0 +1,872 @@
# services.yaml — inference services catalog on irv-ml1
#
# Drives the forthcoming asset-generation UI: each entry produces one
# form (auto-generated from `fields:`) and one response renderer
# (dispatched on `response.type`).
#
# Generated by an Explore-agent pass over stacks/<service>/{README.md,
# compose.yaml,.env.example,server.py|infer-api.py}. Pydantic models
# in code are the ground truth where they disagree with READMEs.
#
# Schema is v1; bump per-service `version:` when fields change so old
# Asset rows can still be interpreted.
services:
- id: kokoro
name: Kokoro 82M TTS
description: >
Lowest-latency English TTS (82M params, fp16). ~300ms TTFA, 35–100x realtime,
60+ built-in voices in 8 languages. Voice mixing via inline weights.
OpenAI-compatible /v1/audio/speech.
category: tts
version: 1
host: irv-ml1
endpoint: http://10.100.79.3:8193/v1/audio/speech
method: POST
content_type: application/json
model:
id: hexgrad/Kokoro-82M
revision: null
image: ghcr.io/remsky/kokoro-fastapi-gpu:v0.2.4-master
fields:
- name: input
type: textarea
label: Text
required: true
max_length: 5000
- name: voice
type: select
label: Voice
source_url: http://10.100.79.3:8193/v1/audio/voices
source_jsonpath: $.voices[*]
default: af_bella
description: >
60+ built-in voices. Custom blends: af_bella(2)+af_aoede(1) syntax for
weighted mixing. Persistent custom voices via playbooks/blend-kokoro-voice.yaml.
- name: response_format
type: select
options: [wav, mp3, opus, flac, pcm]
default: wav
- name: stream
type: bool
default: false
description: Phrase-by-phrase streaming via chunked HTTP.
response:
type: audio
mime_from_field: response_format
reproducibility:
seedable: false
deterministic: true
notes: >
No seed parameter. Model fully deterministic (no sampling); identical
params always produce identical bytes. Image tag is mutable; pin to
digest for v3 reproducibility audit.
estimated_latency:
cold_start_s: 2
warm_per_unit: "~300ms TTFA, 35–100x realtime"
license: Apache-2.0
notes: |
Voice mixing: voice="name1(w1)+name2(w2)+..." normalizes weights.
Custom voices persist at /worktank/kokoro/user_voices (bind-mounted).
- id: chatterbox
name: Chatterbox Turbo TTS
description: >
Resemble AI's low-latency English TTS (350M, ~75ms TTFB, 6× realtime).
Zero-shot voice cloning from ~5s reference. 9 paralinguistic tags.
category: tts
version: 1
host: irv-ml1
endpoint: http://10.100.79.3:8196/v1/audio/speech
method: POST
content_type: application/json
model:
id: ResembleAI/chatterbox-turbo
revision: null
image: devnen/Chatterbox-TTS-Server:latest
fields:
- name: input
type: textarea
label: Text (with optional [tags])
required: true
max_length: 5000
description: >
Inline tags: [laugh] [chuckle] [sigh] [gasp] [cough] [clear throat]
[sniff] [groan] [shush]. Turbo loses base-Chatterbox's exaggeration knob.
- name: model
type: select
options: [chatterbox-turbo]
default: chatterbox-turbo
- name: voice
type: select
label: Voice
default: alloy
description: >
Built-in OpenAI-compat aliases (alloy, echo, fable, onyx, nova, shimmer).
Cloned: 5–15s WAV files in /worktank/chatterbox/reference_audio/.
- name: response_format
type: select
options: [wav, opus, aac, flac, pcm_s16]
default: wav
- name: stream
type: bool
default: false
response:
type: audio
mime_from_field: response_format
reproducibility:
seedable: false
deterministic: true
notes: >
No seed. Wrapper repo updates ~weekly; pin SHA in .env. PerTh watermark
unconditionally applied (Resemble policy).
estimated_latency:
cold_start_s: 3
warm_per_unit: "~75ms TTFB, 6× realtime"
license: MIT
notes: |
Python 3.10 only (wrapper hardcoding).
Multilingual variant (23 languages) also available via .env.
- id: index-tts
name: IndexTTS-2
description: >
Bilibili's emotion-controllable zero-shot TTS. Disentangled emotion control
(timbre from one reference, emotion from another). 22050 Hz fixed output.
category: tts
version: 1
host: irv-ml1
endpoint: http://10.100.79.3:8192/v1/audio/speech
method: POST
content_type: application/json
model:
id: IndexTeam/IndexTTS-2
revision: null
image: local/index-tts:v2
fields:
- name: input
type: textarea
label: Text
required: true
max_length: 5000
- name: voice
type: select
label: Speaker Voice
description: <name>.wav in /worktank/index-tts/voices/. 5–30s clean clips.
- name: response_format
type: select
options: [wav]
default: wav
description: 22050 Hz PCM_16 mono only; no negotiation.
- name: stream
type: bool
default: false
description: >
Segment-level streaming (~120 tokens). Streaming WAV uses placeholder
data-length (0xFFFFFFFF); browsers fine, strict parsers may complain.
- name: emotion_voice
type: select
label: Emotion Reference Voice
optional: true
description: <name>.wav in /worktank/index-tts/emotions/.
- name: emotion_vector
type: json
label: Emotion Vector
optional: true
description: >
8 floats [happy, angry, sad, afraid, disgusted, melancholic, surprised, calm],
each 0.0–1.0.
- name: emotion_text
type: textarea
label: Emotion Description (free text)
optional: true
- name: emotion_alpha
type: slider
min: 0.0
max: 1.0
default: 1.0
label: Emotion Strength
response:
type: audio
mime: audio/wav
reproducibility:
seedable: false
deterministic: true
notes: >
No seed. 22050 Hz hardcoded — resample in caller if 24/48 kHz needed.
Precedence if multiple emotion sources: emotion_voice > vector > text.
estimated_latency:
cold_start_s: 5
warm_per_unit: "segment-latency streaming"
license: "Custom Bilibili (free at small scale; commercial tier 100M MAU)"
notes: |
Three-way mutual-exclusion among emotion_voice / emotion_vector / emotion_text;
precedence as above. UI should expose this as a single picker.
- id: qwen3-tts
name: Qwen3-TTS 1.7B
description: >
Alibaba's open English-first TTS (Apache 2.0). 10 languages, 97ms TTFB,
instruction-driven emotion, voice cloning.
category: tts
version: 1
host: irv-ml1
endpoint: http://10.100.79.3:8191/v1/audio/speech
method: POST
content_type: application/json
model:
id: Qwen/Qwen3-TTS-12Hz-1.7B
revision: null
image: local/qwen3-tts:v2
fields:
- name: model
type: select
options: [Qwen/Qwen3-TTS-12Hz-1.7B, Qwen/Qwen3-TTS-12Hz-0.6B-Base]
default: Qwen/Qwen3-TTS-12Hz-1.7B
- name: input
type: textarea
label: Text
required: true
max_length: 5000
- name: voice
type: select
label: Voice
description: >
Built-in or cloned. For cloning: clone:<name> where <name> is a profile
dir under /worktank/qwen3-tts/voices/profiles/. Create via /voice-studio
web UI or manually (meta.json + reference.wav).
- name: instructions
type: textarea
label: Emotion/Style Instructions
optional: true
description: >
Natural-language directive (e.g. "speak with cold contempt").
English instructions verified working ~2026-04.
- name: response_format
type: select
options: [wav, mp3, pcm]
default: wav
response:
type: audio
mime_from_field: response_format
reproducibility:
seedable: false
deterministic: true
estimated_latency:
cold_start_s: 5
warm_per_unit: "~97ms TTFB"
license: Apache-2.0
notes: |
Voice cloning shape differs from CosyVoice: profile-based, not voice-id-based.
- id: cosyvoice
name: CosyVoice 3 (Multilingual)
description: >
FunAudioLLM's multilingual expressive TTS. 18+ Chinese dialects + 8 other langs.
English prosody not ElevenLabs-grade — use Qwen3-TTS for English.
category: tts
version: 1
host: irv-ml1
endpoint: http://10.100.79.3:8190/v1/audio/speech
method: POST
content_type: application/json
model:
id: FunAudioLLM/Fun-CosyVoice3-0.5B-2512
revision: null
image: neosun/cosyvoice:v1.3.2
fields:
- name: model
type: select
options: [cosyvoice-v3, cosyvoice-v2]
default: cosyvoice-v3
- name: input
type: textarea
label: Text (with optional XML emotion tags)
required: true
max_length: 5000
description: >
For English: use XML tags <angry>, <sad>, <surprised>, <fast>, <whisper>, etc.
NOT the instruct field — English instruct values get vocalized literally
(upstream bug).
- name: voice
type: select
label: Voice (cloned only — no presets)
description: >
Create via POST /v1/voices/create (multipart with reference audio ≤30s).
Reference must be 16kHz mono ≤30s; longer = AssertionError.
- name: response_format
type: select
options: [wav]
default: wav
- name: speed
type: slider
min: 0.5
max: 2.0
default: 1.0
response:
type: audio
mime: audio/wav
reproducibility:
seedable: false
deterministic: true
notes: >
Reference audio MUST be ≤30s (16kHz mono). instruct field broken for English.
estimated_latency:
cold_start_s: 5
warm_per_unit: "~150ms TTFB streaming"
license: Apache-2.0
notes: |
GOTCHA: instruct field is Chinese-context only — UI should hide it for English
or surface a strong warning.
- id: fish-s2
name: Fish Audio S2-Pro
description: >
Fishaudio's richest-paralinguistic English TTS (15,000+ inline tags).
Trained 10M+ hours, dual-AR, ~150ms streaming TTFB. Released March 2026.
category: tts
version: 1
host: irv-ml1
endpoint: http://10.100.79.3:8195/v1/tts
method: POST
content_type: application/json
model:
id: fishaudio/s2-pro
revision: null
image: local/fish-s2:v1
fields:
- name: text
type: textarea
label: Text (with optional [tags])
required: true
max_length: 5000
description: >
Inline tags: [laugh] [whispers] [super happy] [sigh] [excited]
[heavy breathing] [angry] [sleepy] [crying] [surprise] ... (15,000+).
- name: references
type: json
label: Voice References
optional: true
description: >
Array of {audio: "/app/references/<file>.wav", text: "transcript"}.
Files under /worktank/fish-s2/references/ on host.
response:
type: audio
mime: audio/wav
reproducibility:
seedable: false
deterministic: true
estimated_latency:
cold_start_s: 8
warm_per_unit: "~150ms TTFB"
license: "Research-only (Fishaudio terms; non-commercial)"
license_warning: |
Research/internal only. Not clear for commercial use. Same flag as
Voxtral and SAO — UI must surface when output destined for products.
notes: |
NOT OpenAI-compatible: only /v1/tts. No /v1/audio/voices, no /v1/models.
Voice discovery is manual (host filesystem).
- id: kyutai-tts
name: Kyutai TTS 1.6B EN/FR
description: >
Kyutai's bilingual streaming TTS (1.6B, 2.5M hours). Heritage from Mimi codec
+ Moshi dialogue framework. OpenAI-compat HTTP wrapper over Rust streaming core.
category: tts
version: 1
host: irv-ml1
endpoint: http://10.100.79.3:8198/v1/audio/speech
method: POST
content_type: application/json
model:
id: kyutai/tts-1.6b-en_fr
revision: null
image: local/kyutai-tts:v1
fields:
- name: model
type: select
options: [tts-1.6b-en_fr]
default: tts-1.6b-en_fr
- name: input
type: textarea
label: Text
required: true
max_length: 5000
- name: voice
type: select
label: Voice
source_url: http://10.100.79.3:8198/v1/audio/voices
- name: response_format
type: select
options: [wav, mp3, pcm]
default: wav
- name: stream
type: bool
default: false
response:
type: audio
mime_from_field: response_format
reproducibility:
seedable: false
deterministic: true
notes: >
Wrapper adds Python overhead vs Kyutai's bare 220ms claim.
estimated_latency:
cold_start_s: 4
warm_per_unit: "~220–400ms TTFB (with wrapper)"
license: "TBD (Kyutai — verify)"
- id: vibevoice
name: VibeVoice 1.5B (Long-form)
description: >
Microsoft's diffusion-based long-form multi-speaker TTS. Multi-minute scripts
with speaker switching. Not for low-latency single-line use.
category: tts
version: 1
host: irv-ml1
endpoint: http://10.100.79.3:8194/v1/audio/speech
method: POST
content_type: application/json
model:
id: microsoft/VibeVoice-1.5B
revision: null
image: local/vibevoice:v1
fields:
- name: model
type: select
options: [vibevoice]
default: vibevoice
- name: input
type: textarea
label: Text (or Speaker N: ... script)
required: true
description: >
Single-speaker: plain text. Multi-speaker: "Speaker 0: ...\nSpeaker 1: ..."
via /v1/vibevoice/generate (extended endpoint).
- name: voice
type: select
label: Voice
default: Carter
description: >
Built-in: Carter, Davis, Emma, Frank, Grace, Mike, Samuel.
Custom: drop WAV/MP3/FLAC/M4A into /worktank/vibevoice/voices/, restart container.
Voice cloning training NOT released by Microsoft.
- name: response_format
type: select
options: [wav, mp3]
default: wav
- name: stream
type: bool
default: false
description: Single-shot endpoint doesn't stream; segment-level on multi-speaker.
response:
type: audio
mime_from_field: response_format
reproducibility:
seedable: false
deterministic: true
estimated_latency:
cold_start_s: 8
warm_per_unit: "30–60s first generation; segment-based after"
license: MIT
notes: |
flash_attention_2 default; sdpa fallback.
7B variant (rsxdalv/VibeVoice-Large, ~18GB) needs int8 for <10GB VRAM.
- id: voxtral
name: Voxtral 4B TTS
description: >
Mistral AI's 4B multilingual streaming TTS (CC BY-NC — research/internal only).
8 languages, 70ms model latency, 9.7× realtime. Served via vLLM-Omni.
category: tts
version: 1
host: irv-ml1
endpoint: http://10.100.79.3:8197/v1/audio/speech
method: POST
content_type: application/json
model:
id: mistralai/Voxtral-4B-TTS-2603
revision: null
image: vllm/vllm-omni:v0.18.0
fields:
- name: model
type: select
options: [mistralai/Voxtral-4B-TTS-2603]
default: mistralai/Voxtral-4B-TTS-2603
- name: input
type: textarea
label: Text
required: true
max_length: 5000
- name: voice
type: select
label: Voice
source_url: http://10.100.79.3:8197/v1/audio/voices
default: alloy
- name: response_format
type: select
options: [wav, mp3, pcm]
default: wav
- name: stream
type: bool
default: false
response:
type: audio
mime_from_field: response_format
reproducibility:
seedable: false
deterministic: true
notes: vLLM-Omni v0.18.0+ required (first Voxtral-aware release, 2026-03-29).
estimated_latency:
cold_start_s: 6
warm_per_unit: "~70ms model latency, 9.7× realtime"
license: "CC BY-NC (Mistral) — non-commercial only"
license_warning: |
NOT clear for commercial use. UI must surface this when output destined
for products. Use Kokoro/Chatterbox/Fish/IndexTTS/Qwen3/CosyVoice for
commercial paths.
- id: parakeet
name: Parakeet TDT (ASR)
description: >
NVIDIA Parakeet-TDT 0.6B (int8 ONNX, ~400MB) via sherpa-onnx.
Transcription only.
category: asr
version: 1
host: irv-ml1
endpoint: http://10.100.79.3:8765/transcribe
method: POST
content_type: multipart/form-data
model:
id: nvidia/parakeet-tdt-v2-en-int8
revision: null
image: local/parakeet:sherpa-onnx-v2
fields:
- name: file
type: file
label: Audio File
required: true
accepted_types: [audio/wav, audio/mp3, audio/flac, audio/ogg]
response:
type: text
output_field: text
reproducibility:
seedable: false
deterministic: true
estimated_latency:
cold_start_s: 1
warm_per_unit: "~realtime"
license: "Proprietary (NVIDIA model) + Apache-2.0 (sherpa-onnx)"
notes: |
Aliased at /v1/audio/transcriptions for OpenAI compat.
v3 model (25 languages) available via env-only change.
- id: stable-audio-open
name: Stable Audio Open 1.0 (SFX)
description: >
Stability AI's text-to-audio diffusion. SFX/foley/ambience only — NOT music.
Max 47s clips. Seeded, fully deterministic.
category: sfx
version: 1
host: irv-ml1
endpoint: http://10.100.79.3:8211/v1/audio/sfx
method: POST
content_type: application/json
model:
id: stabilityai/stable-audio-open-1.0
revision: null
image: local/stable-audio-open:v1
fields:
- name: prompt
type: textarea
label: Prompt
required: true
- name: negative_prompt
type: textarea
label: Negative Prompt
default: "Low quality."
- name: duration
type: slider
min: 0.5
max: 47.0
default: 10.0
label: Duration (seconds)
- name: steps
type: slider
min: 10
max: 300
default: 100
label: Diffusion Steps
- name: cfg_scale
type: slider
min: 0.0
max: 20.0
default: 7.0
label: CFG Scale
- name: seed
type: number
label: Seed
optional: true
response:
type: audio
mime: audio/wav
reproducibility:
seedable: true
deterministic: true
notes: >
Identical seed+prompt+steps+cfg = bit-identical bytes. Wrapper serializes
concurrent requests via asyncio.Lock (StableAudioPipeline not reentrant).
estimated_latency:
cold_start_s: 3
warm_per_unit: "~30–60s per 10s clip"
license: "Stability AI Community (non-commercial)"
license_warning: |
Non-commercial only. Same flag as Voxtral for commercial paths.
- id: ace-step
name: ACE-Step 1.5 (Music)
description: >
Apache-2.0 hybrid diffusion+LLM music generation. Multi-minute lyric-aware
songs with vocals + instrumentation.
category: music
version: 1
host: irv-ml1
endpoint: http://10.100.79.3:8210/generate
method: POST
content_type: application/json
model:
id: ace-step/ACE-Step
revision: main
image: local/ace-step:v1
fields:
- name: prompt
type: textarea
label: Musical Prompt
required: true
description: Style/mood/instrumentation, e.g. "uplifting pop with synth leads".
- name: lyrics
type: textarea
label: Lyrics
optional: true
- name: audio_duration
type: slider
min: 5.0
max: 600.0
default: 30.0
label: Duration (seconds)
- name: audio_format
type: select
options: [wav, mp3, flac]
default: wav
- name: infer_step
type: number
default: 20
label: Inference Steps
- name: guidance_scale
type: slider
min: 1.0
max: 15.0
default: 7.5
- name: scheduler_type
type: select
options: [linear, squared, sqrt]
default: linear
- name: cfg_type
type: select
options: [none, cfg, cfg_rw]
default: cfg
- name: omega_scale
type: slider
min: 0.0
max: 1.0
default: 0.5
- name: actual_seeds
type: json
label: Seeds
default: [42]
- name: guidance_interval
type: slider
min: 0.0
max: 1.0
default: 0.0
- name: guidance_interval_decay
type: slider
min: 0.0
max: 1.0
default: 1.0
- name: min_guidance_scale
type: slider
min: 0.0
max: 10.0
default: 1.0
- name: use_erg_tag
type: bool
default: false
- name: use_erg_lyric
type: bool
default: false
- name: use_erg_diffusion
type: bool
default: false
- name: oss_steps
type: json
default: []
- name: guidance_scale_text
type: slider
min: 0.0
max: 15.0
default: 0.0
- name: guidance_scale_lyric
type: slider
min: 0.0
max: 15.0
default: 0.0
- name: audio2audio_enable
type: bool
default: false
- name: ref_audio_strength
type: slider
min: 0.0
max: 1.0
default: 0.5
- name: ref_audio_input
type: text
label: Reference Audio Path
optional: true
- name: lora_name_or_path
type: text
label: LoRA Repo/Path
optional: true
- name: lora_weight
type: slider
min: 0.0
max: 2.0
default: 1.0
- name: bf16
type: bool
default: true
- name: torch_compile
type: bool
default: false
- name: device_id
type: number
default: 0
label: GPU Device Index
response:
type: audio
mime_from_field: audio_format
output_field: output_path
reproducibility:
seedable: true
deterministic: true
notes: >
actual_seeds parameter exposed; identical seeds + params = identical audio.
Local infer-api.py patches upstream's broken 24-arg pipeline signature
(was 18 in upstream — caused crashes with audio_duration in `format` slot).
estimated_latency:
cold_start_s: 30
warm_per_unit: "~10–60s depending on audio_duration + infer_step"
license: Apache-2.0
notes: |
27-field surface — UI must do progressive disclosure (basic/advanced).
Gradio UI fallback: docker exec -it ace-step python3 acestep/gui.py.
- id: comfyui
name: ComfyUI (workflow engine — catalog-deferred)
description: >
Node-based diffusion workflow engine. Native API is workflow JSON DAG, not
form fields. Catalog-deferred until per-asset-type workflow templates are
defined and a wrapper maps form inputs to template node parameters.
category: image
status: catalog-deferred
version: 1
host: irv-ml1
endpoint: http://10.100.79.3:8188
method: WebSocket + POST
content_type: application/json
model:
id: "various (SD, SDXL, Flux, ControlNet, LoRA, upscalers)"
revision: null
image: mmartial/comfyui-nvidia-docker:ubuntu24_cuda12.8-20260312
response:
type: image
mime: image/png
reproducibility:
seedable: true
deterministic: true
notes: >
Seed is a workflow node parameter. Reproducibility requires persisting
full workflow JSON + seed alongside the asset.
license: GPL-3.0
notes: |
To bring into the catalog: define workflow templates per asset-type
(portrait, landscape, sfx-thumbnail, ...), build a wrapper that does
form-field-to-node-parameter substitution, submit via POST /prompt, poll
/history. Until then, expose ComfyUI as an external link in the UI.
User state at /worktank/comfyui/basedir/.
# Reproducibility audit — answers per service: (a) seedable, (b) model
# deterministic without seed, (c) image tag mutable (security/reproducibility risk).
reproducibility_audit:
- service: kokoro
seedable: false
model_deterministic: true
image_tag_mutable: true
notes: ":v0.2.4-master is mutable; pin to digest for v3."
- service: chatterbox
seedable: false
model_deterministic: true
image_tag_mutable: false
notes: "PerTh watermark unconditional (Resemble policy)."
- service: index-tts
seedable: false
model_deterministic: true
image_tag_mutable: false
notes: "22050 Hz hardcoded — caller must resample."
- service: qwen3-tts
seedable: false
model_deterministic: true
image_tag_mutable: false
- service: cosyvoice
seedable: false
model_deterministic: true
image_tag_mutable: false
notes: "instruct field broken for English; XML tags only."
- service: fish-s2
seedable: false
model_deterministic: true
image_tag_mutable: false
notes: "Research-only license — non-commercial."
- service: kyutai-tts
seedable: false
model_deterministic: true
image_tag_mutable: false
- service: vibevoice
seedable: false
model_deterministic: true
image_tag_mutable: false
notes: "Voice cloning training not released."
- service: voxtral
seedable: false
model_deterministic: true
image_tag_mutable: false
notes: "CC BY-NC — non-commercial."
- service: parakeet
seedable: false
model_deterministic: true
image_tag_mutable: false
- service: stable-audio-open
seedable: true
model_deterministic: true
image_tag_mutable: false
notes: "Wrapper serializes concurrent requests (StableAudioPipeline not reentrant)."
- service: ace-step
seedable: true
model_deterministic: true
image_tag_mutable: false
notes: "Local infer-api.py patches upstream's broken pipeline signature."
- service: comfyui
seedable: true
model_deterministic: true
image_tag_mutable: false
notes: "Reproducibility requires persisting full workflow JSON + seed."