esh-pfi-infrastructure/docs/asset-engine/design-brief.md

# Asset Generation Engine — Design Brief

For an internal UI that fronts the inference services on `irv-ml1`
catalogued in [`services.yaml`](services.yaml). Hand this to a design
agent before any pixels.

> **What this is.** A FastAPI + HTMX + Shoelace web app that replaces
> `curl` for routine use of ~12 self-hosted inference services
> (TTS, ASR, SFX, music). Single-tenant in v1; the data model and
> code seams are designed so v2/v3 can grow into multi-user with
> auth, tagging, projects, and async job queues without
> architectural churn.
>
> **Not** a public product. **Not** multi-tenant on day 1. **Not** a
> generic "AI playground" — it's specifically a tool for *generating
> reusable assets* (voiceover takes, hold tones, music beds, SFX,
> transcripts) that get pulled into other projects.

---

## 1. Locked architectural decisions

These shape the data model and the code seams. They are not
negotiable in v1, because retrofitting them after the database has
real rows in it is expensive.

### 1.1 Asset is a first-class entity

Every successful generation writes one `Asset` row:

```
Asset
  id                uuid primary key
  service_id        text     -- 'kokoro', 'sao', 'ace-step', ...
  service_version   int      -- catalog `version:` at generation time
  model_id          text     -- 'hexgrad/Kokoro-82M'
  model_revision    text?    -- HF SHA when known, else null
  image_ref         text     -- container image at generation time (tag or digest)
  params            json     -- the request body that produced this asset
  output_hash       text     -- sha256 of the output bytes
  output_path       text     -- 'outputs/<sha256>.<ext>'
  output_mime       text     -- 'audio/wav', 'image/png', 'text/plain'
  duration_s        float?   -- for audio/video
  byte_size         int
  created_by        int      -- FK -> User.id  (always 1 in v1)
  created_at        datetime
  tags              json     -- list[str]; empty in v1, surfaced in v2
  collection_id     uuid?    -- FK -> Collection; null in v1
  retention         text     -- 'ephemeral' | 'keep' | 'pinned'
  source_asset_id   uuid?    -- FK -> Asset; non-null when this is a fork/regenerate
```

The presence of this row from day 1 is what enables the v2/v3 stories
(library, search, dedupe, regenerate, share, export). Backfilling
this from server logs after the fact is grim — don't.

### 1.2 Content-addressed output storage

Outputs live at `outputs/<sha256-of-bytes>.<ext>`. The asset row
points at the hash. This gets us, for free:

- **Dedupe**: same prompt + seed + model produces the same bytes →
  same hash → one file on disk, multiple Asset rows referencing it.
- **Garbage collection**: `delete from blobs where ref_count = 0`.
- **Verification**: an Asset row whose `output_hash` doesn't match
  the bytes on disk is corrupt; we know to re-run.
- **Migration safety**: blob layout is stable; you can move
  `outputs/` to a different filesystem without touching the DB.

Path conventions like `outputs/2026/05/10/kokoro_<timestamp>.wav` rot
within months. Don't.

### 1.3 Reproducibility is a hard requirement

Every Asset must be regeneratable from its `params` row. This means:

- **Explicit seed** for any service that supports sampling. (SAO and
  ace-step do; everything else is deterministic-without-sampling and
  requires `image_ref` pinning instead.)
- **`model_revision`** when the upstream exposes a SHA. When it
  doesn't, `image_ref` becomes the reproducibility anchor.
- **Service schema versioning** — when `services.yaml` changes a
  field's shape, bump that service's `version:`. Old Asset rows
  remember which version they were generated against.

If a service can't produce reproducible output, that's a bug in the
*service* to fix before it goes in the catalog — not a UI concern.

### 1.4 Job table from day 1

Every Asset is produced by a Job:

```
Job
  id            uuid primary key
  asset_id      uuid?   -- FK -> Asset; null until done
  service_id    text
  params        json
  state         text    -- 'queued' | 'running' | 'done' | 'failed'
  error         text?
  worker        text?   -- which process picked it up; null in v1
  queued_at     datetime
  started_at    datetime?
  finished_at   datetime?
```

In v1 this is synchronous: insert → run inline → update → return.
In v2 it becomes a worker pool. In v3 it gains priority + per-user
quota. **The query surface stays the same throughout** — UI
templates that render "running jobs" or "job history" don't change.

### 1.5 Auth as a no-op dependency, not absent

Every request goes through a `current_user()` dependency:

```python
def current_user() -> User:
    # v1: hardcoded; v2: header-injection from upstream proxy;
    # v3: OIDC.
    return User(id=1, name="me")
```

Downstream code uses `request.user.id` from day 1. v2's swap is a
dependency-injection change, not a refactor. **A code base that
assumes "no users" cannot be retrofitted with auth without
touching everything.**

### 1.6 API surface ≠ UI surface

Every UI action should be powered by a JSON API route. The HTMX
templates call those API routes via `hx-post` / `hx-get` and render
the JSON into HTML partials, OR the API routes serve dual content
(JSON + partial) by route prefix:

- `POST /api/v1/jobs` — JSON body in, JSON body out
- `POST /ui/jobs` — same handler called server-side, returns HTMX partial

Pick the **separate-route** pattern, not content negotiation by
`Accept:` header. It's noisier but clearer; debugging API issues
in v3 with content negotiation in the way is hell.

### 1.7 Service catalog is git-versioned and additive

[`services.yaml`](services.yaml) is the contract. Adding a service is
a YAML edit + server reload — no template authoring, no Python
changes. Each service entry carries `version:`. Schema changes bump
the version. Old assets remember which version they were generated
against (via `Asset.service_version`).

### 1.8 Tags + collections plumbed in the model from day 1

`Asset.tags: list[str]`, `Asset.collection_id: uuid?`. **No UI
required in v1** — but the columns must exist. Adding the columns
later is a migration touching every existing row; harmless to leave
empty for now.

---

## 2. The contract: `services.yaml`

The UI's form generator and response renderer are dispatched off
[`services.yaml`](services.yaml). The agent should treat this file
as the contract — **everything in the UI is parameterized by it**.

### 2.1 Field-type vocabulary

The form generator must support exactly these field types. No
others. New input shapes added to a service must reduce to these:

| `type:`     | Renders as                                | Notes |
|-------------|-------------------------------------------|-------|
| `text`      | `<sl-input>` single-line                  | for short strings |
| `textarea`  | `<sl-textarea>` multi-line, autosize      | prompts, lyrics |
| `number`    | `<sl-input type=number>`                  | for integer/float without bounded range |
| `slider`    | `<sl-range>` + numeric readout            | requires `min`, `max`, optional `step` |
| `select`    | `<sl-select>`                             | static `options:` OR remote `source_url:` |
| `bool`      | `<sl-switch>`                             | |
| `file`      | `<sl-input type=file>` + drag-drop zone   | requires `accepted_types:` |
| `json`      | `<sl-textarea>` with monospace + JSON parse on submit | for arrays-of-floats, refs, etc. |

Field attributes (per service entry):

- `name` (required)
- `type` (required)
- `label` (defaults to titlecased name)
- `required` (default false)
- `optional: true` (sugar for `required: false`)
- `default`
- `min` / `max` / `step` (for `number` / `slider`)
- `options: [...]` (for `select` with static options)
- `source_url: ...` + `source_jsonpath: ...` (for `select` with
  dynamic options — the UI fetches once per session, caches)
- `description` (rendered as Shoelace help-text)
- `max_length`
- `accepted_types: [...]` (for `file`)

If a service's Pydantic model needs something this vocabulary can't
express, the catalog is wrong, not the vocabulary. **Do not extend
the vocabulary** — fix the catalog or fix the service.

### 2.2 Response-renderer vocabulary

| `response.type:` | Renders as                                    |
|------------------|-----------------------------------------------|
| `audio`          | inline `<audio controls>` + waveform thumbnail + download |
| `image`          | inline `<img>` + click-to-zoom + download    |
| `video`          | inline `<video controls>` + download         |
| `text`           | monospace block + copy button                |
| `json`           | collapsible JSON tree                        |
| `file`           | filename + size + download button            |

`response.mime` may be static OR derived from a request field via
`mime_from_field:` (e.g. Kokoro's `response_format` field
determines whether the output is audio/wav, audio/mpeg, etc.).

### 2.3 License warnings

Some services have `license_warning:` (Voxtral CC BY-NC, SAO
Stability Community, Fish-S2 research-only). The UI must surface
these warnings clearly when those services are selected, and again
on the asset detail view. The user is responsible for not shipping
restricted assets to commercial products; the UI's job is to make
it impossible to do so unconsciously.

---

## 3. Tech constraints

- **Backend**: FastAPI, Python 3.11+. Not Flask, not Django.
- **Frontend**: Server-rendered Jinja2 templates + HTMX swaps.
  No SPA framework. No build step. No npm. If a feature can't be
  built with HTMX swap targets and Shoelace components, redesign
  the feature.
- **Components**: Shoelace web components. Use them; don't
  re-style them into oblivion. The point is to inherit a coherent
  look without designing one.
- **Persistence**: SQLite via SQLAlchemy. WAL mode. Single file.
  v3 may upgrade to Postgres; the schema should not depend on
  SQLite-specific behavior (no `WITHOUT ROWID`, no `STRICT` table
  options that diverge from Postgres types).
- **Output storage**: filesystem under `outputs/`. Content-addressed
  per §1.2. No object storage in v1.
- **Process model**: single Python process serves both API and UI
  in v1. v2 may split into web + worker; the Job model already
  permits this.
- **Long jobs**: HTMX polling against `/ui/jobs/<id>` returns the
  current state and renders the asset when done. No WebSockets in
  v1. SSE acceptable if it simplifies a specific surface, but
  default to polling.

---

## 4. Required UI surfaces

These are the *surfaces*, not the *layouts*. The agent decides the
IA — whether they're separate pages, a workbench, a tabbed panel,
etc. But the application must contain at least these:

### 4.1 Service Picker

Some way to select which service to use. The catalog has 13 entries
across 4 categories (`tts`, `asr`, `sfx`, `music`); ace-step is one
service with 27 fields, kokoro is one with 4. The picker should
make category-level discovery easy and surface license warnings at
selection time.

### 4.2 Make (form + submit)

Auto-generated form per the field vocabulary in §2.1. Submit creates
a Job. For services with progressive disclosure (ace-step), basic
fields default-visible, advanced collapsed. Form must remember the
last-used values per service in the session (so iterating on a
prompt doesn't lose the seed).

### 4.3 Live job view

While a job is running, the UI shows progress. For services where
duration is known up front (SAO, ace-step), surface elapsed / est
remaining. For services where it isn't, surface elapsed only.
On completion, swap in the asset preview inline.

### 4.4 Asset detail

A single asset's full record: the output (rendered per §2.2), all
params, the curl that would reproduce it, "regenerate" (same
params), "fork" (open the form pre-filled with these params for
editing), "delete", retention controls. The curl form is a
deliberate v1 affordance — users who want to script later get a
copy-paste handle without leaving the UI.

### 4.5 Library

All assets, sorted recency-first. Filterable by service, by output
type, by date range. **This is the primary navigation surface for
v2** — design v1's IA so the library can grow naturally into the
front door, not get bolted on later.

### 4.6 Job queue (v1: history only)

Every job, ordered by `queued_at`. State (queued/running/done/failed),
duration, link to the resulting asset (if any). v1 may render this
as just a list of past jobs — but the data model already supports
queued/running, so the layout should accommodate them when v2 adds a
worker pool.

---

## 5. Open for the agent to decide

Don't pre-decide these. The agent earning their keep depends on
bringing opinions on:

- **IA**: separate pages? unified workbench with service-switcher?
  dashboard-of-tiles? Each has trade-offs given the 13-service
  catalog and the v2 multi-user direction.
- **Library navigation pattern**: timeline? grid? tag cloud?
  tree-by-collection? The v1 library has no tags and no
  collections, but the design should be obviously extensible.
- **Long-job surfacing**: toast? inline progress card? dedicated
  jobs panel that's always visible? Polling cadence?
- **"Make new" flow when the form is huge** (ace-step): single
  scrolling form with section headers? wizard with steps?
  basic-vs-advanced toggle? presets-as-templates?
- **Asset preview ergonomics**: inline play/scrub for audio
  (waveform UI)? Modal for full-screen images? Asset-comparison
  view (A/B same prompt different services)?
- **License-warning UX**: hard block with confirm? amber banner?
  per-asset watermark? Where in the flow does the warning live so
  it's noticed but not dismissed reflexively?
- **Curl-rebuilder UX**: button? always-visible expandable section?
  hidden in a "developer" panel? Be opinionated about how visible
  this should be.
- **Service picker affordances**: search? category tabs?
  recently-used? favorites? Considering 13 services and growing.

---

## 6. Out of scope (v1)

- **Auth UI** — the seam is in the code; no login screen needed.
- **Multi-user UI** — even though the data model supports it, v1
  is single-user. No "shared with you" lanes, no per-user dashboards.
- **Mobile-responsive** — desktop-only. v2 may revisit.
- **Theming / dark mode toggle** — pick one, ship it. Shoelace's
  default theming applies.
- **Real-time collaboration** — no presence, no co-editing.
- **Account management, billing, quotas** — not relevant.
- **Deployment/CI** — outside the design scope; assume a single
  Python process.
- **The ComfyUI integration** — `services.yaml` flags ComfyUI as
  `status: catalog-deferred`. Either link out to it externally OR
  design a placeholder spot for it; **do not try to fit it into the
  form-based catalog without the workflow-template wrapper**.

---

## 7. Deliverables

In this order, with checkpoints between:

1. **Architecture sketch** (one page): data flow from
   `services.yaml` → form-generator → request → Job → Asset →
   renderer. Identify the seams. ~30 minutes of thought, not a
   formal diagram.
2. **IA proposal** (one page or one wireframe): how the surfaces in
   §4 fit together. Annotated with rationale. **Pause here for
   review** before pixels.
3. **HTML/Jinja mockups** for the core surfaces. Real Shoelace
   components, real catalog entries (use `kokoro`, `sao`, and
   `ace-step` as the worked examples — they span the field-count
   range from 4 to 27 and cover all three response types you'll
   see in v1: audio, audio-with-waveform, audio-with-license-warning).
   No backend logic; mockups are static HTML driven by hand-edited
   `services.yaml` substitution.
4. **Field-type renderer reference**: for each entry in §2.1,
   demonstrate the rendering with a small example. This becomes
   the design system contract for v2 service additions.
5. **Response-renderer reference**: same for §2.2.
6. **Open-questions log**: a list of things the agent flagged as
   intentionally undecided + the reasoning. Useful when v2 starts.

**Not in scope as deliverables**: working backend code, database
migrations, deployment artifacts. Those follow once the IA is
agreed.

---

## 8. Constraints on the design agent

- **Don't extend the field-type vocabulary in §2.1**. If a service
  needs a date picker, that's because the catalog spec is wrong;
  fix the catalog or push back on the service.
- **Don't design auth flows**. The seam is in the code; the UI
  has no login.
- **Don't add features beyond §4**. If you find yourself wanting a
  "share asset" button or a "public gallery", that's v2.
- **Don't choose a JS framework**. HTMX + Shoelace is the rule. If
  a feature feels impossible without React, redesign the feature.
- **Don't theme/restyle Shoelace into a custom design system**. The
  whole point of pulling in a component library is to *not* do that.
  Use the defaults; pick a single accent color; move on.