esh-pfi-infrastructure/docs/archive/asset-engine/design-brief.md

Asset Generation Engine — Design Brief

Archived 2026-05-14. Describes the pre-implementation design intent for asset-engine. Implementation has shipped: source at vh/asset-engine, deployed to ana-docker via stacks/asset-engine/. The live catalog contract is docs/asset-engine/CATALOG-CONTRACT.md; the catalog itself is docs/asset-engine/services.yaml. This brief is kept for the design rationale it carries (why Asset is a first-class entity, why v1 is synchronous, what v2/v3 seams exist), not as a guide to current behavior.

For an internal UI that fronts the inference services on irv-ml1 catalogued in services.yaml. Hand this to a design agent before any pixels.

What this is. A FastAPI + HTMX + Shoelace web app that replaces curl for routine use of ~12 self-hosted inference services (TTS, ASR, SFX, music). Single-tenant in v1; the data model and code seams are designed so v2/v3 can grow into multi-user with auth, tagging, projects, and async job queues without architectural churn.

Not a public product. Not multi-tenant on day 1. Not a generic "AI playground" — it's specifically a tool for generating reusable assets (voiceover takes, hold tones, music beds, SFX, transcripts) that get pulled into other projects.

---

1. Locked architectural decisions

These shape the data model and the code seams. They are not negotiable in v1, because retrofitting them after the database has real rows in it is expensive.

1.1 Asset is a first-class entity

Every successful generation writes one Asset row:

Asset
  id                uuid primary key
  service_id        text     -- 'kokoro', 'sao', 'ace-step', ...
  service_version   int      -- catalog `version:` at generation time
  model_id          text     -- 'hexgrad/Kokoro-82M'
  model_revision    text?    -- HF SHA when known, else null
  image_ref         text     -- container image at generation time (tag or digest)
  params            json     -- the request body that produced this asset
  output_hash       text     -- sha256 of the output bytes
  output_path       text     -- 'outputs/<sha256>.<ext>'
  output_mime       text     -- 'audio/wav', 'image/png', 'text/plain'
  duration_s        float?   -- for audio/video
  byte_size         int
  created_by        int      -- FK -> User.id  (always 1 in v1)
  created_at        datetime
  tags              json     -- list[str]; empty in v1, surfaced in v2
  collection_id     uuid?    -- FK -> Collection; null in v1
  retention         text     -- 'ephemeral' | 'keep' | 'pinned'
  source_asset_id   uuid?    -- FK -> Asset; non-null when this is a fork/regenerate

The presence of this row from day 1 is what enables the v2/v3 stories (library, search, dedupe, regenerate, share, export). Backfilling this from server logs after the fact is grim — don't.

1.2 Content-addressed output storage

Outputs live at outputs/<sha256-of-bytes>.<ext>. The asset row points at the hash. This gets us, for free:

• Dedupe: same prompt + seed + model produces the same bytes → same hash → one file on disk, multiple Asset rows referencing it.
• Garbage collection: delete from blobs where ref_count = 0.
• Verification: an Asset row whose output_hash doesn't match the bytes on disk is corrupt; we know to re-run.
• Migration safety: blob layout is stable; you can move outputs/ to a different filesystem without touching the DB.

Path conventions like outputs/2026/05/10/kokoro_<timestamp>.wav rot within months. Don't.

1.3 Reproducibility is a hard requirement

Every Asset must be regeneratable from its params row. This means:

• Explicit seed for any service that supports sampling. (SAO and ace-step do; everything else is deterministic-without-sampling and requires image_ref pinning instead.)
• model_revision when the upstream exposes a SHA. When it doesn't, image_ref becomes the reproducibility anchor.
• Service schema versioning — when services.yaml changes a field's shape, bump that service's version:. Old Asset rows remember which version they were generated against.

If a service can't produce reproducible output, that's a bug in the service to fix before it goes in the catalog — not a UI concern.

1.4 Job table from day 1

Every Asset is produced by a Job:

Job
  id            uuid primary key
  asset_id      uuid?   -- FK -> Asset; null until done
  service_id    text
  params        json
  state         text    -- 'queued' | 'running' | 'done' | 'failed'
  error         text?
  worker        text?   -- which process picked it up; null in v1
  queued_at     datetime
  started_at    datetime?
  finished_at   datetime?

In v1 this is synchronous: insert → run inline → update → return. In v2 it becomes a worker pool. In v3 it gains priority + per-user quota. The query surface stays the same throughout — UI templates that render "running jobs" or "job history" don't change.

1.5 Auth as a no-op dependency, not absent

Every request goes through a current_user() dependency:

def current_user() -> User:
    # v1: hardcoded; v2: header-injection from upstream proxy;
    # v3: OIDC.
    return User(id=1, name="me")

Downstream code uses request.user.id from day 1. v2's swap is a dependency-injection change, not a refactor. A code base that assumes "no users" cannot be retrofitted with auth without touching everything.

1.6 API surface ≠ UI surface

Every UI action should be powered by a JSON API route. The HTMX templates call those API routes via hx-post / hx-get and render the JSON into HTML partials, OR the API routes serve dual content (JSON + partial) by route prefix:

• POST /api/v1/jobs — JSON body in, JSON body out
• POST /ui/jobs — same handler called server-side, returns HTMX partial

Pick the separate-route pattern, not content negotiation by Accept: header. It's noisier but clearer; debugging API issues in v3 with content negotiation in the way is hell.

1.7 Service catalog is git-versioned and additive

services.yaml is the contract. Adding a service is a YAML edit + server reload — no template authoring, no Python changes. Each service entry carries version:. Schema changes bump the version. Old assets remember which version they were generated against (via Asset.service_version).

1.8 Tags + collections plumbed in the model from day 1

Asset.tags: list[str], Asset.collection_id: uuid?. No UI required in v1 — but the columns must exist. Adding the columns later is a migration touching every existing row; harmless to leave empty for now.

---

2. The contract: services.yaml

The UI's form generator and response renderer are dispatched off services.yaml. The agent should treat this file as the contract — everything in the UI is parameterized by it.

2.1 Field-type vocabulary

The form generator must support exactly these field types. No others. New input shapes added to a service must reduce to these:

type:	Renders as	Notes
text	<sl-input> single-line	for short strings
textarea	<sl-textarea> multi-line, autosize	prompts, lyrics
number	<sl-input type=number>	for integer/float without bounded range
slider	<sl-range> + numeric readout	requires min, max, optional step
select	<sl-select>	static options: OR remote source_url:
bool	<sl-switch>
file	<sl-input type=file> + drag-drop zone	requires accepted_types:
json	<sl-textarea> with monospace + JSON parse on submit	for arrays-of-floats, refs, etc.

Field attributes (per service entry):

• name (required)
• type (required)
• label (defaults to titlecased name)
• required (default false)
• optional: true (sugar for required: false)
• default
• min / max / step (for number / slider)
• options: [...] (for select with static options)
• source_url: ... + source_jsonpath: ... (for select with dynamic options — the UI fetches once per session, caches)
• description (rendered as Shoelace help-text)
• max_length
• accepted_types: [...] (for file)

If a service's Pydantic model needs something this vocabulary can't express, the catalog is wrong, not the vocabulary. Do not extend the vocabulary — fix the catalog or fix the service.

2.2 Response-renderer vocabulary

response.type:	Renders as
audio	inline <audio controls> + waveform thumbnail + download
image	inline <img> + click-to-zoom + download
video	inline <video controls> + download
text	monospace block + copy button
json	collapsible JSON tree
file	filename + size + download button

response.mime may be static OR derived from a request field via mime_from_field: (e.g. Kokoro's response_format field determines whether the output is audio/wav, audio/mpeg, etc.).

2.3 License warnings

Some services have license_warning: (Voxtral CC BY-NC, SAO Stability Community, Fish-S2 research-only). The UI must surface these warnings clearly when those services are selected, and again on the asset detail view. The user is responsible for not shipping restricted assets to commercial products; the UI's job is to make it impossible to do so unconsciously.

---

3. Tech constraints

• Backend: FastAPI, Python 3.11+. Not Flask, not Django.
• Frontend: Server-rendered Jinja2 templates + HTMX swaps. No SPA framework. No build step. No npm. If a feature can't be built with HTMX swap targets and Shoelace components, redesign the feature.
• Components: Shoelace web components. Use them; don't re-style them into oblivion. The point is to inherit a coherent look without designing one.
• Persistence: SQLite via SQLAlchemy. WAL mode. Single file. v3 may upgrade to Postgres; the schema should not depend on SQLite-specific behavior (no WITHOUT ROWID, no STRICT table options that diverge from Postgres types).
• Output storage: filesystem under outputs/. Content-addressed per §1.2. No object storage in v1.
• Process model: single Python process serves both API and UI in v1. v2 may split into web + worker; the Job model already permits this.
• Long jobs: HTMX polling against /ui/jobs/<id> returns the current state and renders the asset when done. No WebSockets in v1. SSE acceptable if it simplifies a specific surface, but default to polling.

---

4. Required UI surfaces

These are the surfaces, not the layouts. The agent decides the IA — whether they're separate pages, a workbench, a tabbed panel, etc. But the application must contain at least these:

4.1 Service Picker

Some way to select which service to use. The catalog has 13 entries across 4 categories (tts, asr, sfx, music); ace-step is one service with 27 fields, kokoro is one with 4. The picker should make category-level discovery easy and surface license warnings at selection time.

4.2 Make (form + submit)

Auto-generated form per the field vocabulary in §2.1. Submit creates a Job. For services with progressive disclosure (ace-step), basic fields default-visible, advanced collapsed. Form must remember the last-used values per service in the session (so iterating on a prompt doesn't lose the seed).

4.3 Live job view

While a job is running, the UI shows progress. For services where duration is known up front (SAO, ace-step), surface elapsed / est remaining. For services where it isn't, surface elapsed only. On completion, swap in the asset preview inline.

4.4 Asset detail

A single asset's full record: the output (rendered per §2.2), all params, the curl that would reproduce it, "regenerate" (same params), "fork" (open the form pre-filled with these params for editing), "delete", retention controls. The curl form is a deliberate v1 affordance — users who want to script later get a copy-paste handle without leaving the UI.

4.5 Library

All assets, sorted recency-first. Filterable by service, by output type, by date range. This is the primary navigation surface for v2 — design v1's IA so the library can grow naturally into the front door, not get bolted on later.

4.6 Job queue (v1: history only)

Every job, ordered by queued_at. State (queued/running/done/failed), duration, link to the resulting asset (if any). v1 may render this as just a list of past jobs — but the data model already supports queued/running, so the layout should accommodate them when v2 adds a worker pool.

---

5. Open for the agent to decide

Don't pre-decide these. The agent earning their keep depends on bringing opinions on:

• IA: separate pages? unified workbench with service-switcher? dashboard-of-tiles? Each has trade-offs given the 13-service catalog and the v2 multi-user direction.
• Library navigation pattern: timeline? grid? tag cloud? tree-by-collection? The v1 library has no tags and no collections, but the design should be obviously extensible.
• Long-job surfacing: toast? inline progress card? dedicated jobs panel that's always visible? Polling cadence?
• "Make new" flow when the form is huge (ace-step): single scrolling form with section headers? wizard with steps? basic-vs-advanced toggle? presets-as-templates?
• Asset preview ergonomics: inline play/scrub for audio (waveform UI)? Modal for full-screen images? Asset-comparison view (A/B same prompt different services)?
• License-warning UX: hard block with confirm? amber banner? per-asset watermark? Where in the flow does the warning live so it's noticed but not dismissed reflexively?
• Curl-rebuilder UX: button? always-visible expandable section? hidden in a "developer" panel? Be opinionated about how visible this should be.
• Service picker affordances: search? category tabs? recently-used? favorites? Considering 13 services and growing.

---

6. Out of scope (v1)

• Auth UI — the seam is in the code; no login screen needed.
• Multi-user UI — even though the data model supports it, v1 is single-user. No "shared with you" lanes, no per-user dashboards.
• Mobile-responsive — desktop-only. v2 may revisit.
• Theming / dark mode toggle — pick one, ship it. Shoelace's default theming applies.
• Real-time collaboration — no presence, no co-editing.
• Account management, billing, quotas — not relevant.
• Deployment/CI — outside the design scope; assume a single Python process.
• The ComfyUI integration — services.yaml flags ComfyUI as status: catalog-deferred. Either link out to it externally OR design a placeholder spot for it; do not try to fit it into the form-based catalog without the workflow-template wrapper.

---

7. Deliverables

In this order, with checkpoints between:

1. Architecture sketch (one page): data flow from services.yaml → form-generator → request → Job → Asset → renderer. Identify the seams. ~30 minutes of thought, not a formal diagram.
2. IA proposal (one page or one wireframe): how the surfaces in §4 fit together. Annotated with rationale. Pause here for review before pixels.
3. HTML/Jinja mockups for the core surfaces. Real Shoelace components, real catalog entries (use kokoro, sao, and ace-step as the worked examples — they span the field-count range from 4 to 27 and cover all three response types you'll see in v1: audio, audio-with-waveform, audio-with-license-warning). No backend logic; mockups are static HTML driven by hand-edited services.yaml substitution.
4. Field-type renderer reference: for each entry in §2.1, demonstrate the rendering with a small example. This becomes the design system contract for v2 service additions.
5. Response-renderer reference: same for §2.2.
6. Open-questions log: a list of things the agent flagged as intentionally undecided + the reasoning. Useful when v2 starts.

Not in scope as deliverables: working backend code, database migrations, deployment artifacts. Those follow once the IA is agreed.

---

8. Constraints on the design agent

• Don't extend the field-type vocabulary in §2.1. If a service needs a date picker, that's because the catalog spec is wrong; fix the catalog or push back on the service.
• Don't design auth flows. The seam is in the code; the UI has no login.
• Don't add features beyond §4. If you find yourself wanting a "share asset" button or a "public gallery", that's v2.
• Don't choose a JS framework. HTMX + Shoelace is the rule. If a feature feels impossible without React, redesign the feature.
• Don't theme/restyle Shoelace into a custom design system. The whole point of pulling in a component library is to not do that. Use the defaults; pick a single accent color; move on.