vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: archive asset-engine design-brief.md

Browse Source 
Moved docs/asset-engine/design-brief.md → docs/archive/asset-engine/design-brief.md
with a 12-line archival header pointing at the live implementation
artifacts (vh/asset-engine source, stacks/asset-engine/ deploy,
CATALOG-CONTRACT.md, services.yaml).

The brief explicitly framed itself as a pre-implementation handoff
("Hand this to a design agent before any pixels"). Implementation
shipped 2026-05-12; the brief's role is past. Kept for the design
rationale it carries (why Asset is first-class, why v1 is synchronous,
v2/v3 seam reasoning) — future contributors benefit from finding it
when wondering "why is it this way."

Surfaced by /tend-docs audit 2026-05-14.
This commit is contained in:
vh
2026-05-14 15:33:15 -07:00
parent 391244b18b
commit 54d1e7137e
1 changed files with 10 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/asset-engine/design-brief.md
-401
View File
@@ -1,401 +0,0 @@
# Asset Generation Engine — Design Brief
For an internal UI that fronts the inference services on `irv-ml1`
catalogued in [`services.yaml`](services.yaml). Hand this to a design
agent before any pixels.
> **What this is.** A FastAPI + HTMX + Shoelace web app that replaces
> `curl` for routine use of ~12 self-hosted inference services
> (TTS, ASR, SFX, music). Single-tenant in v1; the data model and
> code seams are designed so v2/v3 can grow into multi-user with
> auth, tagging, projects, and async job queues without
> architectural churn.
>
> **Not** a public product. **Not** multi-tenant on day 1. **Not** a
> generic "AI playground" — it's specifically a tool for *generating
> reusable assets* (voiceover takes, hold tones, music beds, SFX,
> transcripts) that get pulled into other projects.
---
## 1. Locked architectural decisions
These shape the data model and the code seams. They are not
negotiable in v1, because retrofitting them after the database has
real rows in it is expensive.
### 1.1 Asset is a first-class entity
Every successful generation writes one `Asset` row:
```
Asset
id uuid primary key
service_id text -- 'kokoro', 'sao', 'ace-step', ...
service_version int -- catalog `version:` at generation time
model_id text -- 'hexgrad/Kokoro-82M'
model_revision text? -- HF SHA when known, else null
image_ref text -- container image at generation time (tag or digest)
params json -- the request body that produced this asset
output_hash text -- sha256 of the output bytes
output_path text -- 'outputs/<sha256>.<ext>'
output_mime text -- 'audio/wav', 'image/png', 'text/plain'
duration_s float? -- for audio/video
byte_size int
created_by int -- FK -> User.id (always 1 in v1)
created_at datetime
tags json -- list[str]; empty in v1, surfaced in v2
collection_id uuid? -- FK -> Collection; null in v1
retention text -- 'ephemeral' | 'keep' | 'pinned'
source_asset_id uuid? -- FK -> Asset; non-null when this is a fork/regenerate
```
The presence of this row from day 1 is what enables the v2/v3 stories
(library, search, dedupe, regenerate, share, export). Backfilling
this from server logs after the fact is grim — don't.
### 1.2 Content-addressed output storage
Outputs live at `outputs/<sha256-of-bytes>.<ext>`. The asset row
points at the hash. This gets us, for free:
- **Dedupe**: same prompt + seed + model produces the same bytes →
same hash → one file on disk, multiple Asset rows referencing it.
- **Garbage collection**: `delete from blobs where ref_count = 0`.
- **Verification**: an Asset row whose `output_hash` doesn't match
the bytes on disk is corrupt; we know to re-run.
- **Migration safety**: blob layout is stable; you can move
`outputs/` to a different filesystem without touching the DB.
Path conventions like `outputs/2026/05/10/kokoro_<timestamp>.wav` rot
within months. Don't.
### 1.3 Reproducibility is a hard requirement
Every Asset must be regeneratable from its `params` row. This means:
- **Explicit seed** for any service that supports sampling. (SAO and
ace-step do; everything else is deterministic-without-sampling and
requires `image_ref` pinning instead.)
- **`model_revision`** when the upstream exposes a SHA. When it
doesn't, `image_ref` becomes the reproducibility anchor.
- **Service schema versioning** — when `services.yaml` changes a
field's shape, bump that service's `version:`. Old Asset rows
remember which version they were generated against.
If a service can't produce reproducible output, that's a bug in the
*service* to fix before it goes in the catalog — not a UI concern.
### 1.4 Job table from day 1
Every Asset is produced by a Job:
```
Job
id uuid primary key
asset_id uuid? -- FK -> Asset; null until done
service_id text
params json
state text -- 'queued' | 'running' | 'done' | 'failed'
error text?
worker text? -- which process picked it up; null in v1
queued_at datetime
started_at datetime?
finished_at datetime?
```
In v1 this is synchronous: insert → run inline → update → return.
In v2 it becomes a worker pool. In v3 it gains priority + per-user
quota. **The query surface stays the same throughout** — UI
templates that render "running jobs" or "job history" don't change.
### 1.5 Auth as a no-op dependency, not absent
Every request goes through a `current_user()` dependency:
```python
def current_user() -> User:
# v1: hardcoded; v2: header-injection from upstream proxy;
# v3: OIDC.
return User(id=1, name="me")
```
Downstream code uses `request.user.id` from day 1. v2's swap is a
dependency-injection change, not a refactor. **A code base that
assumes "no users" cannot be retrofitted with auth without
touching everything.**
### 1.6 API surface ≠ UI surface
Every UI action should be powered by a JSON API route. The HTMX
templates call those API routes via `hx-post` / `hx-get` and render
the JSON into HTML partials, OR the API routes serve dual content
(JSON + partial) by route prefix:
- `POST /api/v1/jobs` — JSON body in, JSON body out
- `POST /ui/jobs` — same handler called server-side, returns HTMX partial
Pick the **separate-route** pattern, not content negotiation by
`Accept:` header. It's noisier but clearer; debugging API issues
in v3 with content negotiation in the way is hell.
### 1.7 Service catalog is git-versioned and additive
[`services.yaml`](services.yaml) is the contract. Adding a service is
a YAML edit + server reload — no template authoring, no Python
changes. Each service entry carries `version:`. Schema changes bump
the version. Old assets remember which version they were generated
against (via `Asset.service_version`).
### 1.8 Tags + collections plumbed in the model from day 1
`Asset.tags: list[str]`, `Asset.collection_id: uuid?`. **No UI
required in v1** — but the columns must exist. Adding the columns
later is a migration touching every existing row; harmless to leave
empty for now.
---
## 2. The contract: `services.yaml`
The UI's form generator and response renderer are dispatched off
[`services.yaml`](services.yaml). The agent should treat this file
as the contract — **everything in the UI is parameterized by it**.
### 2.1 Field-type vocabulary
The form generator must support exactly these field types. No
others. New input shapes added to a service must reduce to these:
| `type:` | Renders as | Notes |
|-------------|-------------------------------------------|-------|
| `text` | `<sl-input>` single-line | for short strings |
| `textarea` | `<sl-textarea>` multi-line, autosize | prompts, lyrics |
| `number` | `<sl-input type=number>` | for integer/float without bounded range |
| `slider` | `<sl-range>` + numeric readout | requires `min`, `max`, optional `step` |
| `select` | `<sl-select>` | static `options:` OR remote `source_url:` |
| `bool` | `<sl-switch>` | |
| `file` | `<sl-input type=file>` + drag-drop zone | requires `accepted_types:` |
| `json` | `<sl-textarea>` with monospace + JSON parse on submit | for arrays-of-floats, refs, etc. |
Field attributes (per service entry):
- `name` (required)
- `type` (required)
- `label` (defaults to titlecased name)
- `required` (default false)
- `optional: true` (sugar for `required: false`)
- `default`
- `min` / `max` / `step` (for `number` / `slider`)
- `options: [...]` (for `select` with static options)
- `source_url: ...` + `source_jsonpath: ...` (for `select` with
dynamic options — the UI fetches once per session, caches)
- `description` (rendered as Shoelace help-text)
- `max_length`
- `accepted_types: [...]` (for `file`)
If a service's Pydantic model needs something this vocabulary can't
express, the catalog is wrong, not the vocabulary. **Do not extend
the vocabulary** — fix the catalog or fix the service.
### 2.2 Response-renderer vocabulary
| `response.type:` | Renders as |
|------------------|-----------------------------------------------|
| `audio` | inline `<audio controls>` + waveform thumbnail + download |
| `image` | inline `<img>` + click-to-zoom + download |
| `video` | inline `<video controls>` + download |
| `text` | monospace block + copy button |
| `json` | collapsible JSON tree |
| `file` | filename + size + download button |
`response.mime` may be static OR derived from a request field via
`mime_from_field:` (e.g. Kokoro's `response_format` field
determines whether the output is audio/wav, audio/mpeg, etc.).
### 2.3 License warnings
Some services have `license_warning:` (Voxtral CC BY-NC, SAO
Stability Community, Fish-S2 research-only). The UI must surface
these warnings clearly when those services are selected, and again
on the asset detail view. The user is responsible for not shipping
restricted assets to commercial products; the UI's job is to make
it impossible to do so unconsciously.
---
## 3. Tech constraints
- **Backend**: FastAPI, Python 3.11+. Not Flask, not Django.
- **Frontend**: Server-rendered Jinja2 templates + HTMX swaps.
No SPA framework. No build step. No npm. If a feature can't be
built with HTMX swap targets and Shoelace components, redesign
the feature.
- **Components**: Shoelace web components. Use them; don't
re-style them into oblivion. The point is to inherit a coherent
look without designing one.
- **Persistence**: SQLite via SQLAlchemy. WAL mode. Single file.
v3 may upgrade to Postgres; the schema should not depend on
SQLite-specific behavior (no `WITHOUT ROWID`, no `STRICT` table
options that diverge from Postgres types).
- **Output storage**: filesystem under `outputs/`. Content-addressed
per §1.2. No object storage in v1.
- **Process model**: single Python process serves both API and UI
in v1. v2 may split into web + worker; the Job model already
permits this.
- **Long jobs**: HTMX polling against `/ui/jobs/<id>` returns the
current state and renders the asset when done. No WebSockets in
v1. SSE acceptable if it simplifies a specific surface, but
default to polling.
---
## 4. Required UI surfaces
These are the *surfaces*, not the *layouts*. The agent decides the
IA — whether they're separate pages, a workbench, a tabbed panel,
etc. But the application must contain at least these:
### 4.1 Service Picker
Some way to select which service to use. The catalog has 13 entries
across 4 categories (`tts`, `asr`, `sfx`, `music`); ace-step is one
service with 27 fields, kokoro is one with 4. The picker should
make category-level discovery easy and surface license warnings at
selection time.
### 4.2 Make (form + submit)
Auto-generated form per the field vocabulary in §2.1. Submit creates
a Job. For services with progressive disclosure (ace-step), basic
fields default-visible, advanced collapsed. Form must remember the
last-used values per service in the session (so iterating on a
prompt doesn't lose the seed).
### 4.3 Live job view
While a job is running, the UI shows progress. For services where
duration is known up front (SAO, ace-step), surface elapsed / est
remaining. For services where it isn't, surface elapsed only.
On completion, swap in the asset preview inline.
### 4.4 Asset detail
A single asset's full record: the output (rendered per §2.2), all
params, the curl that would reproduce it, "regenerate" (same
params), "fork" (open the form pre-filled with these params for
editing), "delete", retention controls. The curl form is a
deliberate v1 affordance — users who want to script later get a
copy-paste handle without leaving the UI.
### 4.5 Library
All assets, sorted recency-first. Filterable by service, by output
type, by date range. **This is the primary navigation surface for
v2** — design v1's IA so the library can grow naturally into the
front door, not get bolted on later.
### 4.6 Job queue (v1: history only)
Every job, ordered by `queued_at`. State (queued/running/done/failed),
duration, link to the resulting asset (if any). v1 may render this
as just a list of past jobs — but the data model already supports
queued/running, so the layout should accommodate them when v2 adds a
worker pool.
---
## 5. Open for the agent to decide
Don't pre-decide these. The agent earning their keep depends on
bringing opinions on:
- **IA**: separate pages? unified workbench with service-switcher?
dashboard-of-tiles? Each has trade-offs given the 13-service
catalog and the v2 multi-user direction.
- **Library navigation pattern**: timeline? grid? tag cloud?
tree-by-collection? The v1 library has no tags and no
collections, but the design should be obviously extensible.
- **Long-job surfacing**: toast? inline progress card? dedicated
jobs panel that's always visible? Polling cadence?
- **"Make new" flow when the form is huge** (ace-step): single
scrolling form with section headers? wizard with steps?
basic-vs-advanced toggle? presets-as-templates?
- **Asset preview ergonomics**: inline play/scrub for audio
(waveform UI)? Modal for full-screen images? Asset-comparison
view (A/B same prompt different services)?
- **License-warning UX**: hard block with confirm? amber banner?
per-asset watermark? Where in the flow does the warning live so
it's noticed but not dismissed reflexively?
- **Curl-rebuilder UX**: button? always-visible expandable section?
hidden in a "developer" panel? Be opinionated about how visible
this should be.
- **Service picker affordances**: search? category tabs?
recently-used? favorites? Considering 13 services and growing.
---
## 6. Out of scope (v1)
- **Auth UI** — the seam is in the code; no login screen needed.
- **Multi-user UI** — even though the data model supports it, v1
is single-user. No "shared with you" lanes, no per-user dashboards.
- **Mobile-responsive** — desktop-only. v2 may revisit.
- **Theming / dark mode toggle** — pick one, ship it. Shoelace's
default theming applies.
- **Real-time collaboration** — no presence, no co-editing.
- **Account management, billing, quotas** — not relevant.
- **Deployment/CI** — outside the design scope; assume a single
Python process.
- **The ComfyUI integration** — `services.yaml` flags ComfyUI as
`status: catalog-deferred`. Either link out to it externally OR
design a placeholder spot for it; **do not try to fit it into the
form-based catalog without the workflow-template wrapper**.
---
## 7. Deliverables
In this order, with checkpoints between:
1. **Architecture sketch** (one page): data flow from
`services.yaml` → form-generator → request → Job → Asset →
renderer. Identify the seams. ~30 minutes of thought, not a
formal diagram.
2. **IA proposal** (one page or one wireframe): how the surfaces in
§4 fit together. Annotated with rationale. **Pause here for
review** before pixels.
3. **HTML/Jinja mockups** for the core surfaces. Real Shoelace
components, real catalog entries (use `kokoro`, `sao`, and
`ace-step` as the worked examples — they span the field-count
range from 4 to 27 and cover all three response types you'll
see in v1: audio, audio-with-waveform, audio-with-license-warning).
No backend logic; mockups are static HTML driven by hand-edited
`services.yaml` substitution.
4. **Field-type renderer reference**: for each entry in §2.1,
demonstrate the rendering with a small example. This becomes
the design system contract for v2 service additions.
5. **Response-renderer reference**: same for §2.2.
6. **Open-questions log**: a list of things the agent flagged as
intentionally undecided + the reasoning. Useful when v2 starts.
**Not in scope as deliverables**: working backend code, database
migrations, deployment artifacts. Those follow once the IA is
agreed.
---
## 8. Constraints on the design agent
- **Don't extend the field-type vocabulary in §2.1**. If a service
needs a date picker, that's because the catalog spec is wrong;
fix the catalog or push back on the service.
- **Don't design auth flows**. The seam is in the code; the UI
has no login.
- **Don't add features beyond §4**. If you find yourself wanting a
"share asset" button or a "public gallery", that's v2.
- **Don't choose a JS framework**. HTMX + Shoelace is the rule. If
a feature feels impossible without React, redesign the feature.
- **Don't theme/restyle Shoelace into a custom design system**. The
whole point of pulling in a component library is to *not* do that.
Use the defaults; pick a single accent color; move on.