Staged 28 single-speaker dia voices + glados into /worktank/fish-s2/references/
(internal research use). Discovered the path-form references shape 500s on this
build; reference_id (by name) is the working voice path (verified live). So:
reference_id -> select 'Voice' with the 29 staged names (default Emily, female);
references demoted to advanced inline-base64 custom clones with the path->500
caveat documented. vram 10->24 (measured ~25GB in use); version 2->3.
Fish-S2 rendered as an essentially blank form — only text + a references JSON
blob — despite being the fleet's richest-control engine. Expose the real
fish-speech ServeTTSRequest levers: temperature/top_p/repetition_penalty,
latency (normal|balanced), seed, format (wav/mp3/opus), + advanced cloning
(references/reference_id) and chunking (max_new_tokens/chunk_length/normalize/
use_memory_cache). Defaults from upstream schema.py, verified live 2026-06-01
(no /openapi.json; Kui server). Sections basic/sampling/advanced.
Also corrected: seedable false->true (/tts has seed); cold_start 8s->240s
(measured compile warmup); vram 4->10GB (~9GB BF16 weights); dropped the
misleading '~150ms TTFB' for honest ~realtime throughput. version 1->2.
Chatterbox was producing poor output because the catalog pointed at the thin
OpenAI /v1/audio/speech endpoint, which exposes none of Resemble's emotion/
pacing knobs — and the devnen server's shipped default exaggeration is 1.3
(tuned for its theatrical demo presets), which over-acts.
Re-point to the wrapper's richer /tts and expose the real control surface
(exaggeration, cfg_weight, temperature, speed_factor, seed, voice_mode),
mirroring the sibling dia stack (same devnen author). Defaults sourced live
2026-06-01: exaggeration + cfg_weight = 0.5 (Resemble README 'works well for
most prompts'), temperature 0.8 / speed 1.0 / seed 0 (server generation_
defaults). The shipped 1.3 exaggeration is deliberately NOT adopted.
Voices: expose the 28 built-in predefined voices via /get_predefined_voices
(default Emily.wav, the server default_voice_id) + clone via /get_reference_
files — replacing the wrong 'OpenAI aliases only' claim. Corrected seedable:
false -> true (/tts has seed) and image_tag_mutable -> true (:latest). Bumped
service version 1 -> 2 (breaking field-shape change); status down -> ready
(live + healthy). catalog_version unchanged (no new field types).
Tear down the parked CSM stack (status: down, never successfully built).
Bring-up attempts failed at the image build: upstream
phildougherty/sesame_csm_openai pins no huggingface_hub version, which now
resolves to 1.17.0 where the `huggingface-cli` the Dockerfile relies on has
been removed (replaced by `hf`). Building would require vendoring + patching
the upstream Dockerfile.
Deep-research verdict (primary + community sourced) confirmed it isn't worth
that: the acclaimed Maya/Miles demo runs a fine-tuned, larger CSM variant
Sesame never open-sourced; the open csm-1b is the un-fine-tuned 1B base
(only the smallest of 1B/3B/8B shipped, no newer checkpoint as of mid-2026).
Ships no usable voices, can't generate text, English-only, can't stream
real-time out of the box; absent from current TTS leaderboards and dominated
by Kokoro/Dia2/Fish-S2/IndexTTS for narration.
Removes: stacks/csm/, playbooks/deploy-csm.yaml, the csm catalog service +
reproducibility_audit entries. Host state (compose dir, /worktank/csm) torn
down on irv-ml1; no container/image existed.
Reported: dia2-1b 404 'Reference audio file not found: undefined' when
accepting defaults — voice_mode=clone with no clone_reference_filename made
the UI submit the literal 'undefined'. Fix: voice_mode now defaults to clone
AND clone_reference_filename defaults to Abigail.wav (a staged voice), so the
out-of-box request is a valid, stable voice. Reproduced the 404 and verified
the Abigail.wav path returns 200. Folded into v2 (not yet consumed downstream).
Pre-fill the bare input textareas with format-demonstrating samples
(dia2-2b/dia2-1b: [S1]/[S2] dialogue + a nonverbal; csm: conversational;
zonos: expressive multilingual) and give csm.topk a standard default (50).
Addresses asset-engine-dev's best-practice-defaults request so the picker
forms pre-fill usefully instead of empty. No version bumps — these entries
ship fresh in this batch (no prior consumer-pinned shape).
Replace the single dia entry (legacy Dia 1.6B, retired) with two fixed-model
Dia2 entries (dia2-2b :8200, dia2-1b :8202), status ready (both exercised),
image local/dia:v2. Matching reproducibility_audit rows. catalog_version
unchanged (add/remove services = no vocab change).
Also fix a port collision I introduced earlier: the zonos-api adapter and
csm both claimed 8201 — move zonos-api to 8203 (catalog endpoint + voices
source_url, zonos .env.example, README).
NOTE FOR CONSUMERS: removing the dia id is a breaking catalog change for
asset_engine (it vendored dia in v0.1.4) — re-vendor + drop the dia tile,
add the two dia2 tiles.
zonos: new tts entry routing to the REST adapter on 8201, JSON-envelope
response with reproducibility.seed_field=seed (seedable+deterministic),
lifecycle block, section groups. Matching reproducibility_audit row.
dia: voice select had a default (S1) but no options/source_url -> empty
picker; add options [S1, S2, dialogue] (per asset-engine-dev), bump dia
to version 2. Closed select drops clone-by-filename free-text; flagged
inline for a future source_url-backed picker. catalog_version unchanged
(add-service + field-options = no vocab change).
Resolves the stale-schema gap asset-engine-dev flagged (the published
schema rejected the lifecycle field 12/14 live services already use) and
adds reproducibility.seed_field so a seedable engine can declare which
response key carries the seed used. Authoritative regen path remains
dump_schema.py against asset_engine/catalog.py; reconcile there.
dia (:8200) — OpenAI-compat /v1/audio/speech, seedable (not byte-exact),
Apache-2.0 weights. Clean catalog fit; flip to ready after first exercise.
csm (:8201) — OpenAI-compat, but NO seed + temperature-sampled =
non-reproducible (contract's fix-before-adding case), catalogued by
operator direction with a reproducibility caveat + gated-license warning;
belongs at experimental once running.
Fields read from each wrapper's API docs (2026-05-31), to confirm against
live OpenAPI/Pydantic at deploy. No catalog_version bump (add-service =
no bump). NOTE: services.schema.json is stale (pre-existing — 13 errors;
live catalog uses lifecycle, schema predates it); regen via dump_schema.py.
Moved docs/asset-engine/design-brief.md → docs/archive/asset-engine/design-brief.md
with a 12-line archival header pointing at the live implementation
artifacts (vh/asset-engine source, stacks/asset-engine/ deploy,
CATALOG-CONTRACT.md, services.yaml).
The brief explicitly framed itself as a pre-implementation handoff
("Hand this to a design agent before any pixels"). Implementation
shipped 2026-05-12; the brief's role is past. Kept for the design
rationale it carries (why Asset is first-class, why v1 is synchronous,
v2/v3 seam reasoning) — future contributors benefit from finding it
when wondering "why is it this way."
Surfaced by /tend-docs audit 2026-05-14.
Two related changes shipped together. The stack rename is independent
but adding `vllm-reward` to the existing `vllm-qwen3` would have made
that name actively misleading.
**Rename:** `stacks/vllm-qwen3/ → stacks/vllm/`. Updated all in-repo
references (README.md root, servers/ana-ml2/, stacks/llama-swap/,
configs/restic/ana-ml2/, docs/runbooks/disaster-recovery.md). Two
intentional history mentions retained (servers/ana-ml2 + stacks/vllm
README).
**Add `vllm-reward` service:** serves Skywork-Reward-V2-Llama-3.1-8B-AWQ
on port 8003. The AWQ output is a locally-quantized model (not from HF),
so bind-mounts `/tank/aimodels/llm:/local-models:ro` rather than the
shared HF cache. Model config.json declares LlamaForSequenceClassification
which vLLM's pooling runner picks up automatically — produces a single
reward score per input via /classify.
**Flag note:** the user's spec listed `--task classify`, but vLLM 0.19.1
deprecated --task in favor of --runner pooling (model architecture in
config.json drives the classification head). Compose uses --runner
pooling with a comment explaining the substitution.
**GPU memory:** no rebalance needed — production had already tuned
EMBED/RERANK down from 0.40 to 0.20 each (canonical .env.example now
matches reality). Adding REWARD at 0.30 totals 0.70, leaving ~14 GB
headroom on the 48 GB Ada.
**Server-side:** brought existing vllm-qwen3 down, mv'd
/opt/docker/compose/vllm-qwen3 → /opt/docker/compose/vllm, appended
REWARD_* lines to existing .env (preserving API_KEY/HF_TOKEN), deployed
new compose via scripts/deploy-stack.sh, brought all 3 services up.
**Smoke tests:**
- /health on 8001/8002/8003 → 200
- /v1/models on 8003 → lists Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ
with max_model_len 16384
- /classify with a sample conversation → returns LABEL_0 with prob 0.9999
(single-output regression-style reward score, expected shape for a
reward model)
Booted each service on irv-ml1 to capture the wrapper's actual voice
exposure, then took them back down. Initial state restored.
**Voxtral (v1 → v2)** — 20 native presets from live /v1/audio/voices:
neutral_{female,male}, casual_{female,male}, cheerful_female, plus
8 language-code variants ({ar,de,es,fr,hi,it,nl,pt}_{female,male}).
vLLM-Omni does NOT translate OpenAI aliases; `alloy` would 400. Default
flipped to `neutral_female` (matches Mistral docs).
**Qwen3-TTS (v1 → v2)** — 15 voices from live /v1/voices, NOT the 17
the upstream blog cites: 9 Qwen presets (vivian/serena/uncle_fu/ryan/
aiden/ono_anna/sohee/eric/dylan) + 6 OpenAI-compat aliases (alloy/
echo/fable/nova/onyx/shimmer). Default `vivian` matches the wrapper's
OpenAPI default. Catalog previously stated clone-only — wrong; the
wrapper does ship presets, the upstream blog list just doesn't match
the deployed wrapper. Cloning still works alongside via clone:<name>.
**Kyutai-TTS (v1 → v2)** — NillPointer wrapper does NOT expose any voice
listing endpoint (/v1/audio/voices 404; only /health + POST /v1/audio/
speech are wired). Voices are filesystem-discovered. Catalog now drops
the broken source_url, switches voice field to free-text with default
`unmute-prod-website/default_voice.wav` (upstream's named default).
Description lists the on-disk categories with counts + license posture
(vctk = CC BY 4.0 commercial-safe; expresso = CC BY-NC research-only).
Six default + description tightenings from upstream-source research:
- **Kokoro voice**: af_bella → af_heart. Per upstream VOICES.md, af_heart
is the only A-rated voice; also the kokoro-fastapi container's own
default. Applied to both kokoro (v2 → v3) and kokoro-captioned (v1 → v2).
- **SAO negative_prompt**: "Low quality." → "low quality, average quality".
Per diffusers official docs Tips section — the lowercase comma-separated
shape is the explicit recommendation; the period form was a code-example
string, never a documented default. SAO bumped v1 → v2.
- **SAO prompt description**: added the model card's canonical examples
("128 BPM tech house drum loop", "the sound of a hammer hitting a wooden
surface") plus the Tips advice on descriptive prompts.
- **VibeVoice voice**: en-Carter_man → en-Alice_woman. Per upstream model
card, Alice is the documented default. Description now flags the
Alice-injects-BGM-for-intros foot-gun + the `_bgm`-suffix meaning.
VibeVoice bumped v2 → v3.
- **Fish-s2 text description**: added multi-tag placeholder example
combining emotion + physical tags, per upstream best-practice docs
(physical tags "feel flat without emotional context").
- **ace-step prompt description**: added the upstream Gradio UI's
pre-filled tag string as the canonical prompt-shape example.
Surfaced separately to lkraven (NOT applied here, need decisions):
- Voxtral voice "alloy" may need to become a native preset like
"neutral_female" — depends on whether vLLM-Omni translates OpenAI
aliases.
- Qwen3-TTS — research found 17 presets (Cherry, Ethan, ...) that the
catalog currently says don't exist. Catalog says clone-only; needs
wrapper-level verification before adding.
- Kyutai-TTS — service down, can't probe /v1/audio/voices. Default
remains undefaulted.
Three changes prepping infra for asset_engine's orchestration feature
(SSH-driven bring-up / bring-down of irv-ml1 inference services with
per-device VRAM gating, contract in vh/asset-engine commit 5a36f8c):
1. asset-engine compose + .env.example + playbook gain a read-only
bind-mount for /app/runtime/ssh — the dedicated ed25519 keypair
(generated on ana-docker, not in the repo) plus a pinned known_hosts
for irv-ml1's host fingerprint. Env vars SSH_KEY_PATH and
SSH_KNOWN_HOSTS are exposed for the app to consume.
2. docs/asset-engine/services.yaml gains a `lifecycle: { stack, vram_gb,
gpu_device_id }` block on each of 12 orchestratable irv-ml1 services
(kokoro, chatterbox, index-tts, qwen3-tts, cosyvoice, fish-s2,
kyutai-tts, vibevoice, voxtral, parakeet, stable-audio-open, ace-step).
VRAM numbers are estimates from model footprint at fp16 — tune from
real nvidia-smi measurements once the gate is live. comfyui and
kokoro-captioned are deliberately excluded (variable-VRAM and
shared-container respectively).
3. servers/irv-ml1/README.md docker-stacks table now lists all 13
inference stacks (was only dockge + agents + comfyui) with port +
GPU pinning columns.
Pubkey deployed to ~lkraven/.ssh/authorized_keys on irv-ml1;
end-to-end SSH from ana-docker → irv-ml1 verified with strict
host-key checking.
asset-engine's seed_surface contract needs reproducibility.seed_field to name
which CatalogField carries the seed so the UI can render a Roll button and the
server can fill empty seeds before persistence. SAO's seed field is type=number,
satisfying CatalogService._validate_seed_field.
ace-step is intentionally not declared here — actual_seeds is type=json (a list)
with an upstream reproducibility gap; the planned fix surfaces resolved seeds via
response header, pending a separate contract.
Per althing thread 01KRCNSF0V5NDCKB34H663MXHS — the catalog declared
14 services but 6 of them aren't running on irv-ml1 (chatterbox,
index-tts, qwen3-tts, cosyvoice, voxtral, kyutai-tts; missing from
docker ps entirely). Without action, the asset-engine UI would
declare them as available and consumers would hit unreachable
endpoints.
asset-engine consumer chose option (1) of three I sketched: extend
StatusT with `down` and treat it identically to `catalog-deferred`
in the picker (greyed, non-clickable). Lightweight, declarative, no
runtime health-check machinery, easy to revert when services
return.
Changes:
- StatusT enum (in asset_engine/catalog.py — committed there
separately) extended from
Literal["ready", "catalog-deferred", "experimental"]
to
Literal["ready", "catalog-deferred", "experimental", "down"]
- 6 services flipped to status: down.
- CATALOG-CONTRACT.md: replaced the bare-enum status row with a
four-row sub-table that names each value's meaning AND its picker
behavior. `down` and `catalog-deferred` get the same UI treatment
but the tooltip text differentiates ("Catalog-deferred" vs
"Service down — temporarily unreachable on irv-ml1") so the
semantic distinction (design state vs fleet-ops state) is
preserved.
- CATALOG-CONTRACT.md versioning policy table: new row codifying
"extending an existing enum (StatusT, FieldTypeT, ResponseTypeT,
CategoryT) with a non-conflicting value, with the consumer
updated in the same coordinated change" → no catalog_version
bump. Explicit rule for future enum extensions.
- JSON Schema regenerated.
catalog_version stays at 1.
Operational note (not catalog-side): the down services likely got
reaped 13+ days ago per the docker timestamps when other unrelated
work was done on irv-ml1. Bringing them back is a deploy task
outside this commit's scope. Flip status: down → ready in this file
once each one's confirmed running.
Sweep round caught vibevoice catalog drift in three dimensions; all
verified against the live OpenAPI + endpoint exercise, NOT against
documentation (which is what produced the bad values originally).
model:
was: options=[vibevoice], default=vibevoice
now: options=[tts-1, tts-1-hd, vibevoice], default=tts-1
why: the wrapper accepts all three (OpenAI-compat aliases all map
to VibeVoice internally per upstream README); wire default is tts-1
per /openapi.json. Catalog over-constrained users to a single value.
voice:
was: default=Carter; description listed [Carter, Davis, Emma, Frank,
Grace, Mike, Samuel] as built-ins
now: default=en-Carter_man; options enumerated:
OpenAI: alloy, echo, fable, onyx, nova, shimmer
VibeVoice: en-Alice_woman, en-Carter_man, en-Frank_man,
en-Mary_woman_bgm, en-Maya_woman, in-Samuel_man,
zh-Anchen_man_bgm, zh-Bowen_man, zh-Xinran_woman
why: discovered by hitting the endpoint with the catalog's claimed
"Carter" — wrapper returned 400 with the actual valid list inline
in the error body. The previous catalog values were fabrications,
not derived from any real source.
response_format:
was: options=[wav, mp3]
now: options=[wav, mp3, opus, flac, pcm]; default mp3 (was wav)
why: probed all 7 plausible formats; 5 return audio (200), aac and
m4a return 500. Catalog was over-restrictive; an earlier sweep
draft over-claimed [wav, mp3, opus, aac, flac, pcm, m4a] from
documentation that I refused to apply unverified. Now matches the
empirically-confirmed set.
Bumped vibevoice version 1 -> 2. catalog_version stays at 1.
Lesson reinforced: the only source-of-truth for catalog values is
the live wire. /openapi.json doesn't enumerate enums (returns bare
"string"); error responses from the endpoint with bad inputs are
the most reliable enum-discovery mechanism.
asset_engine consumer audited the entire ace-step entry's defaults
and slider ranges against acestep/ui/components.py (althing thread
01KRCN0SHP9YJGQD58EE95DC5P). The catalog had been authored from
documentation rather than from source; ten defaults were wrong and
several slider ranges were either too narrow or impractically wide.
Defaults changed (catalog -> upstream-authoritative):
infer_step 20 -> 60
guidance_scale 7.5 -> 15.0
cfg_type cfg -> apg
omega_scale 0.5 -> 10.0
guidance_interval 0.0 -> 0.5
guidance_interval_decay 1.0 -> 0.0
min_guidance_scale 1.0 -> 3.0
use_erg_tag false -> true
use_erg_diffusion false -> true
actual_seeds [42] -> [] (random per call)
Slider ranges adopted from upstream where reasonable; bounded
locally where upstream's range is so wide it's unusable as a UI
slider:
guidance_scale [1.0, 15.0] -> [0.0, 30.0] (upstream)
guidance_scale_text [0.0, 15.0] -> [0.0, 10.0] (upstream)
guidance_scale_lyric [0.0, 15.0] -> [0.0, 10.0] (upstream)
lora_weight [0.0, 2.0] -> [-3.0, 3.0] (upstream)
audio_duration [5.0, 600.0] -> [5.0, 240.0] (upstream max)
omega_scale [0.0, 1.0] -> [-10.0, 30.0] (UI bound; upstream is [-100, 100])
min_guidance_scale [0.0, 10.0] -> [0.0, 20.0] (UI bound; upstream is [0, 200])
Verified empty-string actual_seeds path against the live pipeline
source: pipeline_ace_step.py:set_seeds() falls through to
torch.randint when manual_seeds is "" (string, no comma, not all
digits). Smoked end-to-end: HTTP 200 in 11s, real WAV bytes back.
Reproducibility gap honestly documented in the entry's
reproducibility.notes and the actual_seeds field description: with
the new default `actual_seeds: []`, the wrapper rolls a random seed
inside the pipeline but doesn't capture or surface the chosen seed
back through the response. Default-defaulted assets cannot be
regenerated bit-exact; users requiring reproducibility must set
actual_seeds explicitly. Wrapper enhancement to surface the chosen
seed via X-Actual-Seeds header + a CatalogResponse.header_accessories
schema field is the planned fix.
ace-step bumped version 4 -> 5. catalog_version stays at 1 (no
schema changes).
Also added a "source-of-truth precedence" subsection to
CATALOG-CONTRACT.md's service-authoring notes, codifying the
read-order (Pydantic model > handler/pipeline code > Gradio UI >
README). Three ace-step bugs in three rounds (missing field, wrong
enums, stranded bytes, wrong defaults — really four) all share the
same root cause: catalog authored from doc surfaces that lie by
omission.
The pre-fix wrapper at stacks/ace-step/infer-api.py returned a JSON
{output_path: "..."} reference to a file written inside the
container at /app/outputs/. That path was unreachable from outside
the container — every consumer got 134 bytes of JSON-pretending-to-
be-WAV instead of audio. Surfaced by the asset_engine consumer's
end-to-end smoke (althing thread 01KRCJF7NGMXYE9F62Q1A6KFD4 msg 5);
my own earlier smoke missed it because I checked HTTP=200 and stopped
reading instead of inspecting the response body.
Wrapper now reads back the file the pipeline writes and streams the
bytes via fastapi.responses.Response with media_type set from the
audio_format request field (audio/wav | audio/mpeg | audio/flac).
The in-container path is exposed via X-Output-Path header for log
correlation but is no longer load-bearing.
Verified end-to-end against live ace-step on irv-ml1:
POST /generate -> HTTP 200 in 80s
content-type: audio/wav
content-length: 945226
x-output-path: /app/outputs/output_cfe87d1d....wav
$ file response.wav
RIFF (little-endian) data, WAVE audio, Microsoft PCM, 16 bit,
stereo 48000 Hz
Catalog: ace-step bumped version 3 -> 4. Dropped
response.output_field (no longer applicable). reproducibility.notes
expanded to record both the v2 18-arg-tuple fix and this v4
inline-streaming change so the history is auditable from the
catalog itself.
Stale ACEStepOutput Pydantic model left in infer-api.py for now —
unused but small; future cleanup.
Smoke testing in the asset_engine consumer surfaced an
UnboundLocalError 500 from ace-step (althing thread
01KRCJF7NGMXYE9F62Q1A6KFD4 msg 3). Root cause: this catalog had
invented enum values for scheduler_type and cfg_type that don't
exist in the upstream pipeline.
Read pipeline_ace_step.py inside the running container:
scheduler_type dispatch:
if == "euler": scheduler = FlowMatchEulerDiscreteScheduler(...)
elif== "heun": scheduler = FlowMatchHeunDiscreteScheduler(...)
elif== "pingpong": scheduler = FlowMatchPingPongScheduler(...)
# no else -> "linear" / "squared" / "sqrt" leave scheduler unbound
cfg_type dispatch:
accepts: apg | cfg | cfg_star
Catalog had:
scheduler_type: [linear, squared, sqrt] / default linear <- all invalid
cfg_type: [none, cfg, cfg_rw] / default cfg <- only cfg works
Fixed:
scheduler_type: [euler, heun, pingpong] / default euler
cfg_type: [apg, cfg, cfg_star] / default cfg
Bumped ace-step version 2 -> 3. Existing assets generated under v2
with scheduler_type=linear cannot reproduce (the value is now invalid);
v2 assets with the accidentally-valid cfg_type=cfg + a corrected
scheduler can be regenerated under v3 by mapping linear -> euler.
catalog_version stays at 1 (no schema change).
Verified end-to-end against live ace-step on irv-ml1:
POST /generate { scheduler_type: euler, cfg_type: cfg, ... }
-> 200, output_path returned, ~8s wall time
Lesson: OpenAPI introspection isn't enough for accurate catalog
authoring. Upstream OpenAPI returns bare `string` for both fields.
Reading the actual dispatch code is the only way to capture the
allowed values. Will sweep the other 11 service entries against
their implementations before P2 (scale to all services) lands.
asset_engine consumer (althing thread 01KRCJF7NGMXYE9F62Q1A6KFD4)
needed structure for ace-step's 27-field form. Two additive Pydantic
changes — backward-compatible, no catalog_version bump per the
policy table:
- CatalogField.section: str | None = None
- CatalogService.section_groups: list[CatalogSectionGroup] = []
- new CatalogSectionGroup model: {id, label, hint?}
Validator: every Field.section value must reference a declared
section_groups[].id within the same service; section_groups[].id
values are unique. CATALOG-CONTRACT.md updated with both the new
service-fields row and a versioning-policy row covering
"add optional Field/Service keys -> no bump."
ace-step entry rewritten to use the new schema:
- bumped version 1 -> 2
- declared 6 section groups (basic / generation / conditioning /
a2a / lora / output) with hints
- tagged every field with a section
- added previously-missing checkpoint_path (required: true,
default: "/app/checkpoints" — the container's mount path).
Wrapper-side cleanup (default in infer-api.py) queued as
follow-up.
- changed lyrics from optional: true -> required: true with
default "" to match upstream's `lyrics: str` shape (empty
string satisfies it).
JSON Schema regenerated.
Pydantic-model side of this change lives in asset_engine at
src/asset_engine/catalog.py — committed there separately.
Consumer-side renderer for the JSON-envelope + timestamps shape
shipped (althing thread 01KRCF4W66X3, msg 5). Smoke + regression
clean. Per the contract on the entry's notes block, flipping to
ready now that the renderer is in place.
asset_engine consumer needed to render kokoro-captioned, whose wire
shape is a JSON envelope carrying base64-encoded audio plus a
structured timestamps array. Modeling it as response.type=json
would force either a per-service-id renderer (forbidden by
brief §1.7) or extending the closed response-type vocabulary
(forbidden by brief §2.2 without a coordinated bump).
Resolution (per althing thread 01KRCF4W66X3): keep response.type
closed at the existing six values and decompose at the response
*field* level instead — the same flexibility seam already used by
mime / mime_from_field / output_field. Adds three optional keys:
- audio_field: JSON key holding base64-encoded audio bytes
- audio_format_field: JSON key holding the decoded audio MIME
- timestamps_field: JSON key holding a structured timestamps array
(independent of type, declared by any service emitting time-
aligned markers)
Validators in CatalogResponse enforce sane combinations:
- audio_field requires response.type=audio
- audio_field forbids mime_from_field
- audio_format_field requires audio_field
This is additive and backward-compatible — no catalog_version bump,
existing services parse unchanged. CATALOG-CONTRACT.md updated with
the new rows in the response-field table and a versioning-policy
row codifying that adding optional keys to response: doesn't bump.
kokoro-captioned re-shaped to use the new schema:
response:
type: audio
audio_field: audio
audio_format_field: audio_format
timestamps_field: timestamps
And marked status: experimental until the asset_engine consumer's
audio-with-timestamps renderer ships.
JSON Schema regenerated to reflect the new Pydantic shape.
Pydantic-model side of this change lives in the asset_engine repo
at src/asset_engine/catalog.py — committed there separately.
Per a request from the asset_engine consumer (althing thread
01KRCF4W66X3N24B01FF2Y7V3D), and verified against the live kokoro
OpenAPI + exercised endpoints:
* kokoro: version 1 → 2; adds three fields surfaced by the upstream
schema but not previously declared:
- speed (slider 0.25–4.0, default 1.0)
- volume_multiplier (slider 0.5–2.0, default 1.0; UI-bounded
since upstream is unbounded — noted in description)
- lang_code (text, optional override of the voice-name-derived
language hint)
* kokoro-captioned: new service entry wrapping
/dev/captioned_speech. Same model + image as kokoro proper but
separate catalog entry because the response shape is structured
JSON (audio inline as base64 + word-level timestamps), not raw
audio bytes. Verified shape captured in reproducibility.notes
so future consumers don't have to re-discover it. response.type
= json (consumer renders custom: player + subtitle overlay).
* reproducibility_audit: row added for kokoro-captioned.
Deferred (separate from this commit):
- kokoro-blend-voice. /v1/audio/voices/combine returns 403 on the
default config (allow_local_voice_saving=False); even with the
flag flipped it writes to a temp dir, not /worktank/kokoro/user_voices.
The persistent blend mechanism in this fleet is
playbooks/blend-kokoro-voice.yaml. Ad-hoc blending already works
through /v1/audio/speech via the inline syntax voice="a(w)+b(w)";
consumer can surface that as a UI affordance without any
catalog change.
catalog_version stays at 1 (no field-type vocabulary changes).
JSON Schema regeneration produced byte-identical output.
Adds the supporting infra around the service catalog now that it
has external consumers (the asset_engine UI being the first; CLIs,
monitoring, other services may follow):
- CATALOG-CONTRACT.md: the consumer-facing contract. Defines
versioning policy (catalog_version vs per-service version),
closed field-type and response-type vocabularies, recommended
vendor+drift-check sync workflow, known-consumers list, service
authoring notes.
- services.schema.json: JSON Schema (draft 2020-12) for the
catalog. Generated from the Pydantic model in
~/development/asset_engine/src/asset_engine/catalog.py via
`uv run scripts/dump_schema.py --publish`. Lets non-Python
consumers validate against the same shape.
- services.yaml: adds catalog_version: 1 at the root and reframes
the file's header to call out its first-class-contract status.
Quotes a vibevoice label that contained an unescaped colon
(caught by the asset_engine's strict YAML parser on first sync).
services.yaml: form-generator contract for the forthcoming
asset-generation UI. 13 inference services on irv-ml1 (TTS, ASR,
SFX, music) catalogued with field schemas extracted from Pydantic
models, response types, reproducibility audit, and license
warnings. ComfyUI flagged catalog-deferred (workflow-DAG API
doesn't fit a form-based UI without a per-asset-type wrapper).
design-brief.md: the prompt to give a frontend-design agent before
any pixels. Locks in the data-model decisions whose later cost is
asymmetric (asset-as-first-class entity, content-addressed output
storage, reproducibility hard requirement, job table, auth as a
no-op DI seam, API surface ≠ UI surface, schema versioning,
tags/collections plumbed in v1 with no UI). Defines a closed
field-type vocabulary (8 types) and response-renderer vocabulary
(6 types) — agent isn't allowed to extend them. Pre-decides the
required UI surfaces; leaves IA, library-nav pattern, long-job
UX, and big-form ergonomics open for the agent to opine on.
Replaced the 9-month-stale 'Last Updated: 2025-07-14' line with a
status disclaimer — this is KB-derived advisory material, not
something we maintain in lockstep with code. Tracer-date claim was
implying a freshness contract we don't fulfill. Found via
/tend-docs.
Same anti-pattern as the deleted model-list.md — a hardcoded snapshot
of mutable config that's guaranteed to drift. Replaced the 15-row
table with a one-liner pointing at stacks/llama-swap/conf/config.yaml
(the canonical source) and the live /v1/models HTTP endpoint. Found
via /tend-docs.
Self-reported "Synchronized with llama-swap config.yaml on 2025-07-18"
— 9 months stale. Canonical model truth is stacks/llama-swap/conf/config.yaml;
the static snapshot was guaranteed to drift. No historical value (not
an ADR, post-mortem, or migration runbook).
Found via /tend-docs.
New runbook captures the three-phase process:
Phase 1 — Drop --append-only via DSM Container Manager web UI
Phase 2 — sudo resticprofile forget --prune --verbose on each of
nh3-docker, nh3-dev, irv-ml1 (interactive sudo per host)
Phase 3 — Restore --append-only via DSM
Why each phase looks the way it does, what to expect (largely no-op
runs for the first 6 months while no snapshots have aged out of the
keep window), how to verify each phase non-destructively (curl 401
on the rest-server root proves the container's up + serving), what
to do if Phase 2 fails with `repository is configured as append-only`
(skipped Phase 1 / DSM didn't apply), and the path to future
automation (find docker bin path on DSM, NOPASSWD-lock syncuser to
the specific recreate command).
Includes a "last run history" table seeded with today's first
post-pipeline run (no-op, irv-ml1 only had 3 snapshots due to the
04-25→27 CUDA stall).
Cross-referenced from docs/README.md (runbook tree), docs/
orientation.md (where-to-look table), and STATUS.md item 9 (which
now points at the runbook + records the next-round date 2026-07-27).
Second pass on docs/ cleanup (item #15 in STATUS.md):
- pfi/chromadb-setup.md: deleted. References configs/pfi-ana/... and
scripts/setup-chromadb.sh, neither of which exist in this repo
(artifacts of an earlier project layout). ChromaDB is already live
per docker-stack.md; the operational truth lives there.
- pfi/docker-stack.md: removed the cross-link to the deleted file and
pulled the bit of useful content from it (auth-token generation +
client Settings example) into the inline ChromaDB section.
- pfi/vm-102-matrix-{synapse,appservice}.md: stripped the YAML
frontmatter. The `path:` values pointed at docs/pfi-ana/... which
doesn't exist in this repo; no toolchain consumed the metadata.
- README.md: tree updated to reflect the deletion.
VM-102 docs kept separate by design — each is right-sized; merging
would push past the ~500-line guideline.
Bundles the inventory expansion since 2026-04-22:
- New host dirs (READMEs + ssh-target where dir name doesn't resolve):
ana-nas, ana-wg, esh-vm-db, nh3-nas, pbs-ana, pbs-nh3.
- New PFI VM snapshots (registered + key-installed 2026-04-23):
ana-filebot, pfi-ana-webhost, pfi-postgres, pfi-pteradactyl,
pfi-tacticalrmm, sf-ana-container, sfsrv-ana (system + proxmox).
- servers/irv-ml1: ONBOARDING.md (the first-time setup notes from when
the host was brought into the fleet) + ssh-target (10.100.79.3 over
the WG tunnel — name doesn't DNS-resolve from this workstation).
- servers/{ana-ml2,pfi-pve,sf-r630}/README.md: updates to capture BMC
IPs, the iDRAC vs OS hostname distinction (sf-r630 hardware =
sfsrv-ana OS), and the ana-ml2 Supermicro BMC (10.250.250.50,
distinct from the Dell R750xs iDRAC).
- configs/homepage/docker.yaml: irv-ml1-docker provider added so
homepage auto-discovers irv-ml1's stacks over the WG tunnel.
- docs/orientation.md: narrative fleet overview written for fresh
Claude sessions — sites, backup architecture, governing principles,
gotchas, where-to-look guide. Pointed at from CLAUDE.md.
Bundles the post-2026-04-21 work that built out the two-layer backup
architecture (PBS for VM images + restic for file/DB), plus the cross-
site mirror and the disaster-recovery runbook.
- configs/restic/esh-docker-vm/profiles.yaml: drop the obsolete
*_offen_backup_data exclude (offen sidecars retired fleet-wide
2026-04-23; restic now covers the equivalent scope directly).
- configs/restic/esh-vm-db/: new profile for the dedicated DB VM
(10.0.50.60), with pre-backup pg_dumpall + mongodump hooks.
- configs/rsync/: ana-nas → nh3-nas (04:00 daily, runs as lkraven)
and nh3-nas → ana-nas (05:00 daily, runs as root because DSM
rest-server-nh3 writes mode-400 files only root can read).
- docs/runbooks/pbs-deployment.md: 9-phase PBS rollout runbook,
refined during the 2026-04-22 deployment with per-hypervisor
namespaces, NFSv3 + ZFS-case-insensitivity workaround, and the
Synology syno_acl flatten step.
- docs/runbooks/disaster-recovery.md: blast-radius runbook ordered
Tier 0 → 5 (ana-nas → hypervisors → Docker hosts → VMs → specialty);
references incident memory + recovery-step playbooks per consumer.
- services.yaml: PFI-UDMSE (10.100.0.1, NH3 edge) under Infra - NH3,
ESH-UDMPM (10.0.0.1) under Infra - ESH. PFI-UDMSE replaces the
retired Fortigate 101F at NH3 — comment updated. Both use si-ubiquiti.
Diff also folds in the previously-deployed-but-uncommitted PBS-ANA /
PBS-NH3 / IRV section / retired NH3-SW1 cleanup, bringing the tracked
copy in sync with what's live on esh-docker-vm.
- docs/README.md: navigation map of the docs/ tree (orientation /
runbooks / pfi) with what-goes-where conventions.
- docs/runbooks/tea-0.14.0-linux-amd64.sha256: deleted (build artifact,
not a runbook).
- STATUS.md: items 11 / 13 / 15 marked done; 15 leaves Matrix and
chromadb consolidation candidates as future focused-session work.
VMIDs aren't globally unique across PVE hosts (esh-pve and sfsrv-ana
both have VM 100, etc.). Without namespaces, per-hypervisor backups
collide under the same /vm/<vmid>/ path in the shared datastore.
Changes:
- New Phase 1.4b: create one namespace per hypervisor up front
(pfi-pve, nh3-pve, esh-pve, esh-pve-nas, sfsrv-ana).
- Phase 2.1 storage-entry template now lists Namespace as a required
field, set to the hypervisor's own name.
- Critical-note explaining the collision risk so future deployers
don't skip this step.
ACL grants remain at the datastore level; they apply across all child
namespaces so the existing fleet-vzdump token continues to work. Sync
job (Phase 6) preserves namespace tree to PBS-NH3 automatically.
Discovered during Phase 1.3 (mount NAS datastore). The Debian NAS's
/mnt/backup is on ZFS with casesensitivity=insensitive; NFSv4 writes
fail with EACCES even for root with no_root_squash. Known-bad
combination at the ZFS-on-Linux + NFSv4 layer.
Workaround: mount the NFS share with vers=3. PBS's chunk-based
access pattern works fine over NFSv3.
Added a section 0.5 to Phase 0 documenting the issue + both fixes
(quick: use NFSv3; cleaner: create case-sensitive child dataset).
Future deployments against this NAS won't rediscover the same wall.
Current NFS exports on the Debian NAS (10.250.50.50) use root_squash,
which blocks PBS from writing its datastore metadata (chunks, locks,
GC state — all root-owned operations). Rest-server-ana worked around
this by running its container as UID 1000, but PBS's service model
doesn't accommodate that pattern cleanly.
Solution baked into Phase 0: create a dedicated NFS export for the
PBS-ANA datastore subtree, scoped to only the PBS-ANA VM's IP, with
no_root_squash. Bounded exposure (single client), kept in a separate
.exports file so Cockpit's File Sharing module doesn't clobber it.
Flag world-scoped export on /mnt/pve-VMStorage as a non-blocking
hygiene item for a later Cockpit pass.
End-to-end runbook for standing up Proxmox Backup Server across the
fleet. Path A architecture: single primary at ANA, one-way sync to NH3
for disaster recovery. All 5 hypervisors (pfi-pve, nh3-pve, esh-pve,
esh-pve-nas, sfsrv-ana) migrate from local-dump vzdump to PBS-ANA.
Key decisions captured in the runbook:
- PBS in a Debian VM (not LXC) for clean capability model.
- PBS-ANA on pfi-pve, datastore via NFS from 10.250.50.50 —
separates backup data from hypervisor boot disk.
- PBS-NH3 on nh3-pve with local storage (independent failure
domain from ANA).
- Dedicated fleet-vzdump API token; read-only sync token for
PBS-NH3's pull job.
- sfsrv-ana specifically goes from zero backup coverage to full
vzdump coverage in Phase 3.
9 phases, each self-contained with a done-state and rollback
posture. User can stop between phases without leaving the fleet in a
bad state.
STATUS.md: added item 6b tracking this deployment. Original item 6
(cross-site rsync) now scoped to restic-only since PBS handles the
VM-image cross-site redundancy directly.
Network probes (no web admin, Debian SSH banner, only file-server ports
open) show 10.250.50.50 is vanilla Debian 12 with hand-configured NFS/SMB,
reprovisioned from the original TrueNAS SCALE install. Update stack
comments, README storage notes, Backrest description, proxmox-vms.md
entry for VM 100, and the restic configs intro to match.
Cross-site sync plan simplifies to plain rsync over SSH on both sides —
no appliance-specific tooling needed.
Captures the full workspace state built up to this point:
- CLAUDE.md + README.md describing conventions and the four-host fleet
(ana-ml2, ana-docker, nh3-docker, esh-docker-vm).
- Per-host notes under servers/<host>/ with ssh-target fallback files
and latest system-details snapshots (two in-compose credential leaks
scrubbed; the upstream compose files still need to move those to .env).
- scripts/: server_inspect.sh (read-only remote diagnostic),
refresh-server-info.sh (dir-driven discovery + snapshot capture with
validation warnings), add-host.sh, sync-stacks.sh (pull
compose/conf trees), deploy-stack.sh (push with per-file diff + prompt).
- stacks/: canonical compose for backrest, beszel, dozzle, llama-swap,
rest-server-ana, rest-server-nh3, vllm-qwen3, plus the retired
infinity reference. All use the .env-driven + traefik-net + homepage
label pattern.
- configs/restic/ana-docker/: first resticprofile config + pre-backup
hook (Synapse pg_dump, Seafile mysqldump, Vaultwarden SQLite); templates
for the other three hosts to come.
- docs/pfi/: general infrastructure reference carried over.
- .gitignore excludes .env, stacks-mirror/, and assorted secret/state
filenames to prevent re-leaks on later commits.
vh