vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

catalog(canonical): remove chatterbox dead levers + add csm-expressiva

Browse Source 
The asset-engine catalog source of truth. Removes the chatterbox
exaggeration/cfg_weight sliders (proven Turbo no-ops) — reconciling the
canonical with the fix previously applied only to asset-engine's vendored
copy — and adds the csm-expressiva whisper TTS entry (irv-ml1:8198).
This commit is contained in:
Vuong Hoang
2026-06-04 18:02:13 -07:00
parent df340c808f
commit cf35adaf6d
1 changed files with 109 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/asset-engine/services.yaml
+109 -39
View File
@@ -203,8 +203,8 @@ services:
description: >
Resemble AI's low-latency English TTS (Chatterbox-Turbo, 350M, ~75ms TTFB,
6× realtime). 28 built-in predefined voices + zero-shot cloning from a
5–30s reference. Inline paralinguistic tags, plus Resemble's signature
exaggeration / cfg_weight emotion + pacing control.
5–30s reference. Inline paralinguistic tags. (Resemble's exaggeration /
cfg_weight emotion knobs are no-ops on the Turbo checkpoint — omitted.)
category: tts
version: 2
status: ready
@@ -225,7 +225,7 @@ services:
label: Text & voice
- id: sampling
label: Expression & sampling
hint: Resemble's neutral defaults (exaggeration 0.5 / cfg_weight 0.5). Raise exaggeration or lower cfg_weight for drama.
hint: Turbo honors temperature / seed / speed_factor. (exaggeration / cfg_weight are Turbo no-ops — removed.)
- id: advanced
label: Advanced
fields:
@@ -276,32 +276,12 @@ services:
Required when voice_mode=clone. 5–30s clean WAV (16 kHz+ mono) under
/worktank/chatterbox/reference_audio/; upload via the server's
/upload_reference. Match the clip's language to `language` to avoid
accent transfer (or set cfg_weight=0).
- name: exaggeration
type: slider
section: sampling
min: 0.25
max: 2.0
step: 0.05
default: 0.5
description: >
Emotional intensity. Resemble's docs: 0.5 "works well for most prompts
across all languages"; ~0.7+ for dramatic delivery (which also speeds
speech up). NOTE: the devnen server *ships* 1.3 (tuned for its
theatrical demo presets) — 0.5 is the general-use value and the catalog
default; the shipped 1.3 is the likely cause of over-acted/unstable output.
- name: cfg_weight
type: slider
section: sampling
min: 0.0
max: 1.0
step: 0.05
default: 0.5
description: >
Pacing / prompt adherence (Resemble default 0.5). Lower to ~0.3 to
slow delivery, for fast/intense reference speakers, or alongside a
raised exaggeration for drama; 0 effectively disables guidance (useful
to reduce reference-accent transfer).
accent transfer.
# exaggeration / cfg_weight intentionally removed: Chatterbox-Turbo IGNORES
# both. Proven 2026-06-04 — same text/voice/seed across cfg 0.0/0.5/0.9 gave
# byte-identical audio, and the model logs "CFG, min_p and exaggeration are
# not supported by Turbo version and will be ignored." They were dead UI
# sliders. They'd only do something on a non-turbo Chatterbox checkpoint.
- name: temperature
type: slider
section: sampling
@@ -368,16 +348,16 @@ services:
warm_per_unit: "~75ms TTFB, 6× realtime"
license: MIT
notes: |
Routes to the devnen wrapper's richer /tts (full control surface:
exaggeration / cfg_weight / temperature / speed_factor / seed / voice_mode)
instead of the thin OpenAI /v1/audio/speech, which exposes NONE of the
emotion knobs — that omission was why prior output was poor. Same wrapper
author as the `dia` stack; identical predefined/clone voice model.
Defaults sourced from Resemble's README (exaggeration + cfg_weight = 0.5)
and the server's generation_defaults (temperature 0.8, speed 1.0, seed 0),
read live 2026-06-01; the server's shipped exaggeration 1.3 is demo-tuned
and deliberately NOT adopted. Python 3.10 only (wrapper hardcoding);
multilingual (23-language) variant available via the stack .env.
Routes to the devnen wrapper's /tts (control surface: temperature /
speed_factor / seed / voice_mode) rather than the thin OpenAI
/v1/audio/speech. exaggeration / cfg_weight were DROPPED 2026-06-04 — the
Turbo checkpoint ignores them (proven byte-identical across cfg values; the
model warns they're "not supported by Turbo version and will be ignored"),
so they were dead UI sliders. Same wrapper author as the `dia` stack;
identical predefined/clone voice model. Defaults from the server's
generation_defaults (temperature 0.8, speed 1.0, seed 0), read live
2026-06-01. Python 3.10 only (wrapper hardcoding); multilingual (23-language)
variant available via the stack .env.
- id: chatterbox-fast
name: Chatterbox Fast (streaming)
@@ -673,6 +653,91 @@ services:
notes: |
Voice cloning shape differs from CosyVoice: profile-based, not voice-id-based.
- id: csm-expressiva
name: CSM Expressiva (whisper)
description: >
Experimental LoRA fine-tune of Sesame's CSM-1B (Conversational Speech Model)
on Expresso's whispering voice — WHISPER-style speech only, trained speaker
= 4. Custom thin server (no upstream CSM server image exists). Autoregressive,
so ~RTF 1.8 on the A6000 — good for offline/clip generation, not live
streaming. License cc-by-nc-4.0 (non-commercial). No output watermark.
category: tts
version: 1
status: experimental
host: irv-ml1
lifecycle:
stack: csm-expressiva
vram_gb: 4
gpu_device_id: 1
endpoint: http://10.100.79.3:8198/tts
method: POST
content_type: application/json
model:
id: senstella/csm-expressiva-1b
revision: null
image: local/csm-expressiva:v1
section_groups:
- id: basic
label: Text
- id: sampling
label: Sampling
fields:
- name: text
type: textarea
label: Text
section: basic
required: true
max_length: 2000
description: >
Text to synthesize as a whisper. CSM is conversational — short
utterances work best. No inline paralinguistic tag support.
- name: speaker
type: number
section: basic
required: false
default: 4
description: >
Speaker id. The fine-tune only trained speaker 4 (Expresso whisper);
leave at 4 — other ids fall back to off-distribution base-CSM behavior.
- name: max_audio_length_ms
type: number
section: sampling
required: false
default: 20000
description: Hard cap on generated audio length (ms); generation stops early on EOS.
- name: temperature
type: slider
section: sampling
min: 0.05
max: 1.5
step: 0.05
default: 0.8
description: Sampling temperature. Repo (csm-mlx) example default 0.8.
- name: topk
type: number
section: sampling
required: false
default: 50
description: Top-k sampling cutoff. Repo example default 50.
response:
type: audio
mime: audio/wav
reproducibility:
seedable: false
deterministic: false
notes: >
No seed field exposed; temperature-sampled, so takes vary run to run.
estimated_latency:
cold_start_s: 30
warm_per_unit: "~RTF 1.8 (≈9s compute per ≈5s of audio on the A6000)"
license: cc-by-nc-4.0
notes: |
Whisper-only experimental fine-tune. Custom thin FastAPI wrapper around the
vendored Sesame csm generator (watermark + silentcipher stripped; ungated
Llama-3.2 tokenizer mirror). Runs eager — CSM has no torch.compile hot path,
so the compiled image gained nothing. NOT behind the LiteLLM gateway (custom
/tts schema, not OpenAI /v1/audio/speech).
- id: cosyvoice
name: CosyVoice 3 (Multilingual)
description: >
@@ -2132,6 +2197,11 @@ reproducibility_audit:
model_deterministic: false
image_tag_mutable: true
notes: "/tts exposes seed (0=random) — one-shot byte-exact for fixed seed+params (verified 2026-06-02); STREAMING non-reproducible (timing-dependent chunk boundaries), auditions ephemeral. PerTh watermark unconditional. image local/chatterbox-fast:v1 is tag-pinned + mutable — pin a digest for true repro."
- service: csm-expressiva
seedable: false
model_deterministic: false
image_tag_mutable: true
notes: "No seed field exposed; temperature-sampled, takes vary run to run. No watermark (stripped). image local/csm-expressiva:v1 is tag-pinned + mutable — pin a digest for true repro. Runs eager (CSM has no torch.compile hot path)."
- service: index-tts
seedable: false
model_deterministic: true