diff --git a/docs/asset-engine/services.yaml b/docs/asset-engine/services.yaml index fee16c5..6c1803a 100644 --- a/docs/asset-engine/services.yaml +++ b/docs/asset-engine/services.yaml @@ -203,8 +203,8 @@ services: description: > Resemble AI's low-latency English TTS (Chatterbox-Turbo, 350M, ~75ms TTFB, 6× realtime). 28 built-in predefined voices + zero-shot cloning from a - 5–30s reference. Inline paralinguistic tags, plus Resemble's signature - exaggeration / cfg_weight emotion + pacing control. + 5–30s reference. Inline paralinguistic tags. (Resemble's exaggeration / + cfg_weight emotion knobs are no-ops on the Turbo checkpoint — omitted.) category: tts version: 2 status: ready @@ -225,7 +225,7 @@ services: label: Text & voice - id: sampling label: Expression & sampling - hint: Resemble's neutral defaults (exaggeration 0.5 / cfg_weight 0.5). Raise exaggeration or lower cfg_weight for drama. + hint: Turbo honors temperature / seed / speed_factor. (exaggeration / cfg_weight are Turbo no-ops — removed.) - id: advanced label: Advanced fields: @@ -276,32 +276,12 @@ services: Required when voice_mode=clone. 5–30s clean WAV (16 kHz+ mono) under /worktank/chatterbox/reference_audio/; upload via the server's /upload_reference. Match the clip's language to `language` to avoid - accent transfer (or set cfg_weight=0). - - name: exaggeration - type: slider - section: sampling - min: 0.25 - max: 2.0 - step: 0.05 - default: 0.5 - description: > - Emotional intensity. Resemble's docs: 0.5 "works well for most prompts - across all languages"; ~0.7+ for dramatic delivery (which also speeds - speech up). NOTE: the devnen server *ships* 1.3 (tuned for its - theatrical demo presets) — 0.5 is the general-use value and the catalog - default; the shipped 1.3 is the likely cause of over-acted/unstable output. - - name: cfg_weight - type: slider - section: sampling - min: 0.0 - max: 1.0 - step: 0.05 - default: 0.5 - description: > - Pacing / prompt adherence (Resemble default 0.5). Lower to ~0.3 to - slow delivery, for fast/intense reference speakers, or alongside a - raised exaggeration for drama; 0 effectively disables guidance (useful - to reduce reference-accent transfer). + accent transfer. + # exaggeration / cfg_weight intentionally removed: Chatterbox-Turbo IGNORES + # both. Proven 2026-06-04 — same text/voice/seed across cfg 0.0/0.5/0.9 gave + # byte-identical audio, and the model logs "CFG, min_p and exaggeration are + # not supported by Turbo version and will be ignored." They were dead UI + # sliders. They'd only do something on a non-turbo Chatterbox checkpoint. - name: temperature type: slider section: sampling @@ -368,16 +348,16 @@ services: warm_per_unit: "~75ms TTFB, 6× realtime" license: MIT notes: | - Routes to the devnen wrapper's richer /tts (full control surface: - exaggeration / cfg_weight / temperature / speed_factor / seed / voice_mode) - instead of the thin OpenAI /v1/audio/speech, which exposes NONE of the - emotion knobs — that omission was why prior output was poor. Same wrapper - author as the `dia` stack; identical predefined/clone voice model. - Defaults sourced from Resemble's README (exaggeration + cfg_weight = 0.5) - and the server's generation_defaults (temperature 0.8, speed 1.0, seed 0), - read live 2026-06-01; the server's shipped exaggeration 1.3 is demo-tuned - and deliberately NOT adopted. Python 3.10 only (wrapper hardcoding); - multilingual (23-language) variant available via the stack .env. + Routes to the devnen wrapper's /tts (control surface: temperature / + speed_factor / seed / voice_mode) rather than the thin OpenAI + /v1/audio/speech. exaggeration / cfg_weight were DROPPED 2026-06-04 — the + Turbo checkpoint ignores them (proven byte-identical across cfg values; the + model warns they're "not supported by Turbo version and will be ignored"), + so they were dead UI sliders. Same wrapper author as the `dia` stack; + identical predefined/clone voice model. Defaults from the server's + generation_defaults (temperature 0.8, speed 1.0, seed 0), read live + 2026-06-01. Python 3.10 only (wrapper hardcoding); multilingual (23-language) + variant available via the stack .env. - id: chatterbox-fast name: Chatterbox Fast (streaming) @@ -673,6 +653,91 @@ services: notes: | Voice cloning shape differs from CosyVoice: profile-based, not voice-id-based. + - id: csm-expressiva + name: CSM Expressiva (whisper) + description: > + Experimental LoRA fine-tune of Sesame's CSM-1B (Conversational Speech Model) + on Expresso's whispering voice — WHISPER-style speech only, trained speaker + = 4. Custom thin server (no upstream CSM server image exists). Autoregressive, + so ~RTF 1.8 on the A6000 — good for offline/clip generation, not live + streaming. License cc-by-nc-4.0 (non-commercial). No output watermark. + category: tts + version: 1 + status: experimental + host: irv-ml1 + lifecycle: + stack: csm-expressiva + vram_gb: 4 + gpu_device_id: 1 + endpoint: http://10.100.79.3:8198/tts + method: POST + content_type: application/json + model: + id: senstella/csm-expressiva-1b + revision: null + image: local/csm-expressiva:v1 + section_groups: + - id: basic + label: Text + - id: sampling + label: Sampling + fields: + - name: text + type: textarea + label: Text + section: basic + required: true + max_length: 2000 + description: > + Text to synthesize as a whisper. CSM is conversational — short + utterances work best. No inline paralinguistic tag support. + - name: speaker + type: number + section: basic + required: false + default: 4 + description: > + Speaker id. The fine-tune only trained speaker 4 (Expresso whisper); + leave at 4 — other ids fall back to off-distribution base-CSM behavior. + - name: max_audio_length_ms + type: number + section: sampling + required: false + default: 20000 + description: Hard cap on generated audio length (ms); generation stops early on EOS. + - name: temperature + type: slider + section: sampling + min: 0.05 + max: 1.5 + step: 0.05 + default: 0.8 + description: Sampling temperature. Repo (csm-mlx) example default 0.8. + - name: topk + type: number + section: sampling + required: false + default: 50 + description: Top-k sampling cutoff. Repo example default 50. + response: + type: audio + mime: audio/wav + reproducibility: + seedable: false + deterministic: false + notes: > + No seed field exposed; temperature-sampled, so takes vary run to run. + estimated_latency: + cold_start_s: 30 + warm_per_unit: "~RTF 1.8 (≈9s compute per ≈5s of audio on the A6000)" + license: cc-by-nc-4.0 + notes: | + Whisper-only experimental fine-tune. Custom thin FastAPI wrapper around the + vendored Sesame csm generator (watermark + silentcipher stripped; ungated + Llama-3.2 tokenizer mirror). Runs eager — CSM has no torch.compile hot path, + so the compiled image gained nothing. NOT behind the LiteLLM gateway (custom + /tts schema, not OpenAI /v1/audio/speech). + - id: cosyvoice name: CosyVoice 3 (Multilingual) description: > @@ -2132,6 +2197,11 @@ reproducibility_audit: model_deterministic: false image_tag_mutable: true notes: "/tts exposes seed (0=random) — one-shot byte-exact for fixed seed+params (verified 2026-06-02); STREAMING non-reproducible (timing-dependent chunk boundaries), auditions ephemeral. PerTh watermark unconditional. image local/chatterbox-fast:v1 is tag-pinned + mutable — pin a digest for true repro." + - service: csm-expressiva + seedable: false + model_deterministic: false + image_tag_mutable: true + notes: "No seed field exposed; temperature-sampled, takes vary run to run. No watermark (stripped). image local/csm-expressiva:v1 is tag-pinned + mutable — pin a digest for true repro. Runs eager (CSM has no torch.compile hot path)." - service: index-tts seedable: false model_deterministic: true