vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

catalog(dia2): expose full /tts control surface + stable predefined voices

Browse Source 
Repoint both dia2 entries from /v1/audio/speech to the wrapper's richer /tts
endpoint (CustomTTSRequest), exposing the levers that fix the random-voice
problem: voice_mode, clone_reference_filename, cfg_scale, temperature, top_p,
cfg_filter_top_k, speed_factor, seed, split_text, chunk_size, transcript,
max_tokens. All defaults are the wrapper's Pydantic blessed values (cfg 3.0 /
temp 1.3 / top_p 0.95 / top_k 35 / speed_factor 0.94 / chunk 300). Fields
grouped (basic/sampling/advanced). dia2 -> version 2 (field-shape change).

Voice stability: Dia2 samples a random speaker per call unless anchored. The
43 curated voices baked at /app/voices aren't reachable from /tts's clone path
(reference_audio dir only), so they're staged into reference_audio; the
clone_reference_filename picker now sources /get_reference_files. voice_mode=
clone + a reference filename pins voice/gender. Verified /tts clone end-to-end
(HTTP 200, Ogg/Opus 24 kHz). README documents the staging + two-instance shape.
This commit is contained in:
Vuong Hoang
2026-05-31 15:18:53 -07:00
parent e97d80cb8e
commit 55602b7251
2 changed files with 260 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/asset-engine/services.yaml
+215 -50
View File
@@ -1248,77 +1248,159 @@ services:
turn-taking in one pass with [S1]/[S2] speaker tags and nonverbals
(laughs)/(coughs)/(sighs). 24 kHz. Served via devnen/Dia-TTS-Server.
category: tts
version: 1
version: 2
status: ready
host: irv-ml1
lifecycle:
stack: dia
vram_gb: 8
gpu_device_id: 0
endpoint: http://10.100.79.3:8200/v1/audio/speech
endpoint: http://10.100.79.3:8200/tts
method: POST
content_type: application/json
model:
id: nari-labs/Dia2-2B
revision: null
image: local/dia:v2
section_groups:
- id: basic
label: Text & voice
- id: sampling
label: Sampling
hint: Wrapper-canonical Dia2 defaults; lower temperature/cfg for steadier output.
- id: advanced
label: Advanced
fields:
- name: input
- name: text
type: textarea
label: Text ([S1]/[S2] dialogue + nonverbals)
section: basic
required: true
max_length: 5000
default: "[S1] Hello Dia2!" # canonical: nari-labs/dia2 README usage example
description: >
[S1]/[S2] tags mark speaker turns; nonverbals like (laughs),
(coughs), (sighs), (clears throat) go inline.
- name: voice
- name: voice_mode
type: select
label: Voice / mode
options: [dialogue, single_s1, single_s2, predefined]
default: dialogue
label: Voice mode
section: basic
options: [single_s1, single_s2, dialogue, clone]
default: single_s1
description: >
Speaker mode from the model's voice set. `clone` (prefix_speaker)
needs a reference clip, so it isn't a plain picker choice here.
- name: response_format
Canonical default single_s1. Unconditioned modes (single_s1/s2,
dialogue) sample a RANDOM speaker each call — gender varies. For a
STABLE voice, set this to clone and pick a clone_reference_filename.
- name: clone_reference_filename
type: select
label: Voice (clone reference)
section: basic
optional: true
source_url: http://10.100.79.3:8200/get_reference_files
source_jsonpath: $[*]
description: >
Required when voice_mode=clone. Lists the reference_audio library —
43 curated voices (singles + [S1]/[S2] dialogue pairs like
Abigail_Taylor.wav) staged there from the image's /app/voices, plus
any you upload. Pins voice/gender deterministically.
- name: transcript
type: textarea
label: Reference transcript
section: advanced
optional: true
description: >
Optional transcript of a custom clone reference (overrides its local
.txt sidecar). Leave blank for predefined voices.
- name: output_format
type: select
section: basic
options: [opus, wav]
default: opus
description: 24 kHz output (Dia2; note dia1 was 44.1 kHz).
- name: speed
description: 24 kHz (Dia2).
- name: cfg_scale
type: slider
section: sampling
min: 1.0
max: 5.0
step: 0.1
default: 3.0
description: Classifier-free guidance scale (wrapper default 3.0).
- name: temperature
type: slider
section: sampling
min: 0.1
max: 1.5
step: 0.05
default: 1.3
description: Sampling temperature; lower = steadier (wrapper default 1.3).
- name: top_p
type: slider
section: sampling
min: 0.1
max: 1.0
step: 0.01
default: 0.95
- name: cfg_filter_top_k
type: number
section: sampling
required: false
default: 35
description: Top-k filter for CFG guidance (1–100; wrapper default 35).
- name: speed_factor
type: slider
section: sampling
min: 0.5
max: 2.0
step: 0.05
default: 1.0
description: Post-generation playback speed multiplier.
step: 0.01
default: 0.94
description: Playback speed (wrapper canonical default 0.94).
- name: seed
type: number
section: sampling
required: false
default: -1
description: -1 = random; any integer for repeatable (not byte-exact) output.
description: -1 = random; a fixed integer repeats the same take.
- name: split_text
type: bool
section: advanced
default: true
description: Auto-split long text into chunks.
- name: chunk_size
type: slider
section: advanced
min: 100
max: 1000
step: 10
default: 300
description: Target chunk length in chars when splitting (wrapper default 300).
- name: max_tokens
type: number
section: advanced
required: false
description: Max audio tokens per chunk; omit to use the model's internal default.
response:
type: audio
mime_from_field: response_format
mime_from_field: output_format
reproducibility:
seedable: true
deterministic: false
notes: >
Seed gives consistent voice/prosody but NOT byte-exact output (Dia2 is
temperature-sampled). Raw-bytes response carries no seed echo, so no
seed_field. A richer custom /tts endpoint exposes
cfg_scale/temperature/top_p/cfg_filter_top_k.
Per the dia2 README, voices vary per generation unless anchored — set
voice_mode=clone + a clone_reference_filename for a stable voice/gender.
seed (-1=random) repeats a take but doesn't choose the voice;
temperature-sampled, not byte-exact; raw bytes -> no seed_field. All
field defaults are the wrapper's CustomTTSRequest Pydantic blessed values.
estimated_latency:
cold_start_s: 30
warm_per_unit: "dialogue one-pass; ~realtime on the 3090"
license: "Apache-2.0 (Dia2 weights); MIT (devnen wrapper)"
notes: |
One of two fixed-model instances in the `dia` stack; sibling dia2-1b
(:8202). `docker compose up` on the stack starts BOTH services — the
lifecycle vram_gb here is this instance alone (~8 GB); budget for both
if gating the whole stack. Image local/dia:v2 = upstream devnen wrapper
+ the dia2 package baked in (see stacks/dia/dia2-image/). Verified
end-to-end 2026-05-31 (HTTP 200, Ogg/Opus 24 kHz).
Routes to the wrapper's richer /tts endpoint (full control surface:
cfg_scale/temperature/top_p/cfg_filter_top_k/voice_mode/clone) rather than
the OpenAI /v1/audio/speech. One of two fixed-model instances in the `dia`
stack; sibling dia2-1b (:8202). `docker compose up` starts BOTH (~8 GB
here; budget both if gating the whole stack). Image local/dia:v2 (see
stacks/dia/dia2-image/). Verified end-to-end 2026-05-31.
- id: dia2-1b
name: Dia2 1B (Streaming)
@@ -1327,73 +1409,156 @@ services:
dialogue + nonverbals as dia2-2b, lower latency / lighter weight. 24 kHz.
Served via devnen/Dia-TTS-Server.
category: tts
version: 1
version: 2
status: ready
host: irv-ml1
lifecycle:
stack: dia
vram_gb: 5
gpu_device_id: 0
endpoint: http://10.100.79.3:8202/v1/audio/speech
endpoint: http://10.100.79.3:8202/tts
method: POST
content_type: application/json
model:
id: nari-labs/Dia2-1B
revision: null
image: local/dia:v2
section_groups:
- id: basic
label: Text & voice
- id: sampling
label: Sampling
hint: Wrapper-canonical Dia2 defaults; lower temperature/cfg for steadier output.
- id: advanced
label: Advanced
fields:
- name: input
- name: text
type: textarea
label: Text ([S1]/[S2] dialogue + nonverbals)
section: basic
required: true
max_length: 5000
default: "[S1] Hello Dia2!" # canonical: nari-labs/dia2 README usage example
description: >
[S1]/[S2] tags mark speaker turns; nonverbals like (laughs),
(coughs), (sighs), (clears throat) go inline.
- name: voice
- name: voice_mode
type: select
label: Voice / mode
options: [dialogue, single_s1, single_s2, predefined]
default: dialogue
label: Voice mode
section: basic
options: [single_s1, single_s2, dialogue, clone]
default: single_s1
description: >
Speaker mode from the model's voice set. `clone` (prefix_speaker)
needs a reference clip, so it isn't a plain picker choice here.
- name: response_format
Canonical default single_s1. Unconditioned modes (single_s1/s2,
dialogue) sample a RANDOM speaker each call — gender varies. For a
STABLE voice, set this to clone and pick a clone_reference_filename.
- name: clone_reference_filename
type: select
label: Voice (clone reference)
section: basic
optional: true
source_url: http://10.100.79.3:8202/get_reference_files
source_jsonpath: $[*]
description: >
Required when voice_mode=clone. Lists the reference_audio library —
43 curated voices (singles + [S1]/[S2] dialogue pairs like
Abigail_Taylor.wav) staged there from the image's /app/voices, plus
any you upload. Pins voice/gender deterministically.
- name: transcript
type: textarea
label: Reference transcript
section: advanced
optional: true
description: >
Optional transcript of a custom clone reference (overrides its local
.txt sidecar). Leave blank for predefined voices.
- name: output_format
type: select
section: basic
options: [opus, wav]
default: opus
description: 24 kHz output.
- name: speed
description: 24 kHz (Dia2).
- name: cfg_scale
type: slider
section: sampling
min: 1.0
max: 5.0
step: 0.1
default: 3.0
description: Classifier-free guidance scale (wrapper default 3.0).
- name: temperature
type: slider
section: sampling
min: 0.1
max: 1.5
step: 0.05
default: 1.3
description: Sampling temperature; lower = steadier (wrapper default 1.3).
- name: top_p
type: slider
section: sampling
min: 0.1
max: 1.0
step: 0.01
default: 0.95
- name: cfg_filter_top_k
type: number
section: sampling
required: false
default: 35
description: Top-k filter for CFG guidance (1–100; wrapper default 35).
- name: speed_factor
type: slider
section: sampling
min: 0.5
max: 2.0
step: 0.05
default: 1.0
description: Post-generation playback speed multiplier.
step: 0.01
default: 0.94
description: Playback speed (wrapper canonical default 0.94).
- name: seed
type: number
section: sampling
required: false
default: -1
description: -1 = random; any integer for repeatable (not byte-exact) output.
description: -1 = random; a fixed integer repeats the same take.
- name: split_text
type: bool
section: advanced
default: true
description: Auto-split long text into chunks.
- name: chunk_size
type: slider
section: advanced
min: 100
max: 1000
step: 10
default: 300
description: Target chunk length in chars when splitting (wrapper default 300).
- name: max_tokens
type: number
section: advanced
required: false
description: Max audio tokens per chunk; omit to use the model's internal default.
response:
type: audio
mime_from_field: response_format
mime_from_field: output_format
reproducibility:
seedable: true
deterministic: false
notes: >
Same determinism story as dia2-2b (temperature-sampled, seed gives
consistent prosody not byte-exact). No seed echo -> no seed_field.
Same story as dia2-2b: voices vary per generation unless anchored
(voice_mode=clone + clone_reference_filename); temperature-sampled,
not byte-exact; raw bytes -> no seed_field. Defaults are the wrapper's
CustomTTSRequest Pydantic blessed values.
estimated_latency:
cold_start_s: 30
warm_per_unit: "streaming; lower latency than dia2-2b"
license: "Apache-2.0 (Dia2 weights); MIT (devnen wrapper)"
notes: |
Sibling of dia2-2b in the same `dia` stack (see its notes for the
single-model-per-endpoint rationale and the both-start-together caveat).
Image local/dia:v2. Verified end-to-end 2026-05-31 (HTTP 200, Ogg/Opus
24 kHz).
Routes to the wrapper's richer /tts endpoint (same control surface as
dia2-2b). Sibling of dia2-2b in the same `dia` stack (see its notes for
the single-model-per-endpoint rationale + both-start-together caveat).
Image local/dia:v2. Verified end-to-end 2026-05-31.
- id: csm
name: Sesame CSM (conversational)
stacks/dia/README.md
+45 -10
View File
@@ -54,15 +54,50 @@ scripts/deploy-stack.sh irv-ml1 dia
First boot pulls the checkpoint (~6-10 GB) into `DIA_CACHE_DIR` and can
take several minutes; the healthcheck's 600 s `start_period` covers it.
## Deployment shape (as of 2026-05-31)
This stack now runs **two fixed-model instances** from `local/dia:v2`
(the dia2-capable image — see [`dia2-image/Dockerfile`](dia2-image/Dockerfile)):
| service | model | port | notes |
|---|---|---|---|
| `dia2-2b` | nari-labs/Dia2-2B | 8200 | highest quality |
| `dia2-1b` | nari-labs/Dia2-1B | 8202 | streaming / lower latency |
The wrapper is single-model and ignores per-request model selection, so
one fixed instance per model is the only way to offer both as real
asset-engine choices. Legacy Dia 1.6B was retired. `local/dia:v2` is built
in two stages: upstream wrapper → `local/dia:v1`, then `dia2-image/` layers
in the dia2 package + its deps. Each instance pins its model via a mounted
`/opt/docker/conf/dia2-*/config.yaml`.
## Voices — stabilizing the random-voice behavior
Dia2 samples a **random speaker (random gender) per generation** unless
anchored (per the dia2 README: "voices vary per generation … use with
prefix … for stable output"). To pin a voice, use the richer **`/tts`**
endpoint with `voice_mode: clone` + a `clone_reference_filename`.
The image bakes **43 curated voices** at `/app/voices` (singles +
`[S1]/[S2]` dialogue pairs like `Abigail_Taylor.wav`), but `/tts`'s clone
path only reads the **reference_audio** dir — so they're **staged** into it:
```bash
# one-time per host (writes through the shared /worktank/dia/reference_audio mount):
docker exec dia2-2b sh -c 'cp -n /app/voices/* /app/reference_audio/'
```
After staging, `GET /get_reference_files` lists them and asset-engine's
`clone_reference_filename` picker (sourced from that endpoint) offers a
stable, known voice. Restic-included, so it survives once staged.
## Notes
- **Pin `DIA_SHA`** to a full 40-char commit before relying on this —
`.env.example` ships `main` for convenience, but `main` is not
reproducible (chatterbox learned this when an upstream restructure
broke its `main` build).
- **Model switching** is config.yaml-driven in the wrapper, or live from
the Web UI at `http://10.100.79.3:8200/`. To pin a non-default model
declaratively, mount a host `config.yaml` (see the commented volume in
`compose.yaml`).
- Endpoints: `/v1/audio/speech` (OpenAI-compat), `/health` (liveness),
`/api/model-status` (download/load progress), `/api/model-info`.
- **Model switching** within an instance is config.yaml-driven (the mounted
`config.yaml` pins `model.repo_id`); the Web UI can hot-swap live but only
the mounted config survives recreate.
- Endpoints: `/tts` (rich: cfg_scale/temperature/top_p/cfg_filter_top_k/
voice_mode/clone — what asset-engine targets), `/v1/audio/speech`
(OpenAI-compat; its `voice` param also resolves predefined voices by name),
`/get_reference_files`, `/get_predefined_voices`, `/health`,
`/api/model-status`, `/api/model-info`.