stacks: add csm (Sesame Conversational Speech Model) to the TTS bench
Sesame CSM-1B via phildougherty/sesame_csm_openai — OpenAI-compat /v1/audio/speech, context-aware conversational speech (voice-agent layer, not a plain reader). Port 8201 on irv-ml1. Gated model: requires CSM_HF_TOKEN (license acceptance) — placeholder in .env.example, real token only in host .env.
This commit is contained in:
@@ -0,0 +1,49 @@
|
||||
# Sesame CSM stack tunables. Copy to `.env` on irv-ml1 before deploying.
|
||||
|
||||
# ── build pin ────────────────────────────────────────────────────────
|
||||
# SHA of phildougherty/sesame_csm_openai. Use the FULL 40-char SHA;
|
||||
# `main` works but is NOT reproducible — pin before relying on it.
|
||||
# https://github.com/phildougherty/sesame_csm_openai/commits/main
|
||||
CSM_SHA=main
|
||||
|
||||
# Local image tag — bump to force a fresh layer build.
|
||||
CSM_TAG=v1
|
||||
|
||||
# ── network ──────────────────────────────────────────────────────────
|
||||
# Host port. Container listens on 8000 internally.
|
||||
# Reserved on irv-ml1: 8188 ComfyUI, 8190 CosyVoice, 8191 Qwen3-TTS,
|
||||
# 8192 IndexTTS-2, 8193 Kokoro, 8194 VibeVoice, 8195 Fish-S2,
|
||||
# 8196 Chatterbox, 8197 Voxtral, 8198 Kyutai, 8199 Zonos, 8200 Dia,
|
||||
# 8765 Parakeet. 8201 picked here.
|
||||
CSM_PORT=8201
|
||||
|
||||
# Bind address. 0.0.0.0 exposes on all interfaces (incl. the WG tunnel
|
||||
# interface 10.100.79.3); 127.0.0.1 restricts to local-only.
|
||||
CSM_BIND=0.0.0.0
|
||||
|
||||
# ── runtime / GPU ────────────────────────────────────────────────────
|
||||
# Devices visible inside the container. "0" pins to the RTX 3090
|
||||
# (24 GB) — CSM-1B (~8 GB) fits; use "1" for the A6000.
|
||||
CSM_GPU_DEVICES=0
|
||||
|
||||
# Optional HF device_map for multi-GPU: auto | balanced | sequential.
|
||||
# Leave empty for single-GPU (the common case here).
|
||||
CSM_DEVICE_MAP=
|
||||
|
||||
# ── secrets ──────────────────────────────────────────────────────────
|
||||
# REQUIRED. sesame/csm-1b is a GATED model: accept its license at
|
||||
# https://huggingface.co/sesame/csm-1b
|
||||
# then paste a HF token (read scope) here. Without it, first-boot model
|
||||
# download 401s. This file is .env.example (committed) — put the REAL
|
||||
# token only in the .env on the host, which is gitignored.
|
||||
CSM_HF_TOKEN=
|
||||
|
||||
# ── persistent storage on the host ───────────────────────────────────
|
||||
# Voices dir — cloned/custom voice samples the wrapper serves by ID.
|
||||
# Included in restic (clones need the original sample to recreate).
|
||||
CSM_VOICES_DIR=/worktank/csm/voices
|
||||
|
||||
# HuggingFace cache — csm-1b + Llama-3.2-1B tokenizer + Mimi land here
|
||||
# on first start. Bind-mounted to survive recreate. Excluded from restic
|
||||
# (regenerable from HF, given the token + license acceptance).
|
||||
CSM_CACHE_DIR=/worktank/csm/cache
|
||||
@@ -0,0 +1,65 @@
|
||||
# Sesame CSM
|
||||
|
||||
Sesame's **Conversational Speech Model** (the engine behind the
|
||||
"Maya/Miles" demo) — a context-aware speech generator (Llama backbone +
|
||||
Mimi audio codec) — served via
|
||||
[phildougherty/sesame_csm_openai](https://github.com/phildougherty/sesame_csm_openai),
|
||||
an OpenAI-compat wrapper.
|
||||
|
||||
**Server:** irv-ml1 (Irvine, WireGuard-only)
|
||||
**Port:** 8201 (container listens on 8000)
|
||||
**GPUs:** pins to device 0 (RTX 3090) by default; ~8 GB VRAM
|
||||
**Image:** `local/csm:v1` — built locally from a pinned git SHA of the
|
||||
wrapper via docker buildx's git URL context
|
||||
**Upstream wrapper:** [phildougherty/sesame_csm_openai](https://github.com/phildougherty/sesame_csm_openai) (MIT)
|
||||
**Upstream model:** [sesame/csm-1b](https://huggingface.co/sesame/csm-1b)
|
||||
(**gated** — Sesame's own license) + `unsloth/Llama-3.2-1B` tokenizer + Mimi codec
|
||||
|
||||
## Why this stack exists — and what it is *not*
|
||||
|
||||
CSM is a **conversational** speech model, not a plain reader. It
|
||||
conditions on prior conversation turns (text **and** audio) to choose
|
||||
prosody and tone — it's designed to be the **speech layer of a voice
|
||||
agent**, where a separate LLM produces the words and CSM voices them
|
||||
with context-aware delivery.
|
||||
|
||||
- As **pure TTS** it works standalone (OpenAI-compat
|
||||
`POST /v1/audio/speech`, voices `alloy/echo/fable/onyx/nova/shimmer`
|
||||
plus cloned IDs).
|
||||
- Its real edge — contextual prosody across turns — only pays off in
|
||||
**interactive / voice-agent** use, not monologue narration. For
|
||||
skaldsong's *reader*, the emotive single-voice engines (Fish S2-Pro,
|
||||
IndexTTS-2) and Dia (dialogue) remain the better fits; CSM is here for
|
||||
voice-agent experiments.
|
||||
|
||||
## ⚠️ Gated model — token required before first boot
|
||||
|
||||
`sesame/csm-1b` is gated. Before deploying:
|
||||
|
||||
1. Accept the license at <https://huggingface.co/sesame/csm-1b>.
|
||||
2. Put a HF token (read scope) in `CSM_HF_TOKEN` in the host `.env`
|
||||
(never in `.env.example`).
|
||||
|
||||
Without it the first-boot model download 401s.
|
||||
|
||||
## Deploy
|
||||
|
||||
```bash
|
||||
# from this workstation (irv-ml1 is WG-only — routes via ana-wg):
|
||||
scripts/deploy-stack.sh irv-ml1 csm
|
||||
# then on irv-ml1, first run builds from the pinned SHA:
|
||||
# docker compose up -d --build
|
||||
```
|
||||
|
||||
First boot pulls csm-1b + the Llama-3.2-1B tokenizer + Mimi into
|
||||
`CSM_CACHE_DIR`; the 600 s `start_period` covers it.
|
||||
|
||||
## Notes
|
||||
|
||||
- **Pin `CSM_SHA`** to a full 40-char commit before relying on this —
|
||||
`.env.example` ships `main`, which is not reproducible.
|
||||
- The wrapper can also serve Dia-1.6B, but we run Dia from its own
|
||||
[`dia`](../dia/) stack — keep this one CSM-only to avoid overlap.
|
||||
- Endpoints: `/v1/audio/speech` (OpenAI-compat), `/v1/audio/voices`,
|
||||
`/v1/audio/models`. No dedicated `/health` route — the healthcheck
|
||||
probes `/v1/audio/voices` (only answers once the model is loaded).
|
||||
@@ -0,0 +1,62 @@
|
||||
# Sesame CSM-1B (Conversational Speech Model) served via
|
||||
# phildougherty/sesame_csm_openai — an OpenAI-compat wrapper around
|
||||
# Sesame's context-aware speech model (Llama backbone + Mimi codec).
|
||||
#
|
||||
# Why this stack exists alongside the other TTS:
|
||||
# * CSM is a CONVERSATIONAL speech layer, not a plain reader — it
|
||||
# conditions on prior turns (text + audio) to pick prosody, built
|
||||
# for voice-AGENT turn-taking. As pure TTS it works standalone;
|
||||
# its differentiator only pays off in interactive/agent use.
|
||||
# * OpenAI-compat /v1/audio/speech with 6 standard voices
|
||||
# (alloy/echo/fable/onyx/nova/shimmer) + cloned voice IDs.
|
||||
# * ~8 GB VRAM; fits the 3090 (device 0) or A6000 (device 1).
|
||||
#
|
||||
# NOTE: sesame/csm-1b is a GATED model — you must accept its license at
|
||||
# https://huggingface.co/sesame/csm-1b and supply CSM_HF_TOKEN in .env
|
||||
# before first boot, or the model download 401s.
|
||||
#
|
||||
# Image built locally from the upstream wrapper via buildx git-context.
|
||||
# All tunables live in .env — edit that, not this file.
|
||||
|
||||
services:
|
||||
csm:
|
||||
image: local/csm:${CSM_TAG}
|
||||
build:
|
||||
context: https://github.com/phildougherty/sesame_csm_openai.git#${CSM_SHA}
|
||||
dockerfile: Dockerfile
|
||||
container_name: csm
|
||||
restart: unless-stopped
|
||||
runtime: nvidia
|
||||
ports:
|
||||
- "${CSM_BIND:-0.0.0.0}:${CSM_PORT}:8000"
|
||||
environment:
|
||||
- NVIDIA_VISIBLE_DEVICES=${CSM_GPU_DEVICES:-0}
|
||||
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
||||
# Required: gated-model access token (accept the csm-1b license on
|
||||
# HF first). Set the real value in .env, NEVER here.
|
||||
- HF_TOKEN=${CSM_HF_TOKEN}
|
||||
- HF_HUB_ENABLE_HF_TRANSFER=1
|
||||
- HF_HOME=/app/hf_cache
|
||||
# Optional multi-GPU split: auto | balanced | sequential.
|
||||
- CSM_DEVICE_MAP=${CSM_DEVICE_MAP:-}
|
||||
volumes:
|
||||
- ${CSM_VOICES_DIR}:/app/voices
|
||||
- ${CSM_CACHE_DIR}:/app/hf_cache
|
||||
healthcheck:
|
||||
# The wrapper exposes no /health route; GET /v1/audio/voices
|
||||
# returns the voice list only once the model is loaded, so it
|
||||
# doubles as liveness + ready. python urllib (no curl in image),
|
||||
# bound to 127.0.0.1 (uvicorn is IPv4-only).
|
||||
test: ["CMD-SHELL", "python3 -c \"import urllib.request,sys; urllib.request.urlopen('http://127.0.0.1:8000/v1/audio/voices', timeout=5); sys.exit(0)\""]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
# First boot pulls csm-1b + the Llama-3.2-1B tokenizer + Mimi —
|
||||
# gated download, generous deadline.
|
||||
start_period: 600s
|
||||
labels:
|
||||
- homepage.group=AI Systems
|
||||
- homepage.name=Sesame CSM
|
||||
- homepage.icon=mdi-account-voice-outline
|
||||
- homepage.description=Conversational speech model — context-aware voice-agent TTS (irv-ml1)
|
||||
- homepage.href=http://10.100.79.3:${CSM_PORT}
|
||||
Reference in New Issue
Block a user