vh
0655a37bf6
- max_tokens default 2400->3500 (~42s) in wrapper + gateway-chat client, with a _cap() clamp so prompt+gen never exceeds MAX_CTX (4096) — a cloning ref block is ~1100 tokens, so an unclamped 3500 would overflow context on the clone path. - Staged clone voices: /voices dir of <name>.wav + <name>.txt, each encoded to its Orpheus reference block at startup; voice="<name>" zero-shot clones it. Beatrice (a chatterbox reference) staged as the first normal-voice clone. GET /voices lists baddy + clones. - compose: mount voices dir + pass MORPHEUS_MAX_LEN to the wrapper (clamp must match engine). vLLM concurrency (measured, --max-num-seqs 8, 250-tok reqs): near-linear batching — 8 concurrent finish in the same ~2.8s as 1 (707 tok/s, 8.1x single, flat per-req latency). Chunked-sentence production can fan out for ~8x throughput; CPU SNAC decode is the scale bottleneck, not generation.
61 lines
3.3 KiB
Markdown
61 lines
3.3 KiB
Markdown
# mOrpheus — uncensored Orpheus TTS (irv-ml1)
|
||
|
||
Permanent serving stack for **`MrDragonFox/mOrpheus_3B-1Base_early_preview-v1-25000`** — an
|
||
uncensored Orpheus TTS finetune (Llama-3.2-3B LLM → SNAC 24 kHz audio). Trained speaker
|
||
**"baddy"**; supports **zero-shot voice cloning** from a reference clip.
|
||
|
||
> **INTERNAL RESEARCH ONLY.** License is **CC-BY-NC-4.0** (non-commercial). Do **not** expose
|
||
> this endpoint externally or use it in any commercial-facing product.
|
||
|
||
## Shape
|
||
|
||
Two containers (see `compose.yaml`):
|
||
|
||
| service | where | role |
|
||
|---|---|---|
|
||
| `vllm-morpheus` | GPU (3090), FP8 | serves the mOrpheus LLM; emits Orpheus audio tokens |
|
||
| `morpheus-tts` | CPU | SNAC-decodes tokens → 24 kHz WAV; the public `/tts` endpoint |
|
||
|
||
**Real-time:** ~165 tok/s single-stream on the 3090 (FP8) ⇒ **RTF ≈ 0.50 (2× real-time)**,
|
||
measured. A ~4 s clip generates in ~2 s. (Whole-clip decode in v1; chunked streaming for
|
||
lower time-to-first-audio is a future enhancement.)
|
||
|
||
## Endpoints (`http://10.100.79.3:8299`)
|
||
|
||
- `POST /tts` → `audio/wav`. Body: `{"text": "...", "voice": "baddy", "temperature": 0.6,
|
||
"max_tokens": 1200, "repetition_penalty": 1.1}`.
|
||
- **Zero-shot clone (ad-hoc):** add `"reference_audio_b64": "<base64 WAV>"` + `"reference_text":
|
||
"<its transcript>"`. Keep `repetition_penalty <= 1.1` for cloning (higher penalizes the
|
||
in-context reference audio tokens and breaks generation).
|
||
- **Staged clone voices:** drop `<name>.wav` + `<name>.txt` (its transcript) into the voices
|
||
dir (`/home/lkraven/morpheus/voices/`); each is encoded to its reference block once at
|
||
startup, so `voice: "<name>"` zero-shot clones it (e.g. `beatrice`). `GET /voices` lists them.
|
||
- `max_tokens` defaults to 3500 (~42 s), auto-clamped so prompt + gen never exceeds the
|
||
4096 context (a cloning reference block is ~1,100 tokens). `repetition_penalty` 1.1 is
|
||
load-bearing — at 1.0 the model never emits end-of-speech and rambles to the cap.
|
||
- `GET /voices`, `GET /health`, `GET /docs` (OpenAPI UI).
|
||
|
||
**Expressive tags** (baddy is trained for these): `<sigh> <gasp> <laugh> <chuckle> <pant>
|
||
<groan> <moan>` etc. Use **real carrier sentences with sparse, sentence-boundary tags** —
|
||
stacking many tags with little text sends this early checkpoint into a repeat-loop.
|
||
|
||
## Deploy (irv-ml1, as `lkraven` — docker-group, no sudo)
|
||
|
||
```bash
|
||
# one-time: stage weights (from the audition dir or a fresh pull-hf-repo) + copy the stack
|
||
mkdir -p /home/lkraven/morpheus/models
|
||
mv /home/lkraven/orpheus-audition/models/mOrpheus /home/lkraven/morpheus/models/
|
||
mv /home/lkraven/orpheus-audition/models/snac_24khz /home/lkraven/morpheus/models/
|
||
# copy compose.yaml + tts/ to /home/lkraven/morpheus/, cp .env.example .env
|
||
cd /home/lkraven/morpheus && docker compose build && docker compose up -d
|
||
```
|
||
|
||
## Gotchas
|
||
|
||
- **Pin `vllm/vllm-openai:v0.23.0`** — `latest` ships Blackwell-only kernels (oink/aiter)
|
||
that crash on Ampere *import*. Do not bump to `latest` on this box.
|
||
- **GPU = 3090, not the A6000** — the A6000 is comfy's and spikes to ~41 GB without warning
|
||
(OOM'd two launches). FP8's ~5 GB footprint coexists with the 3090 audio zoo.
|
||
- **FP8** on Ampere is a VRAM save (upcast), no compute speedup — real-time comes from vLLM.
|
||
- Canonical copy lives here; deployed copy is `/home/lkraven/morpheus/` on irv-ml1.
|