vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

parakeet + cosyvoice: add stacks + deploy to irv-ml1

Browse Source 
Two new speech stacks on irv-ml1, both on the /worktank/<stack>/
pattern, no tnet (irv-ml1 is local-endpoints-only for now).

parakeet — ASR via Shadowfita/parakeet-tdt-0.6b-v2-fastapi:
  - docker buildx git context pinned to SHA 31c5652; no source
    vendored. Rebuild on SHA bump.
  - GPU-capable FastAPI + Silero VAD + WS streaming.
  - API: POST /transcribe, WS /ws/transcribe, GET /healthz. Not the
    literal OpenAI `/v1/audio/transcriptions` path — note in README.
  - HF cache at /worktank/parakeet/models/ (excluded from restic).
  - Build ~158s first time; steady-state start ~40s.

cosyvoice — TTS via neosun/cosyvoice:v1.3.2 shipping
Fun-CosyVoice3-0.5B-2512 (CosyVoice 3, chosen over v2 for the
expanded 5,000-hour instruction-following data covering emotions,
speed, tones, dialects, accents, role-playing; ~150ms streaming
TTFB matches v2). API: /v1/audio/speech (OpenAI drop-in),
/v1/voices/create (cloning), /health.
  - Host port 8190 (container 8188; host 8188 already taken by comfyui).
  - /worktank/cosyvoice/{voices,input,output}/; voices include in
    restic (precious — reproducing a clone needs the original ref
    audio), input+output excluded (scratch).
  - Model weights (~2-3 GB) live inside image layer; re-download on
    tag bump, persist across `compose up -d`.

Both healthy on first deploy.
This commit is contained in:
vh
2026-04-23 23:40:36 -07:00
parent 06476745f1
commit 1a67370138
6 changed files with 410 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/parakeet/.env.example
+49
View File
@@ -0,0 +1,49 @@
# Parakeet ASR stack tunables. Copy to `.env` on irv-ml1 before deploying.
#
# cp .env.example .env
# # edit as needed
# docker compose build
# docker compose up -d
# Pinned git SHA to build from. Bump + rebuild when you want upstream
# fixes. `main` latest as of 2026-04:
# https://github.com/Shadowfita/parakeet-tdt-0.6b-v2-fastapi
PARAKEET_SHA=31c5652b62d09653ad5ea8190c0ad0d35394174d
# Host port for the FastAPI server (container listens on 8000)
PARAKEET_PORT=8765
# Bind address. 0.0.0.0 exposes on all interfaces including the WG
# tunnel IP (10.100.79.3). Use 127.0.0.1 to restrict to local-only.
PARAKEET_BIND=0.0.0.0
# Host path for the HuggingFace cache (parakeet-tdt-0.6b-v2 weights
# ~2.5 GB). Persistent across container recreates. Must exist before
# first `up` with ownership matching the container user (root inside
# this image — no UID juggling needed, but the host dir needs to be
# writable by the container).
PARAKEET_MODELS_DIR=/worktank/parakeet/models
# Inference precision. fp16 halves VRAM and is lossless for parakeet
# in practice; use fp32 only if fp16 shows degraded WER for your
# domain audio.
PARAKEET_MODEL_PRECISION=fp16
# Batch size for the transcribe queue. Larger = better throughput
# under load at the cost of per-request latency.
PARAKEET_BATCH_SIZE=4
# Max single-clip duration (seconds). Longer inputs get rejected
# by the server with 400. Upstream default.
PARAKEET_MAX_AUDIO_DURATION=30
# Silero VAD threshold (0–1). Higher = stricter about what counts
# as speech (fewer false wake-ups on silence, more chance of clipping
# soft speech). 0.5 is upstream default.
PARAKEET_VAD_THRESHOLD=0.5
# End-to-end processing timeout per request (seconds).
PARAKEET_PROCESSING_TIMEOUT=60
# Log level: DEBUG | INFO | WARNING | ERROR
PARAKEET_LOG_LEVEL=INFO
stacks/parakeet/README.md
+88
View File
@@ -0,0 +1,88 @@
# Parakeet ASR
NVIDIA Parakeet-TDT 0.6B v2 speech-to-text served via a FastAPI
wrapper with Silero VAD and WebSocket streaming.
**Server:** irv-ml1 (Irvine, WireGuard-only)
**Port:** 8765 (configurable via `.env`)
**GPUs:** both exposed (`NVIDIA_VISIBLE_DEVICES=all`); upstream
respects `CUDA_VISIBLE_DEVICES` if later pinning is needed
**Upstream:** [Shadowfita/parakeet-tdt-0.6b-v2-fastapi](https://github.com/Shadowfita/parakeet-tdt-0.6b-v2-fastapi)
**Image:** built locally from a pinned git SHA via docker buildx's
git URL context — no source vendored into this workspace
## API endpoints
| Method + path | Purpose |
|---|---|
| `POST /transcribe` | Batch transcription (multipart file upload) |
| `WS /ws/transcribe` | Streaming with Silero VAD — partial + final segments |
| `GET /healthz` | Health probe (used by docker healthcheck) |
Not the literal OpenAI `/v1/audio/transcriptions` path. If you have
a downstream client that demands that URL shape, either point its
base URL at `/transcribe`, or add a Traefik/nginx path alias in
front.
## Path layout
| Host path | Container path | Purpose | Restic? |
|---|---|---|---|
| `/worktank/parakeet/models/` | `/models` (`HF_HOME`) | HF cache for parakeet-tdt-0.6b-v2 weights (~2.5 GB) | excluded (regenerable from HF) |
The container runs as root internally; host dir just needs to exist
and be writable.
## First-time deploy on irv-ml1
```bash
# 1. Push compose + env template
scripts/deploy-stack.sh irv-ml1 parakeet
# 2. Create the models dir on the host. One-time sudo — /worktank
# itself is root-owned.
ssh -t irv-ml1 'sudo mkdir -p /worktank/parakeet/models && \
sudo chown -R lkraven:lkraven /worktank/parakeet'
# 3. Build the image (first time only; ~5–10 min for torch + NeMo
# wheels). Then `up`.
ssh irv-ml1 '
cd /opt/docker/compose/parakeet && \
cp -n .env.example .env && \
docker compose config >/dev/null && \
docker compose build && \
docker compose up -d && \
docker compose logs -f --tail=30
'
```
First `/transcribe` request downloads parakeet-tdt-0.6b-v2 weights
to `/worktank/parakeet/models/` (~2.5 GB).
## Smoke test
```bash
# From the workstation over WG
curl -F "file=@sample.wav" http://10.100.79.3:8765/transcribe
```
## Rebuild against a newer upstream commit
```bash
ssh irv-ml1 '
cd /opt/docker/compose/parakeet && \
sed -i "s/^PARAKEET_SHA=.*/PARAKEET_SHA=<new-sha>/" .env && \
docker compose build && \
docker compose up -d
'
```
Models dir is unaffected.
## Deploy updates
```bash
# After editing compose.yaml or .env.example here
scripts/deploy-stack.sh irv-ml1 parakeet
ssh irv-ml1 'cd /opt/docker/compose/parakeet && docker compose up -d'
```
stacks/parakeet/compose.yaml
+67
View File
@@ -0,0 +1,67 @@
# Parakeet ASR — NVIDIA Parakeet-TDT 0.6B v2 speech-to-text.
#
# Wraps Shadowfita/parakeet-tdt-0.6b-v2-fastapi (FastAPI + Silero VAD +
# WebSocket streaming). Upstream provides no prebuilt image, so we
# build from a pinned git commit via docker buildx's git URL context
# — no source files vendored into this repo.
#
# Runs on irv-ml1 (dual GPU). Both GPUs exposed; upstream respects
# CUDA_VISIBLE_DEVICES if you want to pin later.
#
# HF weights (~2.5 GB for parakeet-tdt-0.6b-v2) cache to
# ${PARAKEET_MODELS_DIR} via HF_HOME=/models, persistent across
# container recreates.
#
# API routes (per upstream README):
# POST /transcribe — batch transcription
# WS /ws/transcribe — streaming with Silero VAD
# GET /healthz
#
# Not a literal OpenAI `/v1/audio/transcriptions` path; point clients
# at /transcribe directly, or add a reverse-proxy alias if drop-in
# compat is needed later.
#
# First `up` triggers a fresh docker build (python:3.10-slim +
# torch + NeMo ≈ 5–10 min). Subsequent starts reuse the cached
# image unless PARAKEET_SHA changes.
#
# All tunables live in .env — edit that, not this file.
services:
parakeet:
image: local/parakeet:${PARAKEET_SHA}
build:
context: https://github.com/Shadowfita/parakeet-tdt-0.6b-v2-fastapi.git#${PARAKEET_SHA}
container_name: parakeet
restart: unless-stopped
runtime: nvidia
ports:
- "${PARAKEET_BIND:-0.0.0.0}:${PARAKEET_PORT}:8000"
environment:
- NVIDIA_VISIBLE_DEVICES=all
- HF_HOME=/models
- DEVICE=cuda
- MODEL_PRECISION=${PARAKEET_MODEL_PRECISION:-fp16}
- BATCH_SIZE=${PARAKEET_BATCH_SIZE:-4}
- TARGET_SAMPLE_RATE=16000
- MAX_AUDIO_DURATION=${PARAKEET_MAX_AUDIO_DURATION:-30}
- VAD_THRESHOLD=${PARAKEET_VAD_THRESHOLD:-0.5}
- PROCESSING_TIMEOUT=${PARAKEET_PROCESSING_TIMEOUT:-60}
- LOG_LEVEL=${PARAKEET_LOG_LEVEL:-INFO}
volumes:
- ${PARAKEET_MODELS_DIR}:/models
healthcheck:
test: ["CMD-SHELL", "curl -fsS http://localhost:8000/healthz >/dev/null || exit 1"]
interval: 30s
timeout: 10s
retries: 3
# First `up` may spend several minutes on torch/NeMo install
# during the image build phase; after the image exists, startup
# is ~30-60s (NeMo model load).
start_period: 180s
labels:
- homepage.group=AI Systems
- homepage.name=Parakeet ASR
- homepage.icon=mdi-microphone
- homepage.description=Parakeet-TDT speech-to-text (irv-ml1)
- homepage.href=http://10.100.79.3:${PARAKEET_PORT}