vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

parakeet: rewrite on sherpa-onnx; own the wrapper end-to-end

Browse Source 
The Shadowfita FastAPI wrapper hit two unfixed upstream bugs on the
first real /transcribe call — chunker return-shape mismatch (open
issue #16) and a `torchaudio.tensor` that doesn't exist (open #10).
Rather than babysit someone else's half-tested code, switched to
sherpa-onnx with the prebuilt int8 Parakeet-TDT tarball from k2-fsa,
and wrote our own ~60-line FastAPI wrapper.

Moving parts now owned in-tree:
  Dockerfile      CUDA 12.8 + cuDNN 9 runtime base, installs
                  sherpa-onnx==1.12.39+cuda12.cudnn9 + fastapi +
                  soundfile + libasound2 (sherpa-onnx links to ALSA
                  at load time even when we never touch a mic).
  app.py          OfflineRecognizer.from_transducer() once at startup;
                  /transcribe and /v1/audio/transcriptions both accept
                  multipart uploads and return {"text": ...}.
  entrypoint.sh   Idempotent model download to /models on first run
                  (~400 MB int8 tarball), then exec uvicorn.

Smoke test: 0.wav (bundled in the tarball, The House of the Seven
Gables excerpt) transcribes cleanly in ~1.2s on GPU.

PARAKEET_MODEL_URL in .env lets you swap to the v3 (25-language)
tarball without touching any other files. Wipe *.onnx + tokens.txt
from the models dir and the entrypoint re-downloads.
This commit is contained in:
vh
2026-04-24 00:18:45 -07:00
parent 82f95d7428
commit 01c5380059
6 changed files with 272 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/parakeet/README.md
+62 -36
View File
@@ -1,51 +1,56 @@
# Parakeet ASR
NVIDIA Parakeet-TDT 0.6B v2 speech-to-text served via a FastAPI
wrapper with Silero VAD and WebSocket streaming.
NVIDIA Parakeet-TDT 0.6B (int8 ONNX) served by our own thin FastAPI
wrapper over [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)
(ONNX Runtime + CUDA).
**Server:** irv-ml1 (Irvine, WireGuard-only)
**Port:** 8765 (configurable via `.env`)
**GPUs:** both exposed (`NVIDIA_VISIBLE_DEVICES=all`); upstream
respects `CUDA_VISIBLE_DEVICES` if later pinning is needed
**Upstream:** [Shadowfita/parakeet-tdt-0.6b-v2-fastapi](https://github.com/Shadowfita/parakeet-tdt-0.6b-v2-fastapi)
**Image:** built locally from a pinned git SHA via docker buildx's
git URL context — no source vendored into this workspace
**Port:** 8765 (container 8000)
**GPU:** both exposed (`NVIDIA_VISIBLE_DEVICES=all`); sherpa-onnx uses
whichever CUDA ExecutionProvider picks
**Image:** `local/parakeet:sherpa-onnx-v1` — built from `Dockerfile` +
`app.py` + `entrypoint.sh` in this directory; **we own all the code**
## Why not the FastAPI community wrappers
Both `Shadowfita/parakeet-tdt-0.6b-v2-fastapi` and
`pnivek/Parakeet-ASR-FastAPI` look appealing on paper but have open,
unfixed bugs in the actual transcribe path (return-shape mismatches
after an unpinned `torchaudio` upgrade, `torchaudio.tensor` which
doesn't exist, etc.). We tried Shadowfita and hit #16+#10 on the
first real request. Rather than babysit someone else's half-tested
code, we moved to sherpa-onnx — ONNX Runtime is a stable base, k2-fsa
publishes prebuilt int8 Parakeet weights per release, and the
recognizer API is a three-line call.
## API endpoints
| Method + path | Purpose |
|---|---|
| `POST /transcribe` | Batch transcription (multipart file upload) |
| `WS /ws/transcribe` | Streaming with Silero VAD — partial + final segments |
| `POST /transcribe` | Multipart file upload → `{"text": "..."}` |
| `POST /v1/audio/transcriptions` | Same body; OpenAI-compatible path |
| `GET /healthz` | Health probe (used by docker healthcheck) |
Not the literal OpenAI `/v1/audio/transcriptions` path. If you have
a downstream client that demands that URL shape, either point its
base URL at `/transcribe`, or add a Traefik/nginx path alias in
front.
## Path layout
| Host path | Container path | Purpose | Restic? |
|---|---|---|---|
| `/worktank/parakeet/models/` | `/models` (`HF_HOME`) | HF cache for parakeet-tdt-0.6b-v2 weights (~2.5 GB) | excluded (regenerable from HF) |
The container runs as root internally; host dir just needs to exist
and be writable.
| `/worktank/parakeet/models/` | `/models` | ONNX encoder+decoder+joiner+tokens (~400 MB int8) | excluded (regenerable — re-downloads from the URL on first run if absent) |
## First-time deploy on irv-ml1
```bash
# 1. Push compose + env template
# 1. Push compose + Dockerfile + app + entrypoint
scripts/deploy-stack.sh irv-ml1 parakeet
# 2. Create the models dir on the host. One-time sudo — /worktank
# itself is root-owned.
# 2. Make sure the models dir exists (one-time, already done from the
# earlier Shadowfita deploy; this is idempotent)
ssh -t irv-ml1 'sudo mkdir -p /worktank/parakeet/models && \
sudo chown -R lkraven:lkraven /worktank/parakeet'
# 3. Build the image (first time only; ~5–10 min for torch + NeMo
# wheels). Then `up`.
# 3. Build the image and bring up. First boot does a ~400 MB model
# download via the entrypoint; allow 1–2 minutes before /healthz
# flips healthy.
ssh irv-ml1 '
cd /opt/docker/compose/parakeet && \
cp -n .env.example .env && \
@@ -56,33 +61,54 @@ ssh irv-ml1 '
'
```
First `/transcribe` request downloads parakeet-tdt-0.6b-v2 weights
to `/worktank/parakeet/models/` (~2.5 GB).
## Smoke test
```bash
# From the workstation over WG
# Over WG from the workstation
curl -F "file=@sample.wav" http://10.100.79.3:8765/transcribe
# → {"text": "hello world"}
# OpenAI-shape alias (for clients that only know /v1/audio/transcriptions)
curl -F "file=@sample.wav" http://10.100.79.3:8765/v1/audio/transcriptions
```
## Rebuild against a newer upstream commit
## Switching to the v3 (multilingual) model
The env var `PARAKEET_MODEL_URL` picks the release tarball. To swap
from the English-only v2 to the 25-language v3:
```bash
ssh irv-ml1 '
cd /opt/docker/compose/parakeet && \
sed -i "s/^PARAKEET_SHA=.*/PARAKEET_SHA=<new-sha>/" .env && \
docker compose build && \
docker compose up -d
sed -i "s|v2-int8|v3-int8|" .env && \
# Wipe the v2 weights so the entrypoint re-downloads v3 on next up:
rm -f /worktank/parakeet/models/*.onnx /worktank/parakeet/models/tokens.txt && \
docker compose up -d && \
docker compose logs -f --tail=30
'
```
Models dir is unaffected.
## Upgrade sherpa-onnx or change the base image
## Deploy updates
Bump `PARAKEET_TAG` in `.env` to force a rebuild of the local image
after editing the `Dockerfile`, then:
```bash
# After editing compose.yaml or .env.example here
scripts/deploy-stack.sh irv-ml1 parakeet
ssh irv-ml1 'cd /opt/docker/compose/parakeet && docker compose up -d'
ssh irv-ml1 'cd /opt/docker/compose/parakeet && docker compose build && docker compose up -d'
```
Model files under `/worktank/parakeet/models/` are preserved across
image rebuilds.
## File layout
```
stacks/parakeet/
├── Dockerfile # CUDA 12.8 + cuDNN 9 base, sherpa-onnx-cu12 wheel
├── app.py # FastAPI — ~60 lines
├── entrypoint.sh # downloads model on first run, then uvicorn
├── compose.yaml # one service, bind-mounts the models dir
├── .env.example # template; real .env lives on the server
└── README.md # this file
```