vh
01c5380059
The Shadowfita FastAPI wrapper hit two unfixed upstream bugs on the first real /transcribe call — chunker return-shape mismatch (open issue #16) and a `torchaudio.tensor` that doesn't exist (open #10). Rather than babysit someone else's half-tested code, switched to sherpa-onnx with the prebuilt int8 Parakeet-TDT tarball from k2-fsa, and wrote our own ~60-line FastAPI wrapper. Moving parts now owned in-tree: Dockerfile CUDA 12.8 + cuDNN 9 runtime base, installs sherpa-onnx==1.12.39+cuda12.cudnn9 + fastapi + soundfile + libasound2 (sherpa-onnx links to ALSA at load time even when we never touch a mic). app.py OfflineRecognizer.from_transducer() once at startup; /transcribe and /v1/audio/transcriptions both accept multipart uploads and return {"text": ...}. entrypoint.sh Idempotent model download to /models on first run (~400 MB int8 tarball), then exec uvicorn. Smoke test: 0.wav (bundled in the tarball, The House of the Seven Gables excerpt) transcribes cleanly in ~1.2s on GPU. PARAKEET_MODEL_URL in .env lets you swap to the v3 (25-language) tarball without touching any other files. Wipe *.onnx + tokens.txt from the models dir and the entrypoint re-downloads.
115 lines
3.9 KiB
Markdown
115 lines
3.9 KiB
Markdown
# Parakeet ASR
|
||
|
||
NVIDIA Parakeet-TDT 0.6B (int8 ONNX) served by our own thin FastAPI
|
||
wrapper over [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)
|
||
(ONNX Runtime + CUDA).
|
||
|
||
**Server:** irv-ml1 (Irvine, WireGuard-only)
|
||
**Port:** 8765 (container 8000)
|
||
**GPU:** both exposed (`NVIDIA_VISIBLE_DEVICES=all`); sherpa-onnx uses
|
||
whichever CUDA ExecutionProvider picks
|
||
**Image:** `local/parakeet:sherpa-onnx-v1` — built from `Dockerfile` +
|
||
`app.py` + `entrypoint.sh` in this directory; **we own all the code**
|
||
|
||
## Why not the FastAPI community wrappers
|
||
|
||
Both `Shadowfita/parakeet-tdt-0.6b-v2-fastapi` and
|
||
`pnivek/Parakeet-ASR-FastAPI` look appealing on paper but have open,
|
||
unfixed bugs in the actual transcribe path (return-shape mismatches
|
||
after an unpinned `torchaudio` upgrade, `torchaudio.tensor` which
|
||
doesn't exist, etc.). We tried Shadowfita and hit #16+#10 on the
|
||
first real request. Rather than babysit someone else's half-tested
|
||
code, we moved to sherpa-onnx — ONNX Runtime is a stable base, k2-fsa
|
||
publishes prebuilt int8 Parakeet weights per release, and the
|
||
recognizer API is a three-line call.
|
||
|
||
## API endpoints
|
||
|
||
| Method + path | Purpose |
|
||
|---|---|
|
||
| `POST /transcribe` | Multipart file upload → `{"text": "..."}` |
|
||
| `POST /v1/audio/transcriptions` | Same body; OpenAI-compatible path |
|
||
| `GET /healthz` | Health probe (used by docker healthcheck) |
|
||
|
||
## Path layout
|
||
|
||
| Host path | Container path | Purpose | Restic? |
|
||
|---|---|---|---|
|
||
| `/worktank/parakeet/models/` | `/models` | ONNX encoder+decoder+joiner+tokens (~400 MB int8) | excluded (regenerable — re-downloads from the URL on first run if absent) |
|
||
|
||
## First-time deploy on irv-ml1
|
||
|
||
```bash
|
||
# 1. Push compose + Dockerfile + app + entrypoint
|
||
scripts/deploy-stack.sh irv-ml1 parakeet
|
||
|
||
# 2. Make sure the models dir exists (one-time, already done from the
|
||
# earlier Shadowfita deploy; this is idempotent)
|
||
ssh -t irv-ml1 'sudo mkdir -p /worktank/parakeet/models && \
|
||
sudo chown -R lkraven:lkraven /worktank/parakeet'
|
||
|
||
# 3. Build the image and bring up. First boot does a ~400 MB model
|
||
# download via the entrypoint; allow 1–2 minutes before /healthz
|
||
# flips healthy.
|
||
ssh irv-ml1 '
|
||
cd /opt/docker/compose/parakeet && \
|
||
cp -n .env.example .env && \
|
||
docker compose config >/dev/null && \
|
||
docker compose build && \
|
||
docker compose up -d && \
|
||
docker compose logs -f --tail=30
|
||
'
|
||
```
|
||
|
||
## Smoke test
|
||
|
||
```bash
|
||
# Over WG from the workstation
|
||
curl -F "file=@sample.wav" http://10.100.79.3:8765/transcribe
|
||
# → {"text": "hello world"}
|
||
|
||
# OpenAI-shape alias (for clients that only know /v1/audio/transcriptions)
|
||
curl -F "file=@sample.wav" http://10.100.79.3:8765/v1/audio/transcriptions
|
||
```
|
||
|
||
## Switching to the v3 (multilingual) model
|
||
|
||
The env var `PARAKEET_MODEL_URL` picks the release tarball. To swap
|
||
from the English-only v2 to the 25-language v3:
|
||
|
||
```bash
|
||
ssh irv-ml1 '
|
||
cd /opt/docker/compose/parakeet && \
|
||
sed -i "s|v2-int8|v3-int8|" .env && \
|
||
# Wipe the v2 weights so the entrypoint re-downloads v3 on next up:
|
||
rm -f /worktank/parakeet/models/*.onnx /worktank/parakeet/models/tokens.txt && \
|
||
docker compose up -d && \
|
||
docker compose logs -f --tail=30
|
||
'
|
||
```
|
||
|
||
## Upgrade sherpa-onnx or change the base image
|
||
|
||
Bump `PARAKEET_TAG` in `.env` to force a rebuild of the local image
|
||
after editing the `Dockerfile`, then:
|
||
|
||
```bash
|
||
scripts/deploy-stack.sh irv-ml1 parakeet
|
||
ssh irv-ml1 'cd /opt/docker/compose/parakeet && docker compose build && docker compose up -d'
|
||
```
|
||
|
||
Model files under `/worktank/parakeet/models/` are preserved across
|
||
image rebuilds.
|
||
|
||
## File layout
|
||
|
||
```
|
||
stacks/parakeet/
|
||
├── Dockerfile # CUDA 12.8 + cuDNN 9 base, sherpa-onnx-cu12 wheel
|
||
├── app.py # FastAPI — ~60 lines
|
||
├── entrypoint.sh # downloads model on first run, then uvicorn
|
||
├── compose.yaml # one service, bind-mounts the models dir
|
||
├── .env.example # template; real .env lives on the server
|
||
└── README.md # this file
|
||
```
|