Files

T

vh 01c5380059 parakeet: rewrite on sherpa-onnx; own the wrapper end-to-end

The Shadowfita FastAPI wrapper hit two unfixed upstream bugs on the
first real /transcribe call — chunker return-shape mismatch (open
issue #16) and a `torchaudio.tensor` that doesn't exist (open #10).
Rather than babysit someone else's half-tested code, switched to
sherpa-onnx with the prebuilt int8 Parakeet-TDT tarball from k2-fsa,
and wrote our own ~60-line FastAPI wrapper.

Moving parts now owned in-tree:
  Dockerfile      CUDA 12.8 + cuDNN 9 runtime base, installs
                  sherpa-onnx==1.12.39+cuda12.cudnn9 + fastapi +
                  soundfile + libasound2 (sherpa-onnx links to ALSA
                  at load time even when we never touch a mic).
  app.py          OfflineRecognizer.from_transducer() once at startup;
                  /transcribe and /v1/audio/transcriptions both accept
                  multipart uploads and return {"text": ...}.
  entrypoint.sh   Idempotent model download to /models on first run
                  (~400 MB int8 tarball), then exec uvicorn.

Smoke test: 0.wav (bundled in the tarball, The House of the Seven
Gables excerpt) transcribes cleanly in ~1.2s on GPU.

PARAKEET_MODEL_URL in .env lets you swap to the v3 (25-language)
tarball without touching any other files. Wipe *.onnx + tokens.txt
from the models dir and the entrypoint re-downloads.

2026-04-24 00:18:45 -07:00

3.9 KiB

Raw Blame History

Parakeet ASR

NVIDIA Parakeet-TDT 0.6B (int8 ONNX) served by our own thin FastAPI wrapper over sherpa-onnx (ONNX Runtime + CUDA).

Server: irv-ml1 (Irvine, WireGuard-only) Port: 8765 (container 8000) GPU: both exposed (NVIDIA_VISIBLE_DEVICES=all); sherpa-onnx uses whichever CUDA ExecutionProvider picks Image: local/parakeet:sherpa-onnx-v1 — built from Dockerfile + app.py + entrypoint.sh in this directory; we own all the code

Why not the FastAPI community wrappers

Both Shadowfita/parakeet-tdt-0.6b-v2-fastapi and pnivek/Parakeet-ASR-FastAPI look appealing on paper but have open, unfixed bugs in the actual transcribe path (return-shape mismatches after an unpinned torchaudio upgrade, torchaudio.tensor which doesn't exist, etc.). We tried Shadowfita and hit #16+#10 on the first real request. Rather than babysit someone else's half-tested code, we moved to sherpa-onnx — ONNX Runtime is a stable base, k2-fsa publishes prebuilt int8 Parakeet weights per release, and the recognizer API is a three-line call.

API endpoints

Method + path	Purpose
`POST /transcribe`	Multipart file upload → `{"text": "..."}`
`POST /v1/audio/transcriptions`	Same body; OpenAI-compatible path
`GET /healthz`	Health probe (used by docker healthcheck)

Path layout

Host path	Container path	Purpose	Restic?
`/worktank/parakeet/models/`	`/models`	ONNX encoder+decoder+joiner+tokens (~400 MB int8)	excluded (regenerable — re-downloads from the URL on first run if absent)

First-time deploy on irv-ml1

# 1. Push compose + Dockerfile + app + entrypoint
scripts/deploy-stack.sh irv-ml1 parakeet

# 2. Make sure the models dir exists (one-time, already done from the
#    earlier Shadowfita deploy; this is idempotent)
ssh -t irv-ml1 'sudo mkdir -p /worktank/parakeet/models && \
                sudo chown -R lkraven:lkraven /worktank/parakeet'

# 3. Build the image and bring up. First boot does a ~400 MB model
#    download via the entrypoint; allow 1–2 minutes before /healthz
#    flips healthy.
ssh irv-ml1 '
  cd /opt/docker/compose/parakeet && \
  cp -n .env.example .env && \
  docker compose config >/dev/null && \
  docker compose build && \
  docker compose up -d && \
  docker compose logs -f --tail=30
'

Smoke test

# Over WG from the workstation
curl -F "file=@sample.wav" http://10.100.79.3:8765/transcribe
# → {"text": "hello world"}

# OpenAI-shape alias (for clients that only know /v1/audio/transcriptions)
curl -F "file=@sample.wav" http://10.100.79.3:8765/v1/audio/transcriptions

Switching to the v3 (multilingual) model

The env var PARAKEET_MODEL_URL picks the release tarball. To swap from the English-only v2 to the 25-language v3:

ssh irv-ml1 '
  cd /opt/docker/compose/parakeet && \
  sed -i "s|v2-int8|v3-int8|" .env && \
  # Wipe the v2 weights so the entrypoint re-downloads v3 on next up:
  rm -f /worktank/parakeet/models/*.onnx /worktank/parakeet/models/tokens.txt && \
  docker compose up -d && \
  docker compose logs -f --tail=30
'

Upgrade sherpa-onnx or change the base image

Bump PARAKEET_TAG in .env to force a rebuild of the local image after editing the Dockerfile, then:

scripts/deploy-stack.sh irv-ml1 parakeet
ssh irv-ml1 'cd /opt/docker/compose/parakeet && docker compose build && docker compose up -d'

Model files under /worktank/parakeet/models/ are preserved across image rebuilds.

File layout

stacks/parakeet/
├── Dockerfile           # CUDA 12.8 + cuDNN 9 base, sherpa-onnx-cu12 wheel
├── app.py               # FastAPI — ~60 lines
├── entrypoint.sh        # downloads model on first run, then uvicorn
├── compose.yaml         # one service, bind-mounts the models dir
├── .env.example         # template; real .env lives on the server
└── README.md            # this file

3.9 KiB Raw Blame History Unescape Escape