esh-pfi-infrastructure/stacks/parakeet

Parakeet ASR

NVIDIA Parakeet-TDT 0.6B (int8 ONNX) served by our own thin FastAPI wrapper over sherpa-onnx (ONNX Runtime + CUDA).

Server: irv-ml1 (Irvine, WireGuard-only) Port: 8765 (container 8000) GPU: both exposed (NVIDIA_VISIBLE_DEVICES=all); sherpa-onnx uses whichever CUDA ExecutionProvider picks Image: local/parakeet:sherpa-onnx-v1 — built from Dockerfile + app.py + entrypoint.sh in this directory; we own all the code

Why not the FastAPI community wrappers

Both Shadowfita/parakeet-tdt-0.6b-v2-fastapi and pnivek/Parakeet-ASR-FastAPI look appealing on paper but have open, unfixed bugs in the actual transcribe path (return-shape mismatches after an unpinned torchaudio upgrade, torchaudio.tensor which doesn't exist, etc.). We tried Shadowfita and hit #16+#10 on the first real request. Rather than babysit someone else's half-tested code, we moved to sherpa-onnx — ONNX Runtime is a stable base, k2-fsa publishes prebuilt int8 Parakeet weights per release, and the recognizer API is a three-line call.

API endpoints

Method + path	Purpose
POST /transcribe	Multipart file upload → {"text": "..."}
POST /v1/audio/transcriptions	Same body; OpenAI-compatible path
GET /healthz	Health probe (used by docker healthcheck)

Path layout

Host path	Container path	Purpose	Restic?
/worktank/parakeet/models/	/models	ONNX encoder+decoder+joiner+tokens (~400 MB int8)	excluded (regenerable — re-downloads from the URL on first run if absent)

First-time deploy on irv-ml1

# 1. Push compose + Dockerfile + app + entrypoint
scripts/deploy-stack.sh irv-ml1 parakeet

# 2. Make sure the models dir exists (one-time, already done from the
#    earlier Shadowfita deploy; this is idempotent)
ssh -t irv-ml1 'sudo mkdir -p /worktank/parakeet/models && \
                sudo chown -R lkraven:lkraven /worktank/parakeet'

# 3. Build the image and bring up. First boot does a ~400 MB model
#    download via the entrypoint; allow 1–2 minutes before /healthz
#    flips healthy.
ssh irv-ml1 '
  cd /opt/docker/compose/parakeet && \
  cp -n .env.example .env && \
  docker compose config >/dev/null && \
  docker compose build && \
  docker compose up -d && \
  docker compose logs -f --tail=30
'

Smoke test

# Over WG from the workstation
curl -F "file=@sample.wav" http://10.100.79.3:8765/transcribe
# → {"text": "hello world"}

# OpenAI-shape alias (for clients that only know /v1/audio/transcriptions)
curl -F "file=@sample.wav" http://10.100.79.3:8765/v1/audio/transcriptions

Switching to the v3 (multilingual) model

The env var PARAKEET_MODEL_URL picks the release tarball. To swap from the English-only v2 to the 25-language v3:

ssh irv-ml1 '
  cd /opt/docker/compose/parakeet && \
  sed -i "s|v2-int8|v3-int8|" .env && \
  # Wipe the v2 weights so the entrypoint re-downloads v3 on next up:
  rm -f /worktank/parakeet/models/*.onnx /worktank/parakeet/models/tokens.txt && \
  docker compose up -d && \
  docker compose logs -f --tail=30
'

Upgrade sherpa-onnx or change the base image

Bump PARAKEET_TAG in .env to force a rebuild of the local image after editing the Dockerfile, then:

scripts/deploy-stack.sh irv-ml1 parakeet
ssh irv-ml1 'cd /opt/docker/compose/parakeet && docker compose build && docker compose up -d'

Model files under /worktank/parakeet/models/ are preserved across image rebuilds.

File layout

stacks/parakeet/
├── Dockerfile           # CUDA 12.8 + cuDNN 9 base, sherpa-onnx-cu12 wheel
├── app.py               # FastAPI — ~60 lines
├── entrypoint.sh        # downloads model on first run, then uvicorn
├── compose.yaml         # one service, bind-mounts the models dir
├── .env.example         # template; real .env lives on the server
└── README.md            # this file