vh
b9b14b5baf
Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry the vLLM seats at 84-95.5 GB of 96. Changes: - compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used `count: all`, which would have handed a 0.6B ASR seat all four cards); join traefik-net; port 8300; homepage href to the live FV address. - .env.example: default to the v3 int8 model (25 European languages, 464 MiB) rather than English-only v2; models to /tank/parakeet/models. - app.py: warm the recognizer at startup before uvicorn accepts traffic. The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at 45.1s on a second container) against ~0.50s warm. A 45s first request is indistinguishable from a hang and LiteLLM's default timeout abandons it long before it returns. Decoding 1s of silence at load moves the cost inside the healthcheck's 300s start_period; first real request after restart is now 0.65s. Verification, because "provider=cuda" in the log is only an echo of the env var: ORT falls back to CPU silently and still returns correct text, so the service being up and the transcript being right establishes nothing. The discriminator is a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS sentence transcribes near-exactly (positive), 3s of digital silence returns empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread 0.47-0.65s, single-stream, one clip: a smoke measurement with its harness stated, not a benchmark. Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1` (OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's Postgres store where the ext-tts family already lives — no gateway restart, and config.yaml is consequently not a complete picture of what the gateway serves. Both verified end to end. The aliases use a raw IP deliberately: ana-docker resolves no .internal names at all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a hand-pinned extra_hosts entry. A second hosts entry would mean recreating the container and bouncing the gateway for every consumer. Also records the svos_miranda plugin validation pass and its structural findings, and notes that the irv-ml1 parakeet is still running — there are two now, and retiring the old one is the operator's call.
142 lines
6.1 KiB
Markdown
142 lines
6.1 KiB
Markdown
# Parakeet ASR
|
||
|
||
NVIDIA Parakeet-TDT 0.6B (int8 ONNX) served by our own thin FastAPI
|
||
wrapper over [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)
|
||
(ONNX Runtime + CUDA).
|
||
|
||
**Server:** fv-ml1 (Fountain Valley, `10.251.50.54`) — moved from irv-ml1 2026-09-15
|
||
**Port:** 8300 (container 8000)
|
||
**GPU:** **3**, pinned explicitly via `device_ids` — the utility card
|
||
**Image:** `local/parakeet:sherpa-onnx-v4` — built from `Dockerfile` +
|
||
`app.py` + `entrypoint.sh` in this directory; **we own all the code**
|
||
**Model:** `parakeet-tdt-0.6b-v3` int8, 25 European languages (~464 MiB)
|
||
|
||
## Why GPU 3
|
||
|
||
fv-ml1 has four RTX PRO 6000 Blackwell Max-Q (96 GB each). Three carry the
|
||
vLLM serving seats and run 85–98 % full; GPU 3 is the utility card and was
|
||
empty (2 MiB) at placement time. A 0.6 B int8 ASR model is a rounding error
|
||
next to those seats, but it still has to go somewhere that is not fighting
|
||
them for VRAM.
|
||
|
||
⚠ The pin is `deploy.resources.reservations.devices[].device_ids: ["3"]`,
|
||
the fleet convention — **not** `count: all`, which is what the dead on-host
|
||
stub used and which would have handed this seat all four cards. Inside the
|
||
container the pinned card presents as `cuda:0`, which is what sherpa-onnx's
|
||
CUDA execution provider takes by default.
|
||
|
||
## Why not the FastAPI community wrappers
|
||
|
||
Both `Shadowfita/parakeet-tdt-0.6b-v2-fastapi` and
|
||
`pnivek/Parakeet-ASR-FastAPI` look appealing on paper but have open,
|
||
unfixed bugs in the actual transcribe path (return-shape mismatches
|
||
after an unpinned `torchaudio` upgrade, `torchaudio.tensor` which
|
||
doesn't exist, etc.). We tried Shadowfita and hit #16+#10 on the
|
||
first real request. Rather than babysit someone else's half-tested
|
||
code, we moved to sherpa-onnx — ONNX Runtime is a stable base, k2-fsa
|
||
publishes prebuilt int8 Parakeet weights per release, and the
|
||
recognizer API is a three-line call.
|
||
|
||
## API endpoints
|
||
|
||
| Method + path | Purpose |
|
||
|---|---|
|
||
| `POST /transcribe` | Multipart file upload → `{"text": "..."}` |
|
||
| `POST /v1/audio/transcriptions` | Same body; OpenAI-compatible path |
|
||
| `GET /healthz` | Health probe (used by docker healthcheck) |
|
||
|
||
## Path layout
|
||
|
||
| Host path | Container path | Purpose | Restic? |
|
||
|---|---|---|---|
|
||
| `/tank/parakeet/models/` | `/models` | ONNX encoder+decoder+joiner+tokens (~464 MiB int8) | excluded (regenerable — re-downloads from the URL on first run if absent) |
|
||
|
||
## ⚠ Verifying the GPU is actually in use
|
||
|
||
**ONNX Runtime's CUDA execution provider falls back to CPU silently.** It logs
|
||
a warning if you are looking, keeps the process alive, answers `200`, and
|
||
returns *correct transcriptions* — just far slower. So `PROVIDER=cuda` in
|
||
`.env` is a request, not a guarantee, and "the service is up and the text is
|
||
right" does **not** establish that the GPU is doing the work.
|
||
|
||
Blackwell is the reason this matters here rather than being pedantry: these
|
||
cards are `sm_120`, newer than the compute capabilities ORT's prebuilt CUDA
|
||
binaries have historically shipped kernels for, and the irv-ml1 host this
|
||
stack came from was Ampere `sm_86`. The move is exactly the kind that turns a
|
||
green service into a CPU service without a single error.
|
||
|
||
The honest check is to watch the card while a transcription runs:
|
||
|
||
```bash
|
||
# on fv-ml1 — terminal 1
|
||
watch -n0.2 'nvidia-smi --query-compute-apps=pid,process_name,used_memory \
|
||
--format=csv -i 3'
|
||
|
||
# terminal 2 — send real audio, not silence
|
||
curl -s -F file=@sample.wav http://127.0.0.1:8300/v1/audio/transcriptions
|
||
```
|
||
|
||
A process must appear **on GPU 3** for the duration. If GPU 3 stays empty, the
|
||
CUDA EP did not initialise and you are on CPU regardless of what `.env` says.
|
||
Confirm with the container's own startup log, which names the providers ORT
|
||
actually registered:
|
||
|
||
```bash
|
||
docker logs parakeet 2>&1 | grep -i 'provider\|cuda\|onnxruntime'
|
||
```
|
||
|
||
Timing alone is **not** sufficient evidence either way: the int8 model is fast
|
||
enough on a 96-thread EPYC that a CPU fallback still looks brisk on short
|
||
clips. Use the process check as the discriminator and treat throughput as a
|
||
secondary signal.
|
||
|
||
## LiteLLM alias
|
||
|
||
Reached fleet-wide through the gateway rather than by name, engine-neutral so
|
||
the backend can be swapped without touching consumers — the same pattern as
|
||
`ext-tts`:
|
||
|
||
| alias | mode | backend |
|
||
|---|---|---|
|
||
| `ext-stt` | `audio_transcription` | `http://10.251.50.54:8300/v1` |
|
||
| `whisper-1` | `audio_transcription` | same — OpenAI-compatible name so stock SDK clients work unchanged |
|
||
|
||
⚠ **The alias uses a raw IP on purpose.** `ana-docker` (where LiteLLM runs)
|
||
resolves no `.internal` names at all — its `/etc/resolv.conf` points at
|
||
`1.1.1.1`/`1.0.0.1`, and the only reason the `ext-tts` backend resolves is a
|
||
hand-pinned `extra_hosts: irv-ml1.nh3.internal:10.6.110.50` in the LiteLLM
|
||
compose. Adding a second hosts entry would mean recreating the container and
|
||
bouncing the gateway for every consumer; an IP costs nothing and cannot go
|
||
stale silently. See the DNS follow-up in `persistent-memory.md`.
|
||
|
||
Aliases live in LiteLLM's **Postgres store** (`store_model_in_db: true`), not
|
||
in `config.yaml` — that is where the `ext-tts` family lives too, and it means
|
||
adding one needs no gateway restart. It also means `config.yaml` is not a
|
||
complete picture of what the gateway serves: check `/v1/models` or
|
||
`/model/info`, never just the file.
|
||
|
||
## Deploy
|
||
|
||
```bash
|
||
scripts/deploy-stack.sh fv-ml1 parakeet --compose
|
||
# on fv-ml1, first time only:
|
||
# cp .env.example .env # then edit
|
||
docker compose -f /opt/docker/compose/parakeet/compose.yaml build
|
||
docker compose -f /opt/docker/compose/parakeet/compose.yaml up -d
|
||
```
|
||
|
||
First boot downloads ~464 MiB of ONNX weights into `/tank/parakeet/models/`;
|
||
the healthcheck's `start_period` is 300 s to cover it. Subsequent starts skip
|
||
the download.
|
||
|
||
## Switching model variants
|
||
|
||
One line in `.env`, then `docker compose up -d` (not `restart` — the model URL
|
||
is read by the entrypoint at container creation) and delete the old files from
|
||
`/tank/parakeet/models/` so the entrypoint re-downloads:
|
||
|
||
- **v3** (default) — 25 European languages
|
||
- **v2** — English only; slightly better on English-only material
|
||
|
||
Both are ~460 MiB int8 tarballs from the same k2-fsa release page.
|