vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(parakeet): stand up Parakeet STT on fv-ml1 GPU 3 + LiteLLM ext-stt/whisper-1

Browse Source 
Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card
and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry
the vLLM seats at 84-95.5 GB of 96.

Changes:

- compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used
  `count: all`, which would have handed a 0.6B ASR seat all four cards);
  join traefik-net; port 8300; homepage href to the live FV address.
- .env.example: default to the v3 int8 model (25 European languages, 464 MiB)
  rather than English-only v2; models to /tank/parakeet/models.
- app.py: warm the recognizer at startup before uvicorn accepts traffic.

The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes
lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at
45.1s on a second container) against ~0.50s warm. A 45s first request is
indistinguishable from a hang and LiteLLM's default timeout abandons it long
before it returns. Decoding 1s of silence at load moves the cost inside the
healthcheck's 300s start_period; first real request after restart is now 0.65s.

Verification, because "provider=cuda" in the log is only an echo of the env var:
ORT falls back to CPU silently and still returns correct text, so the service
being up and the transcript being right establishes nothing. The discriminator is
a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS
sentence transcribes near-exactly (positive), 3s of digital silence returns
empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread
0.47-0.65s, single-stream, one clip: a smoke measurement with its harness
stated, not a benchmark.

Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1`
(OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's
Postgres store where the ext-tts family already lives — no gateway restart, and
config.yaml is consequently not a complete picture of what the gateway serves.
Both verified end to end.

The aliases use a raw IP deliberately: ana-docker resolves no .internal names at
all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a
hand-pinned extra_hosts entry. A second hosts entry would mean recreating the
container and bouncing the gateway for every consumer.

Also records the svos_miranda plugin validation pass and its structural findings,
and notes that the irv-ml1 parakeet is still running — there are two now, and
retiring the old one is the operator's call.
This commit is contained in:
vh
2026-09-15 01:41:41 -07:00
parent c6b6435c52
commit b9b14b5baf
8 changed files with 366 additions and 93 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/parakeet/README.md
+90 -63
View File
@@ -4,12 +4,26 @@ NVIDIA Parakeet-TDT 0.6B (int8 ONNX) served by our own thin FastAPI
wrapper over [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)
(ONNX Runtime + CUDA).
**Server:** irv-ml1 (Irvine, WireGuard-only)
**Port:** 8765 (container 8000)
**GPU:** both exposed (`NVIDIA_VISIBLE_DEVICES=all`); sherpa-onnx uses
whichever CUDA ExecutionProvider picks
**Image:** `local/parakeet:sherpa-onnx-v1` — built from `Dockerfile` +
**Server:** fv-ml1 (Fountain Valley, `10.251.50.54`) — moved from irv-ml1 2026-09-15
**Port:** 8300 (container 8000)
**GPU:** **3**, pinned explicitly via `device_ids` — the utility card
**Image:** `local/parakeet:sherpa-onnx-v4` — built from `Dockerfile` +
`app.py` + `entrypoint.sh` in this directory; **we own all the code**
**Model:** `parakeet-tdt-0.6b-v3` int8, 25 European languages (~464 MiB)
## Why GPU 3
fv-ml1 has four RTX PRO 6000 Blackwell Max-Q (96 GB each). Three carry the
vLLM serving seats and run 85–98 % full; GPU 3 is the utility card and was
empty (2 MiB) at placement time. A 0.6 B int8 ASR model is a rounding error
next to those seats, but it still has to go somewhere that is not fighting
them for VRAM.
⚠ The pin is `deploy.resources.reservations.devices[].device_ids: ["3"]`,
the fleet convention — **not** `count: all`, which is what the dead on-host
stub used and which would have handed this seat all four cards. Inside the
container the pinned card presents as `cuda:0`, which is what sherpa-onnx's
CUDA execution provider takes by default.
## Why not the FastAPI community wrappers
@@ -35,80 +49,93 @@ recognizer API is a three-line call.
| Host path | Container path | Purpose | Restic? |
|---|---|---|---|
| `/worktank/parakeet/models/` | `/models` | ONNX encoder+decoder+joiner+tokens (~400 MB int8) | excluded (regenerable — re-downloads from the URL on first run if absent) |
| `/tank/parakeet/models/` | `/models` | ONNX encoder+decoder+joiner+tokens (~464 MiB int8) | excluded (regenerable — re-downloads from the URL on first run if absent) |
## First-time deploy on irv-ml1
## ⚠ Verifying the GPU is actually in use
**ONNX Runtime's CUDA execution provider falls back to CPU silently.** It logs
a warning if you are looking, keeps the process alive, answers `200`, and
returns *correct transcriptions* — just far slower. So `PROVIDER=cuda` in
`.env` is a request, not a guarantee, and "the service is up and the text is
right" does **not** establish that the GPU is doing the work.
Blackwell is the reason this matters here rather than being pedantry: these
cards are `sm_120`, newer than the compute capabilities ORT's prebuilt CUDA
binaries have historically shipped kernels for, and the irv-ml1 host this
stack came from was Ampere `sm_86`. The move is exactly the kind that turns a
green service into a CPU service without a single error.
The honest check is to watch the card while a transcription runs:
```bash
# 1. Push compose + Dockerfile + app + entrypoint
scripts/deploy-stack.sh irv-ml1 parakeet
# on fv-ml1 — terminal 1
watch -n0.2 'nvidia-smi --query-compute-apps=pid,process_name,used_memory \
--format=csv -i 3'
# 2. Make sure the models dir exists (one-time, already done from the
# earlier Shadowfita deploy; this is idempotent)
ssh -t irv-ml1 'sudo mkdir -p /worktank/parakeet/models && \
sudo chown -R lkraven:lkraven /worktank/parakeet'
# 3. Build the image and bring up. First boot does a ~400 MB model
# download via the entrypoint; allow 1–2 minutes before /healthz
# flips healthy.
ssh irv-ml1 '
cd /opt/docker/compose/parakeet && \
cp -n .env.example .env && \
docker compose config >/dev/null && \
docker compose build && \
docker compose up -d && \
docker compose logs -f --tail=30
'
# terminal 2 — send real audio, not silence
curl -s -F file=@sample.wav http://127.0.0.1:8300/v1/audio/transcriptions
```
## Smoke test
A process must appear **on GPU 3** for the duration. If GPU 3 stays empty, the
CUDA EP did not initialise and you are on CPU regardless of what `.env` says.
Confirm with the container's own startup log, which names the providers ORT
actually registered:
```bash
# Over WG from the workstation
curl -F "file=@sample.wav" http://10.100.79.3:8765/transcribe
# → {"text": "hello world"}
# OpenAI-shape alias (for clients that only know /v1/audio/transcriptions)
curl -F "file=@sample.wav" http://10.100.79.3:8765/v1/audio/transcriptions
docker logs parakeet 2>&1 | grep -i 'provider\|cuda\|onnxruntime'
```
## Switching to the v3 (multilingual) model
Timing alone is **not** sufficient evidence either way: the int8 model is fast
enough on a 96-thread EPYC that a CPU fallback still looks brisk on short
clips. Use the process check as the discriminator and treat throughput as a
secondary signal.
The env var `PARAKEET_MODEL_URL` picks the release tarball. To swap
from the English-only v2 to the 25-language v3:
## LiteLLM alias
Reached fleet-wide through the gateway rather than by name, engine-neutral so
the backend can be swapped without touching consumers — the same pattern as
`ext-tts`:
| alias | mode | backend |
|---|---|---|
| `ext-stt` | `audio_transcription` | `http://10.251.50.54:8300/v1` |
| `whisper-1` | `audio_transcription` | same — OpenAI-compatible name so stock SDK clients work unchanged |
⚠ **The alias uses a raw IP on purpose.** `ana-docker` (where LiteLLM runs)
resolves no `.internal` names at all — its `/etc/resolv.conf` points at
`1.1.1.1`/`1.0.0.1`, and the only reason the `ext-tts` backend resolves is a
hand-pinned `extra_hosts: irv-ml1.nh3.internal:10.6.110.50` in the LiteLLM
compose. Adding a second hosts entry would mean recreating the container and
bouncing the gateway for every consumer; an IP costs nothing and cannot go
stale silently. See the DNS follow-up in `persistent-memory.md`.
Aliases live in LiteLLM's **Postgres store** (`store_model_in_db: true`), not
in `config.yaml` — that is where the `ext-tts` family lives too, and it means
adding one needs no gateway restart. It also means `config.yaml` is not a
complete picture of what the gateway serves: check `/v1/models` or
`/model/info`, never just the file.
## Deploy
```bash
ssh irv-ml1 '
cd /opt/docker/compose/parakeet && \
sed -i "s|v2-int8|v3-int8|" .env && \
# Wipe the v2 weights so the entrypoint re-downloads v3 on next up:
rm -f /worktank/parakeet/models/*.onnx /worktank/parakeet/models/tokens.txt && \
docker compose up -d && \
docker compose logs -f --tail=30
'
scripts/deploy-stack.sh fv-ml1 parakeet --compose
# on fv-ml1, first time only:
# cp .env.example .env # then edit
docker compose -f /opt/docker/compose/parakeet/compose.yaml build
docker compose -f /opt/docker/compose/parakeet/compose.yaml up -d
```
## Upgrade sherpa-onnx or change the base image
First boot downloads ~464 MiB of ONNX weights into `/tank/parakeet/models/`;
the healthcheck's `start_period` is 300 s to cover it. Subsequent starts skip
the download.
Bump `PARAKEET_TAG` in `.env` to force a rebuild of the local image
after editing the `Dockerfile`, then:
## Switching model variants
```bash
scripts/deploy-stack.sh irv-ml1 parakeet
ssh irv-ml1 'cd /opt/docker/compose/parakeet && docker compose build && docker compose up -d'
```
One line in `.env`, then `docker compose up -d` (not `restart` — the model URL
is read by the entrypoint at container creation) and delete the old files from
`/tank/parakeet/models/` so the entrypoint re-downloads:
Model files under `/worktank/parakeet/models/` are preserved across
image rebuilds.
- **v3** (default) — 25 European languages
- **v2** — English only; slightly better on English-only material
## File layout
```
stacks/parakeet/
├── Dockerfile # CUDA 12.8 + cuDNN 9 base, sherpa-onnx-cu12 wheel
├── app.py # FastAPI — ~60 lines
├── entrypoint.sh # downloads model on first run, then uvicorn
├── compose.yaml # one service, bind-mounts the models dir
├── .env.example # template; real .env lives on the server
└── README.md # this file
```
Both are ~460 MiB int8 tarballs from the same k2-fsa release page.