feat(parakeet): stand up Parakeet STT on fv-ml1 GPU 3 + LiteLLM ext-stt/whisper-1
Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry the vLLM seats at 84-95.5 GB of 96. Changes: - compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used `count: all`, which would have handed a 0.6B ASR seat all four cards); join traefik-net; port 8300; homepage href to the live FV address. - .env.example: default to the v3 int8 model (25 European languages, 464 MiB) rather than English-only v2; models to /tank/parakeet/models. - app.py: warm the recognizer at startup before uvicorn accepts traffic. The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at 45.1s on a second container) against ~0.50s warm. A 45s first request is indistinguishable from a hang and LiteLLM's default timeout abandons it long before it returns. Decoding 1s of silence at load moves the cost inside the healthcheck's 300s start_period; first real request after restart is now 0.65s. Verification, because "provider=cuda" in the log is only an echo of the env var: ORT falls back to CPU silently and still returns correct text, so the service being up and the transcript being right establishes nothing. The discriminator is a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS sentence transcribes near-exactly (positive), 3s of digital silence returns empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread 0.47-0.65s, single-stream, one clip: a smoke measurement with its harness stated, not a benchmark. Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1` (OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's Postgres store where the ext-tts family already lives — no gateway restart, and config.yaml is consequently not a complete picture of what the gateway serves. Both verified end to end. The aliases use a raw IP deliberately: ana-docker resolves no .internal names at all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a hand-pinned extra_hosts entry. A second hosts entry would mean recreating the container and bouncing the gateway for every consumer. Also records the svos_miranda plugin validation pass and its structural findings, and notes that the irv-ml1 parakeet is still running — there are two now, and retiring the old one is the operator's call.
This commit is contained in:
@@ -0,0 +1,86 @@
|
||||
# Parakeet STT on fv-ml1 GPU 3 (2026-09-15)
|
||||
|
||||
Operator asked for an STT service on fv-ml1's utility GPU plus a LiteLLM alias.
|
||||
|
||||
## What it is
|
||||
|
||||
`stacks/parakeet/` — Parakeet-TDT 0.6B **v3** int8 ONNX (25 European languages,
|
||||
464 MiB) under sherpa-onnx, behind ~90 lines of FastAPI we own. Container
|
||||
`parakeet`, port **8300**, **GPU 3** pinned by `device_ids`. Image
|
||||
`local/parakeet:sherpa-onnx-v4` (5.09 GB).
|
||||
|
||||
Not greenfield: the stack already existed, targeting irv-ml1. Retargeted rather
|
||||
than rewritten — the Ampere→Blackwell move was the only real question.
|
||||
|
||||
## Why GPU 3
|
||||
|
||||
GPU 0 = 84/96 GB, GPU 1 = 92.9/96, GPU 2 = 95.5/96 (the vLLM seats). **GPU 3 was
|
||||
at 2 MiB.** The dead on-host stub used `count: all`, which would have handed this
|
||||
seat all four cards; replaced with an explicit `device_ids: ["3"]` per the fleet
|
||||
convention. Inside the container the pinned card presents as `cuda:0`, which is
|
||||
what ORT's CUDA EP takes by default.
|
||||
|
||||
## ⚠ The finding worth keeping: a 45-second first decode
|
||||
|
||||
ONNX Runtime's CUDA EP compiles and autotunes lazily, on the **first decode**, not
|
||||
at session creation. On sm_120:
|
||||
|
||||
| | measured |
|
||||
|---|---|
|
||||
| first decode, cold container | **45.7 s** (n=1), reproduced at **45.1 s** on a second container |
|
||||
| warm, 8.52 s clip | **0.50 s** median (n=5: 0.65 / 0.53 / 0.48 / 0.47 / 0.50) |
|
||||
|
||||
≈17× realtime warm, single-stream, one 8.52 s clip, int8, GPU 3 idle otherwise.
|
||||
That is a smoke measurement with its harness stated, **not** a benchmark — no
|
||||
concurrency sweep, no length sweep, one clip.
|
||||
|
||||
A 45 s first request is indistinguishable from a hang to any caller, and LiteLLM's
|
||||
default timeout would abandon it. `_warm()` in `app.py` now decodes 1 s of silence
|
||||
before uvicorn accepts traffic, so the cost lands inside the healthcheck's 300 s
|
||||
`start_period`. First real request after restart: **0.65 s**.
|
||||
|
||||
## ⚠⚠ "provider=cuda" is not evidence the GPU is being used
|
||||
|
||||
ORT's CUDA EP **falls back to CPU silently** — the process lives, answers 200, and
|
||||
returns *correct text*, just slowly. Our own log line `loading OfflineRecognizer
|
||||
(provider=cuda...)` merely echoes the env var and proves nothing.
|
||||
|
||||
The discriminator that actually settles it:
|
||||
|
||||
```
|
||||
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv -i 3
|
||||
-> 1588301, /opt/venv/bin/python3, 922 MiB
|
||||
```
|
||||
|
||||
Timing is **not** a sufficient check either — the int8 model is fast enough on a
|
||||
96-thread EPYC that a CPU fallback still looks brisk on short clips.
|
||||
|
||||
Controls run, both directions:
|
||||
- **positive** — known TTS sentence in, near-exact transcript out (two word errors,
|
||||
both attributable to the source audio: an inserted "um", "Foun Valley").
|
||||
- **null** — 3 s of digital silence → `{"text": ""}`. The instrument does not
|
||||
manufacture signal.
|
||||
|
||||
## LiteLLM
|
||||
|
||||
Two aliases, both `mode: audio_transcription` → `http://10.251.50.54:8300/v1`:
|
||||
`ext-stt` (engine-neutral fleet name, mirrors `ext-tts`) and `whisper-1`
|
||||
(OpenAI-compatible drop-in). Both verified end-to-end through the gateway.
|
||||
|
||||
Registered via `POST /model/new`, i.e. the **Postgres store**, not `config.yaml` —
|
||||
that is where the `ext-tts` family lives, and it needs no gateway restart.
|
||||
⚠ Corollary: `config.yaml` is NOT a complete picture of what the gateway serves
|
||||
(it lists 35 models; the gateway serves 40, and carries stale entries like
|
||||
`granite-4.1-8b`). Read `/v1/models` or `/model/info`, never just the file.
|
||||
|
||||
⚠ **Raw IP on purpose** — see the ana-docker DNS row in the index.
|
||||
|
||||
## Loose ends
|
||||
|
||||
- The **irv-ml1 parakeet is still running** (healthz 200 on `100.64.0.6:8765`).
|
||||
Two Parakeets now. Retiring the old one is the operator's call — not touched.
|
||||
- `/opt/docker/compose/parakeet` and `/tank/parakeet` normalised to `root:docker
|
||||
2775`; the rest of fv-ml1's deploy tree is still `lkraven:lkraven` (it was not
|
||||
part of the 5-host normalisation).
|
||||
- `servers/fv-ml1/README.md` is still broadly stale — it claims 2 GPUs and a
|
||||
2026-07-22 stack list. Only the parakeet/GPU-3 rows were corrected.
|
||||
@@ -0,0 +1,64 @@
|
||||
# svos_miranda Hermes plugin — validation pass (2026-09-15)
|
||||
|
||||
svos-dev asked infra-ops to run `hermes plugins validate` → `doctor` → `compat`
|
||||
on `/home/lkraven/development/svos/hermes_plugin/` and report before enabling.
|
||||
Hermes Agent v0.21.1 (2026.9.7), local `b88e6776`, on nh3-dev.
|
||||
|
||||
## The blocker (found, fixed by svos-dev at `c964e64`)
|
||||
|
||||
Three absolute intra-package imports — `from hermes_plugin._vendored`, `.forward`,
|
||||
`.jwt` — pinned the package to its **source directory name**. The documented
|
||||
install renames it to `svos_miranda`, and Hermes loads directory plugins under the
|
||||
`hermes_plugins.<dir>` namespace; in neither case does a top-level `hermes_plugin`
|
||||
exist. Fix: three relative imports.
|
||||
|
||||
⚠ **The harness hid this from three different readers.** My first `validate` passed
|
||||
the import only because my cwd was the SVOS repo root. svos-dev's test suite
|
||||
imports `hermes_plugin.*` from that same root, and their editable install resolves
|
||||
the name from anywhere on the box — it only reproduced for them once `sys.path` was
|
||||
stripped. Same class as `feedback_filters_that_silently_narrow_the_window`: the
|
||||
instrument carried the result.
|
||||
|
||||
## ⚠ Two of the three commands CANNOT pass this plugin, ever
|
||||
|
||||
Neither is fixable from the plugin side. Both are now documented in its README.
|
||||
|
||||
- **`validate`** — two independent causes. Its `RecordingContext.get_config`
|
||||
(`hermes_cli/plugin_validate.py:219-222`) returns the **default for every key**,
|
||||
ignoring `config.yaml` entirely, so `dispatch_key` is always `""`. And its
|
||||
`register_tool` returns `None`, which the plugin's INV-P6 guard correctly reads
|
||||
as a name collision — so even with a key supplied it raises on the first tool.
|
||||
- **`doctor`** — runs `register()` under a **temp `HERMES_HOME`** with sockets
|
||||
blocked, so no config exists there either.
|
||||
|
||||
## Tool-level gotchas worth remembering
|
||||
|
||||
- ⚠ **`hermes plugins doctor` exits 0 even when it prints ERROR.** Needs `--ci`.
|
||||
- ⚠ **`hermes plugins compat <nonexistent-path>` prints ✓ and exits 0.** A typo'd
|
||||
path reads as a pass. (The instrument itself is sound — verified with a throwaway
|
||||
plugin importing a real deprecated path, which it flagged with file:line, exit 1.)
|
||||
- ⚠ **`doctor`'s sandbox registry starts EMPTY — 0 entries, no built-ins.** So
|
||||
doctor cannot detect tool-name collisions at all. `validate`'s separate static
|
||||
"built-in tool collisions" check is what covers that.
|
||||
- The real `PluginContext.register_tool` (`hermes_cli/plugins.py:449-491`) returns
|
||||
a truthy `PluginRegistration` on success — confirmed against the live runtime.
|
||||
|
||||
## Roster verified another way
|
||||
|
||||
Since neither command can supply config, a probe mirroring validate's context but
|
||||
returning real settings and a truthy handle gave: **8 tools** with
|
||||
`repo_read_enabled: true`, **7** with false or omitted, names matching
|
||||
`plugin.yaml` exactly, zero hooks/middleware/commands. All nine settings-validation
|
||||
controls (quoted booleans, `"90 s"`, zero/negative timeouts, empty/whitespace
|
||||
strings) raise errors naming their own key.
|
||||
|
||||
## A false finding I caught on myself
|
||||
|
||||
A probe registering `read_file` got back a `PluginRegistration` instead of the
|
||||
expected refusal — which looked like the plugin's collision reading was wrong. It
|
||||
was not: doctor's sandbox holds no built-ins, so nothing was claimed and **my
|
||||
positive case was not positive.** Reported as untested rather than as a finding.
|
||||
|
||||
## Where enabling stands
|
||||
|
||||
Blocked on the operator only. See the OPEN row in the index for the sequence.
|
||||
+28
-6
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-15 ~01:15 PT (FV cross-site routing FIXED; mesh membership RETIRED for both fv-ml1 and nh3-dev — six nodes remain, each with a job; fv-ml1 carries a break-glass rejoin instead. Next session: drain svos-dev, then stand up an STT.)_
|
||||
_Last updated: 2026-09-15 ~01:45 PT (Parakeet STT LIVE on fv-ml1 GPU 3 + LiteLLM `ext-stt`/`whisper-1`; svos_miranda plugin validated and its blocker fixed by svos-dev — enabling is the operator's call.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -117,11 +117,25 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
_As of 2026-09-15 ~01:15 PT._
|
||||
|
||||
### ⭐ NEXT SESSION'S NAMED WORK (operator, at snapshot time)
|
||||
1. **Drain svos-dev** — one althing message unread since 00:52 PT, pinged five times,
|
||||
never read. `/althing:inbox` first.
|
||||
2. **Stand up an STT** (speech-to-text) service. Nothing started: no placement decided,
|
||||
no model chosen, no stack authored. Greenfield.
|
||||
### ⭐ BOTH NAMED JOBS CLOSED (2026-09-15 ~01:45 PT)
|
||||
1. **svos-dev drained** — four messages, thread closed from their end. Ran
|
||||
`hermes plugins validate/doctor/compat` on `svos_miranda`; found and A/B-proved a
|
||||
load blocker (absolute intra-package imports), which svos-dev fixed at `c964e64`.
|
||||
⚠ **Enabling is still pending and is the OPERATOR'S call** — see the decision row below.
|
||||
2. **STT stood up** — Parakeet on fv-ml1 GPU 3, `ext-stt` / `whisper-1` in LiteLLM.
|
||||
Not greenfield after all: `stacks/parakeet/` already existed (sherpa-onnx + our own
|
||||
FastAPI wrapper, previously on irv-ml1) and was retargeted rather than rewritten.
|
||||
|
||||
### ⏳ OPEN — svos_miranda enable (needs the operator)
|
||||
Everything technical is done. Enabling requires editing the operator's live
|
||||
`~/.hermes/config.yaml` (this session was guardrailed out of that write, then
|
||||
unblocked — but the go/no-go is his) and svos-dev restarts `:8770` after, which takes
|
||||
the SVOS board down until the roster verifies. Sequence when he says go: install to
|
||||
`~/.hermes/plugins/svos_miranda`; add `plugins.enabled` + `entries.svos_miranda.settings`
|
||||
(dispatch key from `secret get nh3-dev/svos/bifrost-dispatch-key`, `repo_read_enabled: true`);
|
||||
set `platform_toolsets["api_server"] = ["svos_miranda"]` and `agent.disabled_toolsets`;
|
||||
**re-derive that list after registration** from `GET http://127.0.0.1:8765/v1/toolsets`
|
||||
(28 rows now, `svos_miranda` joins once it registers); hand off to svos-dev for the restart.
|
||||
|
||||
### Fleet networking — closed out this session
|
||||
- **FV cross-site routing FIXED.** fv-ml1 reaches NH3/ESH/ANA/IRV/internet via outbound
|
||||
@@ -161,6 +175,14 @@ hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`).
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-15]` **Parakeet STT live on fv-ml1 GPU 3, behind LiteLLM `ext-stt` / `whisper-1`.** Retargeted the existing `stacks/parakeet/` (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ **ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s** — every later call ~0.5 s; a startup warmup in `app.py` now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was **verified by a process on GPU 3 (922 MiB), not by the `provider=cuda` log line**, because ORT falls back to CPU silently and still returns correct text. Silence → `""` (null control), known sentence → near-exact (positive control). → `persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md`
|
||||
|
||||
- `[2026-09-15]` **`svos_miranda` Hermes plugin validated; found its load blocker.** Absolute intra-package imports (`from hermes_plugin.x`) could not resolve at the documented install name — fixed by svos-dev at `c964e64`. ⚠ **`hermes plugins validate` and `doctor` can NEVER pass this plugin**, by construction: validate's probe stub is config-blind AND returns `None` from `register_tool` (which the plugin's guard reads as a collision), and doctor runs under a temp `HERMES_HOME` with no config. ⚠ `doctor` exits **0** on ERROR (use `--ci`); `compat` reads a **nonexistent path as a pass**. Roster verified 8/7 by a probe supplying real settings. → `persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md`
|
||||
|
||||
- `[2026-09-15]` ⚠ **ana-docker resolves NO `.internal` names** — its `/etc/resolv.conf` is `1.1.1.1`/`1.0.0.1`, not the fleet AdGuard. LiteLLM only reaches `irv-ml1.nh3.internal` because of a hand-pinned `extra_hosts` in its compose. New gateway aliases therefore use **raw IPs**; adding a hosts entry would mean recreating the container and bouncing the gateway for every consumer. Fleet-wide DNS fix is unowned.
|
||||
|
||||
- `[2026-09-15]` ⚠ **Two Homepage ASR cards point at the dead wg0 lifeline `10.100.79.3`** (retired at the headscale cutover): the old irv-ml1 `parakeet` :8765 and `Speaches ASR` :8204. The irv-ml1 parakeet is **genuinely still running** (healthz 200 on `100.64.0.6` and `10.6.110.50`) — only its href is stale. So the fleet now has **two** Parakeets. Retiring the irv-ml1 one is the operator's call; not torn down.
|
||||
|
||||
- `[2026-09-15]` **Mesh membership retired for fv-ml1 and nh3-dev — six nodes left, each with a job.** fv-ml1 gets break-glass rejoin instead of standing membership; nh3-dev's retirement also removed the nh3-scale masquerade exception it had required. Exactly one live reusable pre-auth key remains fleet-wide. → `persistent-memory.d/2026-09-15-fv-mesh-watchdog.md`
|
||||
|
||||
- `[2026-09-15]` **FV cross-site routing fixed — one OPNsense outbound-NAT rule had been scoped to Anaheim only.** fv-ml1 now reaches NH3/ESH/IRV/ANA/mesh/internet; four rules, all `src=10.251.50.0/24`. The diagnostic signature is the valuable part: every layer looks correct and the discriminator is that *every other site pair works*. → `persistent-memory.d/2026-09-15-fv-cross-site-snat.md`
|
||||
|
||||
@@ -108,7 +108,16 @@ it on the card with room or evicting a dormant one first.
|
||||
seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`,
|
||||
`ms32-24b-angel`, `qwen3.5-122b`, `qwopus3.5-122b`, `qwen35-vl`, `qwen36-vl`,
|
||||
`qwen36-27b-aeon`, `qwen-image-bench`, `vibevoice`, `comfyui`, `kokoro`,
|
||||
`parakeet`, `vllm-qwen3`.
|
||||
`vllm-qwen3`.
|
||||
|
||||
**GPU 3 — utility card:**
|
||||
|
||||
| Container | Port | Serves | Notes |
|
||||
|-----------|------|--------|-------|
|
||||
| `parakeet` | 8300 | Parakeet-TDT 0.6B v3 int8 (25 languages) | ASR via sherpa-onnx, LiteLLM `ext-stt` / `whisper-1`. Relocated from irv-ml1 2026-09-15. `stacks/parakeet/`. |
|
||||
|
||||
⚠ The other three cards run 85-98% full, so GPU 3 is where a new small seat goes
|
||||
until something bigger claims it.
|
||||
|
||||
**Retired:**
|
||||
- `llama-swap` (former GGUF multiplexer on :9292) — replaced by dedicated
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Parakeet ASR stack tunables. Copy to `.env` on irv-ml1 before deploying.
|
||||
# Parakeet ASR stack tunables. Copy to `.env` on fv-ml1 before deploying.
|
||||
#
|
||||
# cp .env.example .env
|
||||
# # edit as needed
|
||||
@@ -7,29 +7,42 @@
|
||||
|
||||
# Image tag. Bump when you change the Dockerfile / app.py so docker caches
|
||||
# cleanly.
|
||||
PARAKEET_TAG=sherpa-onnx-v2
|
||||
PARAKEET_TAG=sherpa-onnx-v4
|
||||
|
||||
# Host port for the FastAPI server (container listens on 8000)
|
||||
PARAKEET_PORT=8765
|
||||
# Which GPU to pin. fv-ml1 GPU 3 is the utility card — 0/1/2 carry the vLLM
|
||||
# serving seats and sit at 85-98% VRAM, so this is the only one with room.
|
||||
# The container sees whichever card this names as cuda:0 internally.
|
||||
PARAKEET_GPU=3
|
||||
|
||||
# Bind address. 0.0.0.0 exposes on all interfaces including the WG tunnel IP
|
||||
# (10.100.79.3). Use 127.0.0.1 to restrict to local-only.
|
||||
# Host port for the FastAPI server (container listens on 8000). 8300 is
|
||||
# fv-ml1's established parakeet port; the 80xx range belongs to the vLLM seats.
|
||||
PARAKEET_PORT=8300
|
||||
|
||||
# Bind address. 0.0.0.0 exposes on all interfaces. Use 127.0.0.1 to restrict
|
||||
# to local-only — but LiteLLM on ana-docker reaches this over the LAN, so it
|
||||
# has to be 0.0.0.0 for the gateway alias to work.
|
||||
PARAKEET_BIND=0.0.0.0
|
||||
|
||||
# Host path for the ONNX model files — encoder/decoder/joiner/tokens.txt.
|
||||
# Downloaded by the entrypoint on first run if absent. Must exist before
|
||||
# first `up` (directory, not files).
|
||||
PARAKEET_MODELS_DIR=/worktank/parakeet/models
|
||||
# first `up` (directory, not files). Regenerable — exclude from restic.
|
||||
PARAKEET_MODELS_DIR=/tank/parakeet/models
|
||||
|
||||
# Which sherpa-onnx release tarball to fetch on first boot. Default is the
|
||||
# int8-quantized English-only v2 (~400 MB). Switch to the v3 tarball below
|
||||
# to cover 25 European languages at a similar size:
|
||||
# https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8.tar.bz2
|
||||
PARAKEET_MODEL_URL=https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2
|
||||
# Which sherpa-onnx release tarball to fetch on first boot.
|
||||
# v3 (default, 464 MiB) — 25 European languages
|
||||
# v2 — English only, swap the URL below
|
||||
# https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2
|
||||
PARAKEET_MODEL_URL=https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8.tar.bz2
|
||||
|
||||
# ONNX Runtime execution provider. `cuda` uses the GPU (requires nvidia
|
||||
# runtime + matching CUDA/cuDNN in the image). `cpu` falls back to CPU —
|
||||
# fine for low-volume dev use; ~4-8× slower on this host.
|
||||
# ONNX Runtime execution provider. `cuda` uses the GPU (requires the nvidia
|
||||
# container runtime + matching CUDA/cuDNN in the image). `cpu` falls back to
|
||||
# CPU.
|
||||
#
|
||||
# ⚠ ORT's CUDA EP FALLS BACK TO CPU SILENTLY when it cannot initialise — the
|
||||
# server still answers 200 and still returns correct text, just slowly. So
|
||||
# `PROVIDER=cuda` is a REQUEST, not a guarantee, and the only honest check is
|
||||
# to watch `nvidia-smi` during a transcription and confirm a process appears on
|
||||
# the pinned card. See README § Verifying the GPU is actually in use.
|
||||
PARAKEET_PROVIDER=cuda
|
||||
|
||||
# CPU threads per recognizer session. Irrelevant when provider=cuda;
|
||||
|
||||
+90
-63
@@ -4,12 +4,26 @@ NVIDIA Parakeet-TDT 0.6B (int8 ONNX) served by our own thin FastAPI
|
||||
wrapper over [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)
|
||||
(ONNX Runtime + CUDA).
|
||||
|
||||
**Server:** irv-ml1 (Irvine, WireGuard-only)
|
||||
**Port:** 8765 (container 8000)
|
||||
**GPU:** both exposed (`NVIDIA_VISIBLE_DEVICES=all`); sherpa-onnx uses
|
||||
whichever CUDA ExecutionProvider picks
|
||||
**Image:** `local/parakeet:sherpa-onnx-v1` — built from `Dockerfile` +
|
||||
**Server:** fv-ml1 (Fountain Valley, `10.251.50.54`) — moved from irv-ml1 2026-09-15
|
||||
**Port:** 8300 (container 8000)
|
||||
**GPU:** **3**, pinned explicitly via `device_ids` — the utility card
|
||||
**Image:** `local/parakeet:sherpa-onnx-v4` — built from `Dockerfile` +
|
||||
`app.py` + `entrypoint.sh` in this directory; **we own all the code**
|
||||
**Model:** `parakeet-tdt-0.6b-v3` int8, 25 European languages (~464 MiB)
|
||||
|
||||
## Why GPU 3
|
||||
|
||||
fv-ml1 has four RTX PRO 6000 Blackwell Max-Q (96 GB each). Three carry the
|
||||
vLLM serving seats and run 85–98 % full; GPU 3 is the utility card and was
|
||||
empty (2 MiB) at placement time. A 0.6 B int8 ASR model is a rounding error
|
||||
next to those seats, but it still has to go somewhere that is not fighting
|
||||
them for VRAM.
|
||||
|
||||
⚠ The pin is `deploy.resources.reservations.devices[].device_ids: ["3"]`,
|
||||
the fleet convention — **not** `count: all`, which is what the dead on-host
|
||||
stub used and which would have handed this seat all four cards. Inside the
|
||||
container the pinned card presents as `cuda:0`, which is what sherpa-onnx's
|
||||
CUDA execution provider takes by default.
|
||||
|
||||
## Why not the FastAPI community wrappers
|
||||
|
||||
@@ -35,80 +49,93 @@ recognizer API is a three-line call.
|
||||
|
||||
| Host path | Container path | Purpose | Restic? |
|
||||
|---|---|---|---|
|
||||
| `/worktank/parakeet/models/` | `/models` | ONNX encoder+decoder+joiner+tokens (~400 MB int8) | excluded (regenerable — re-downloads from the URL on first run if absent) |
|
||||
| `/tank/parakeet/models/` | `/models` | ONNX encoder+decoder+joiner+tokens (~464 MiB int8) | excluded (regenerable — re-downloads from the URL on first run if absent) |
|
||||
|
||||
## First-time deploy on irv-ml1
|
||||
## ⚠ Verifying the GPU is actually in use
|
||||
|
||||
**ONNX Runtime's CUDA execution provider falls back to CPU silently.** It logs
|
||||
a warning if you are looking, keeps the process alive, answers `200`, and
|
||||
returns *correct transcriptions* — just far slower. So `PROVIDER=cuda` in
|
||||
`.env` is a request, not a guarantee, and "the service is up and the text is
|
||||
right" does **not** establish that the GPU is doing the work.
|
||||
|
||||
Blackwell is the reason this matters here rather than being pedantry: these
|
||||
cards are `sm_120`, newer than the compute capabilities ORT's prebuilt CUDA
|
||||
binaries have historically shipped kernels for, and the irv-ml1 host this
|
||||
stack came from was Ampere `sm_86`. The move is exactly the kind that turns a
|
||||
green service into a CPU service without a single error.
|
||||
|
||||
The honest check is to watch the card while a transcription runs:
|
||||
|
||||
```bash
|
||||
# 1. Push compose + Dockerfile + app + entrypoint
|
||||
scripts/deploy-stack.sh irv-ml1 parakeet
|
||||
# on fv-ml1 — terminal 1
|
||||
watch -n0.2 'nvidia-smi --query-compute-apps=pid,process_name,used_memory \
|
||||
--format=csv -i 3'
|
||||
|
||||
# 2. Make sure the models dir exists (one-time, already done from the
|
||||
# earlier Shadowfita deploy; this is idempotent)
|
||||
ssh -t irv-ml1 'sudo mkdir -p /worktank/parakeet/models && \
|
||||
sudo chown -R lkraven:lkraven /worktank/parakeet'
|
||||
|
||||
# 3. Build the image and bring up. First boot does a ~400 MB model
|
||||
# download via the entrypoint; allow 1–2 minutes before /healthz
|
||||
# flips healthy.
|
||||
ssh irv-ml1 '
|
||||
cd /opt/docker/compose/parakeet && \
|
||||
cp -n .env.example .env && \
|
||||
docker compose config >/dev/null && \
|
||||
docker compose build && \
|
||||
docker compose up -d && \
|
||||
docker compose logs -f --tail=30
|
||||
'
|
||||
# terminal 2 — send real audio, not silence
|
||||
curl -s -F file=@sample.wav http://127.0.0.1:8300/v1/audio/transcriptions
|
||||
```
|
||||
|
||||
## Smoke test
|
||||
A process must appear **on GPU 3** for the duration. If GPU 3 stays empty, the
|
||||
CUDA EP did not initialise and you are on CPU regardless of what `.env` says.
|
||||
Confirm with the container's own startup log, which names the providers ORT
|
||||
actually registered:
|
||||
|
||||
```bash
|
||||
# Over WG from the workstation
|
||||
curl -F "file=@sample.wav" http://10.100.79.3:8765/transcribe
|
||||
# → {"text": "hello world"}
|
||||
|
||||
# OpenAI-shape alias (for clients that only know /v1/audio/transcriptions)
|
||||
curl -F "file=@sample.wav" http://10.100.79.3:8765/v1/audio/transcriptions
|
||||
docker logs parakeet 2>&1 | grep -i 'provider\|cuda\|onnxruntime'
|
||||
```
|
||||
|
||||
## Switching to the v3 (multilingual) model
|
||||
Timing alone is **not** sufficient evidence either way: the int8 model is fast
|
||||
enough on a 96-thread EPYC that a CPU fallback still looks brisk on short
|
||||
clips. Use the process check as the discriminator and treat throughput as a
|
||||
secondary signal.
|
||||
|
||||
The env var `PARAKEET_MODEL_URL` picks the release tarball. To swap
|
||||
from the English-only v2 to the 25-language v3:
|
||||
## LiteLLM alias
|
||||
|
||||
Reached fleet-wide through the gateway rather than by name, engine-neutral so
|
||||
the backend can be swapped without touching consumers — the same pattern as
|
||||
`ext-tts`:
|
||||
|
||||
| alias | mode | backend |
|
||||
|---|---|---|
|
||||
| `ext-stt` | `audio_transcription` | `http://10.251.50.54:8300/v1` |
|
||||
| `whisper-1` | `audio_transcription` | same — OpenAI-compatible name so stock SDK clients work unchanged |
|
||||
|
||||
⚠ **The alias uses a raw IP on purpose.** `ana-docker` (where LiteLLM runs)
|
||||
resolves no `.internal` names at all — its `/etc/resolv.conf` points at
|
||||
`1.1.1.1`/`1.0.0.1`, and the only reason the `ext-tts` backend resolves is a
|
||||
hand-pinned `extra_hosts: irv-ml1.nh3.internal:10.6.110.50` in the LiteLLM
|
||||
compose. Adding a second hosts entry would mean recreating the container and
|
||||
bouncing the gateway for every consumer; an IP costs nothing and cannot go
|
||||
stale silently. See the DNS follow-up in `persistent-memory.md`.
|
||||
|
||||
Aliases live in LiteLLM's **Postgres store** (`store_model_in_db: true`), not
|
||||
in `config.yaml` — that is where the `ext-tts` family lives too, and it means
|
||||
adding one needs no gateway restart. It also means `config.yaml` is not a
|
||||
complete picture of what the gateway serves: check `/v1/models` or
|
||||
`/model/info`, never just the file.
|
||||
|
||||
## Deploy
|
||||
|
||||
```bash
|
||||
ssh irv-ml1 '
|
||||
cd /opt/docker/compose/parakeet && \
|
||||
sed -i "s|v2-int8|v3-int8|" .env && \
|
||||
# Wipe the v2 weights so the entrypoint re-downloads v3 on next up:
|
||||
rm -f /worktank/parakeet/models/*.onnx /worktank/parakeet/models/tokens.txt && \
|
||||
docker compose up -d && \
|
||||
docker compose logs -f --tail=30
|
||||
'
|
||||
scripts/deploy-stack.sh fv-ml1 parakeet --compose
|
||||
# on fv-ml1, first time only:
|
||||
# cp .env.example .env # then edit
|
||||
docker compose -f /opt/docker/compose/parakeet/compose.yaml build
|
||||
docker compose -f /opt/docker/compose/parakeet/compose.yaml up -d
|
||||
```
|
||||
|
||||
## Upgrade sherpa-onnx or change the base image
|
||||
First boot downloads ~464 MiB of ONNX weights into `/tank/parakeet/models/`;
|
||||
the healthcheck's `start_period` is 300 s to cover it. Subsequent starts skip
|
||||
the download.
|
||||
|
||||
Bump `PARAKEET_TAG` in `.env` to force a rebuild of the local image
|
||||
after editing the `Dockerfile`, then:
|
||||
## Switching model variants
|
||||
|
||||
```bash
|
||||
scripts/deploy-stack.sh irv-ml1 parakeet
|
||||
ssh irv-ml1 'cd /opt/docker/compose/parakeet && docker compose build && docker compose up -d'
|
||||
```
|
||||
One line in `.env`, then `docker compose up -d` (not `restart` — the model URL
|
||||
is read by the entrypoint at container creation) and delete the old files from
|
||||
`/tank/parakeet/models/` so the entrypoint re-downloads:
|
||||
|
||||
Model files under `/worktank/parakeet/models/` are preserved across
|
||||
image rebuilds.
|
||||
- **v3** (default) — 25 European languages
|
||||
- **v2** — English only; slightly better on English-only material
|
||||
|
||||
## File layout
|
||||
|
||||
```
|
||||
stacks/parakeet/
|
||||
├── Dockerfile # CUDA 12.8 + cuDNN 9 base, sherpa-onnx-cu12 wheel
|
||||
├── app.py # FastAPI — ~60 lines
|
||||
├── entrypoint.sh # downloads model on first run, then uvicorn
|
||||
├── compose.yaml # one service, bind-mounts the models dir
|
||||
├── .env.example # template; real .env lives on the server
|
||||
└── README.md # this file
|
||||
```
|
||||
Both are ~460 MiB int8 tarballs from the same k2-fsa release page.
|
||||
|
||||
+30
-1
@@ -6,7 +6,7 @@ Load the encoder/decoder/joiner/tokens once at startup; serve:
|
||||
GET /healthz — used by the docker healthcheck
|
||||
|
||||
No VAD chunking, no Silero preprocessing — parakeet-tdt handles long-form natively
|
||||
and the int8 ONNX model on a 24 GB GPU eats everything we're likely to throw at it.
|
||||
and the int8 ONNX model is a rounding error against this host's 96 GB cards.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -14,6 +14,7 @@ from __future__ import annotations
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
@@ -59,8 +60,36 @@ def _load_recognizer() -> sherpa_onnx.OfflineRecognizer:
|
||||
)
|
||||
|
||||
|
||||
def _warm(rec: "sherpa_onnx.OfflineRecognizer") -> None:
|
||||
"""Decode one throwaway buffer before the server accepts traffic.
|
||||
|
||||
⚠ NOT an optimisation — it moves a 45 s stall out of the first real request.
|
||||
ONNX Runtime's CUDA EP compiles and autotunes its kernels lazily, on the first
|
||||
decode, and on this host (RTX PRO 6000 Blackwell, sm_120) that measured **45.7 s**
|
||||
while every subsequent call was ~0.48 s. Without this, the first caller after any
|
||||
container restart sees a 45 s hang and most clients — LiteLLM's default request
|
||||
timeout included — give up long before it returns, which reads as "the service is
|
||||
broken" rather than "the service is warming".
|
||||
|
||||
The healthcheck's `start_period` (300 s) is what makes paying it here safe.
|
||||
"""
|
||||
try:
|
||||
t0 = time.monotonic()
|
||||
stream = rec.create_stream()
|
||||
# 1 s of silence at 16 kHz: enough to force the full encoder/decoder/joiner
|
||||
# path to compile, cheap enough not to matter.
|
||||
stream.accept_waveform(16000, np.zeros(16000, dtype=np.float32))
|
||||
rec.decode_stream(stream)
|
||||
logger.info("warmup decode complete in %.1fs — CUDA kernels compiled", time.monotonic() - t0)
|
||||
except Exception:
|
||||
# A failed warmup must not stop the server: the model is loaded and real
|
||||
# requests would still work, just with the stall back on the first caller.
|
||||
logger.exception("warmup decode failed; first real request will absorb the stall")
|
||||
|
||||
|
||||
app = FastAPI(title="Parakeet ASR (sherpa-onnx)")
|
||||
recognizer = _load_recognizer()
|
||||
_warm(recognizer)
|
||||
|
||||
|
||||
def _decode(raw: bytes) -> str:
|
||||
|
||||
@@ -6,7 +6,18 @@
|
||||
# prebuilt int8 quantized Parakeet-TDT from k2-fsa — and wrote our own ~50-line
|
||||
# wrapper we own end-to-end.
|
||||
#
|
||||
# Model weights (~400 MB int8) download on first run via the entrypoint to
|
||||
# HOST: fv-ml1, GPU 3 (relocated from irv-ml1 2026-09-15). GPU 3 is the utility
|
||||
# card — the other three carry the vLLM serving seats and run 85-98% full, so a
|
||||
# seat placed anywhere else would fight them for VRAM.
|
||||
#
|
||||
# ⚠ GPU pin is `deploy.resources.reservations.devices[].device_ids`, the fleet
|
||||
# convention — NOT `runtime: nvidia` + NVIDIA_VISIBLE_DEVICES, and NOT
|
||||
# `count: all` (which is what the dead on-host stub did, and would have let this
|
||||
# tiny ASR seat see all four cards including the three that are full).
|
||||
# device_ids ["3"] presents that card as cuda:0 INSIDE the container, which is
|
||||
# what sherpa-onnx's CUDAExecutionProvider takes by default.
|
||||
#
|
||||
# Model weights (~460 MB int8) download on first run via the entrypoint to
|
||||
# ${PARAKEET_MODELS_DIR}/ (persistent host bind mount). Subsequent starts skip
|
||||
# the download.
|
||||
#
|
||||
@@ -25,11 +36,9 @@ services:
|
||||
dockerfile: Dockerfile
|
||||
container_name: parakeet
|
||||
restart: unless-stopped
|
||||
runtime: nvidia
|
||||
ports:
|
||||
- "${PARAKEET_BIND:-0.0.0.0}:${PARAKEET_PORT}:8000"
|
||||
environment:
|
||||
- NVIDIA_VISIBLE_DEVICES=0
|
||||
- MODEL_DIR=/models
|
||||
- MODEL_URL=${PARAKEET_MODEL_URL}
|
||||
- PROVIDER=${PARAKEET_PROVIDER:-cuda}
|
||||
@@ -37,17 +46,31 @@ services:
|
||||
- LOG_LEVEL=${PARAKEET_LOG_LEVEL:-INFO}
|
||||
volumes:
|
||||
- ${PARAKEET_MODELS_DIR}:/models
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids: ["${PARAKEET_GPU:-3}"]
|
||||
capabilities: [gpu]
|
||||
networks:
|
||||
- tnet
|
||||
healthcheck:
|
||||
# Image ships wget (apt) but not curl — use wget so the check actually runs.
|
||||
test: ["CMD-SHELL", "wget -q -O /dev/null http://localhost:8000/healthz || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
# First boot may include a ~400 MB model download.
|
||||
# First boot may include a ~460 MB model download.
|
||||
start_period: 300s
|
||||
labels:
|
||||
- homepage.group=AI - Audio Tools
|
||||
- homepage.name=Parakeet ASR
|
||||
- homepage.icon=mdi-microphone
|
||||
- homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (irv-ml1)
|
||||
- homepage.href=http://irv-ml1.nh3.internal:${PARAKEET_PORT}
|
||||
- homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (fv-ml1 GPU 3)
|
||||
- homepage.href=http://10.251.50.54:${PARAKEET_PORT}
|
||||
|
||||
networks:
|
||||
tnet:
|
||||
name: traefik-net
|
||||
external: true
|
||||
|
||||
Reference in New Issue
Block a user