From b9b14b5baf14762a0221f4c3394a8c85b18d0395 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Tue, 15 Sep 2026 01:41:41 -0700 Subject: [PATCH] feat(parakeet): stand up Parakeet STT on fv-ml1 GPU 3 + LiteLLM ext-stt/whisper-1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry the vLLM seats at 84-95.5 GB of 96. Changes: - compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used `count: all`, which would have handed a 0.6B ASR seat all four cards); join traefik-net; port 8300; homepage href to the live FV address. - .env.example: default to the v3 int8 model (25 European languages, 464 MiB) rather than English-only v2; models to /tank/parakeet/models. - app.py: warm the recognizer at startup before uvicorn accepts traffic. The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at 45.1s on a second container) against ~0.50s warm. A 45s first request is indistinguishable from a hang and LiteLLM's default timeout abandons it long before it returns. Decoding 1s of silence at load moves the cost inside the healthcheck's 300s start_period; first real request after restart is now 0.65s. Verification, because "provider=cuda" in the log is only an echo of the env var: ORT falls back to CPU silently and still returns correct text, so the service being up and the transcript being right establishes nothing. The discriminator is a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS sentence transcribes near-exactly (positive), 3s of digital silence returns empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread 0.47-0.65s, single-stream, one clip: a smoke measurement with its harness stated, not a benchmark. Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1` (OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's Postgres store where the ext-tts family already lives — no gateway restart, and config.yaml is consequently not a complete picture of what the gateway serves. Both verified end to end. The aliases use a raw IP deliberately: ana-docker resolves no .internal names at all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a hand-pinned extra_hosts entry. A second hosts entry would mean recreating the container and bouncing the gateway for every consumer. Also records the svos_miranda plugin validation pass and its structural findings, and notes that the irv-ml1 parakeet is still running — there are two now, and retiring the old one is the operator's call. --- .../2026-09-15-parakeet-stt-fv-ml1.md | 86 ++++++++++ ...26-09-15-svos-miranda-plugin-validation.md | 64 ++++++++ persistent-memory.md | 34 +++- servers/fv-ml1/README.md | 11 +- stacks/parakeet/.env.example | 45 ++++-- stacks/parakeet/README.md | 153 ++++++++++-------- stacks/parakeet/app.py | 31 +++- stacks/parakeet/compose.yaml | 35 +++- 8 files changed, 366 insertions(+), 93 deletions(-) create mode 100644 persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md create mode 100644 persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md diff --git a/persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md b/persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md new file mode 100644 index 0000000..0eefdd7 --- /dev/null +++ b/persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md @@ -0,0 +1,86 @@ +# Parakeet STT on fv-ml1 GPU 3 (2026-09-15) + +Operator asked for an STT service on fv-ml1's utility GPU plus a LiteLLM alias. + +## What it is + +`stacks/parakeet/` — Parakeet-TDT 0.6B **v3** int8 ONNX (25 European languages, +464 MiB) under sherpa-onnx, behind ~90 lines of FastAPI we own. Container +`parakeet`, port **8300**, **GPU 3** pinned by `device_ids`. Image +`local/parakeet:sherpa-onnx-v4` (5.09 GB). + +Not greenfield: the stack already existed, targeting irv-ml1. Retargeted rather +than rewritten — the Ampere→Blackwell move was the only real question. + +## Why GPU 3 + +GPU 0 = 84/96 GB, GPU 1 = 92.9/96, GPU 2 = 95.5/96 (the vLLM seats). **GPU 3 was +at 2 MiB.** The dead on-host stub used `count: all`, which would have handed this +seat all four cards; replaced with an explicit `device_ids: ["3"]` per the fleet +convention. Inside the container the pinned card presents as `cuda:0`, which is +what ORT's CUDA EP takes by default. + +## ⚠ The finding worth keeping: a 45-second first decode + +ONNX Runtime's CUDA EP compiles and autotunes lazily, on the **first decode**, not +at session creation. On sm_120: + +| | measured | +|---|---| +| first decode, cold container | **45.7 s** (n=1), reproduced at **45.1 s** on a second container | +| warm, 8.52 s clip | **0.50 s** median (n=5: 0.65 / 0.53 / 0.48 / 0.47 / 0.50) | + +≈17× realtime warm, single-stream, one 8.52 s clip, int8, GPU 3 idle otherwise. +That is a smoke measurement with its harness stated, **not** a benchmark — no +concurrency sweep, no length sweep, one clip. + +A 45 s first request is indistinguishable from a hang to any caller, and LiteLLM's +default timeout would abandon it. `_warm()` in `app.py` now decodes 1 s of silence +before uvicorn accepts traffic, so the cost lands inside the healthcheck's 300 s +`start_period`. First real request after restart: **0.65 s**. + +## ⚠⚠ "provider=cuda" is not evidence the GPU is being used + +ORT's CUDA EP **falls back to CPU silently** — the process lives, answers 200, and +returns *correct text*, just slowly. Our own log line `loading OfflineRecognizer +(provider=cuda...)` merely echoes the env var and proves nothing. + +The discriminator that actually settles it: + +``` +nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv -i 3 +-> 1588301, /opt/venv/bin/python3, 922 MiB +``` + +Timing is **not** a sufficient check either — the int8 model is fast enough on a +96-thread EPYC that a CPU fallback still looks brisk on short clips. + +Controls run, both directions: +- **positive** — known TTS sentence in, near-exact transcript out (two word errors, + both attributable to the source audio: an inserted "um", "Foun Valley"). +- **null** — 3 s of digital silence → `{"text": ""}`. The instrument does not + manufacture signal. + +## LiteLLM + +Two aliases, both `mode: audio_transcription` → `http://10.251.50.54:8300/v1`: +`ext-stt` (engine-neutral fleet name, mirrors `ext-tts`) and `whisper-1` +(OpenAI-compatible drop-in). Both verified end-to-end through the gateway. + +Registered via `POST /model/new`, i.e. the **Postgres store**, not `config.yaml` — +that is where the `ext-tts` family lives, and it needs no gateway restart. +⚠ Corollary: `config.yaml` is NOT a complete picture of what the gateway serves +(it lists 35 models; the gateway serves 40, and carries stale entries like +`granite-4.1-8b`). Read `/v1/models` or `/model/info`, never just the file. + +⚠ **Raw IP on purpose** — see the ana-docker DNS row in the index. + +## Loose ends + +- The **irv-ml1 parakeet is still running** (healthz 200 on `100.64.0.6:8765`). + Two Parakeets now. Retiring the old one is the operator's call — not touched. +- `/opt/docker/compose/parakeet` and `/tank/parakeet` normalised to `root:docker + 2775`; the rest of fv-ml1's deploy tree is still `lkraven:lkraven` (it was not + part of the 5-host normalisation). +- `servers/fv-ml1/README.md` is still broadly stale — it claims 2 GPUs and a + 2026-07-22 stack list. Only the parakeet/GPU-3 rows were corrected. diff --git a/persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md b/persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md new file mode 100644 index 0000000..edd68c2 --- /dev/null +++ b/persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md @@ -0,0 +1,64 @@ +# svos_miranda Hermes plugin — validation pass (2026-09-15) + +svos-dev asked infra-ops to run `hermes plugins validate` → `doctor` → `compat` +on `/home/lkraven/development/svos/hermes_plugin/` and report before enabling. +Hermes Agent v0.21.1 (2026.9.7), local `b88e6776`, on nh3-dev. + +## The blocker (found, fixed by svos-dev at `c964e64`) + +Three absolute intra-package imports — `from hermes_plugin._vendored`, `.forward`, +`.jwt` — pinned the package to its **source directory name**. The documented +install renames it to `svos_miranda`, and Hermes loads directory plugins under the +`hermes_plugins.` namespace; in neither case does a top-level `hermes_plugin` +exist. Fix: three relative imports. + +⚠ **The harness hid this from three different readers.** My first `validate` passed +the import only because my cwd was the SVOS repo root. svos-dev's test suite +imports `hermes_plugin.*` from that same root, and their editable install resolves +the name from anywhere on the box — it only reproduced for them once `sys.path` was +stripped. Same class as `feedback_filters_that_silently_narrow_the_window`: the +instrument carried the result. + +## ⚠ Two of the three commands CANNOT pass this plugin, ever + +Neither is fixable from the plugin side. Both are now documented in its README. + +- **`validate`** — two independent causes. Its `RecordingContext.get_config` + (`hermes_cli/plugin_validate.py:219-222`) returns the **default for every key**, + ignoring `config.yaml` entirely, so `dispatch_key` is always `""`. And its + `register_tool` returns `None`, which the plugin's INV-P6 guard correctly reads + as a name collision — so even with a key supplied it raises on the first tool. +- **`doctor`** — runs `register()` under a **temp `HERMES_HOME`** with sockets + blocked, so no config exists there either. + +## Tool-level gotchas worth remembering + +- ⚠ **`hermes plugins doctor` exits 0 even when it prints ERROR.** Needs `--ci`. +- ⚠ **`hermes plugins compat ` prints ✓ and exits 0.** A typo'd + path reads as a pass. (The instrument itself is sound — verified with a throwaway + plugin importing a real deprecated path, which it flagged with file:line, exit 1.) +- ⚠ **`doctor`'s sandbox registry starts EMPTY — 0 entries, no built-ins.** So + doctor cannot detect tool-name collisions at all. `validate`'s separate static + "built-in tool collisions" check is what covers that. +- The real `PluginContext.register_tool` (`hermes_cli/plugins.py:449-491`) returns + a truthy `PluginRegistration` on success — confirmed against the live runtime. + +## Roster verified another way + +Since neither command can supply config, a probe mirroring validate's context but +returning real settings and a truthy handle gave: **8 tools** with +`repo_read_enabled: true`, **7** with false or omitted, names matching +`plugin.yaml` exactly, zero hooks/middleware/commands. All nine settings-validation +controls (quoted booleans, `"90 s"`, zero/negative timeouts, empty/whitespace +strings) raise errors naming their own key. + +## A false finding I caught on myself + +A probe registering `read_file` got back a `PluginRegistration` instead of the +expected refusal — which looked like the plugin's collision reading was wrong. It +was not: doctor's sandbox holds no built-ins, so nothing was claimed and **my +positive case was not positive.** Reported as untested rather than as a finding. + +## Where enabling stands + +Blocked on the operator only. See the OPEN row in the index for the sequence. diff --git a/persistent-memory.md b/persistent-memory.md index d7d8b56..862b46f 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-15 ~01:15 PT (FV cross-site routing FIXED; mesh membership RETIRED for both fv-ml1 and nh3-dev — six nodes remain, each with a job; fv-ml1 carries a break-glass rejoin instead. Next session: drain svos-dev, then stand up an STT.)_ +_Last updated: 2026-09-15 ~01:45 PT (Parakeet STT LIVE on fv-ml1 GPU 3 + LiteLLM `ext-stt`/`whisper-1`; svos_miranda plugin validated and its blocker fixed by svos-dev — enabling is the operator's call.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -117,11 +117,25 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-09-15 ~01:15 PT._ -### ⭐ NEXT SESSION'S NAMED WORK (operator, at snapshot time) -1. **Drain svos-dev** — one althing message unread since 00:52 PT, pinged five times, - never read. `/althing:inbox` first. -2. **Stand up an STT** (speech-to-text) service. Nothing started: no placement decided, - no model chosen, no stack authored. Greenfield. +### ⭐ BOTH NAMED JOBS CLOSED (2026-09-15 ~01:45 PT) +1. **svos-dev drained** — four messages, thread closed from their end. Ran + `hermes plugins validate/doctor/compat` on `svos_miranda`; found and A/B-proved a + load blocker (absolute intra-package imports), which svos-dev fixed at `c964e64`. + ⚠ **Enabling is still pending and is the OPERATOR'S call** — see the decision row below. +2. **STT stood up** — Parakeet on fv-ml1 GPU 3, `ext-stt` / `whisper-1` in LiteLLM. + Not greenfield after all: `stacks/parakeet/` already existed (sherpa-onnx + our own + FastAPI wrapper, previously on irv-ml1) and was retargeted rather than rewritten. + +### ⏳ OPEN — svos_miranda enable (needs the operator) +Everything technical is done. Enabling requires editing the operator's live +`~/.hermes/config.yaml` (this session was guardrailed out of that write, then +unblocked — but the go/no-go is his) and svos-dev restarts `:8770` after, which takes +the SVOS board down until the roster verifies. Sequence when he says go: install to +`~/.hermes/plugins/svos_miranda`; add `plugins.enabled` + `entries.svos_miranda.settings` +(dispatch key from `secret get nh3-dev/svos/bifrost-dispatch-key`, `repo_read_enabled: true`); +set `platform_toolsets["api_server"] = ["svos_miranda"]` and `agent.disabled_toolsets`; +**re-derive that list after registration** from `GET http://127.0.0.1:8765/v1/toolsets` +(28 rows now, `svos_miranda` joins once it registers); hand off to svos-dev for the restart. ### Fleet networking — closed out this session - **FV cross-site routing FIXED.** fv-ml1 reaches NH3/ESH/ANA/IRV/internet via outbound @@ -161,6 +175,14 @@ hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`). ## Recent decisions +- `[2026-09-15]` **Parakeet STT live on fv-ml1 GPU 3, behind LiteLLM `ext-stt` / `whisper-1`.** Retargeted the existing `stacks/parakeet/` (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ **ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s** — every later call ~0.5 s; a startup warmup in `app.py` now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was **verified by a process on GPU 3 (922 MiB), not by the `provider=cuda` log line**, because ORT falls back to CPU silently and still returns correct text. Silence → `""` (null control), known sentence → near-exact (positive control). → `persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md` + +- `[2026-09-15]` **`svos_miranda` Hermes plugin validated; found its load blocker.** Absolute intra-package imports (`from hermes_plugin.x`) could not resolve at the documented install name — fixed by svos-dev at `c964e64`. ⚠ **`hermes plugins validate` and `doctor` can NEVER pass this plugin**, by construction: validate's probe stub is config-blind AND returns `None` from `register_tool` (which the plugin's guard reads as a collision), and doctor runs under a temp `HERMES_HOME` with no config. ⚠ `doctor` exits **0** on ERROR (use `--ci`); `compat` reads a **nonexistent path as a pass**. Roster verified 8/7 by a probe supplying real settings. → `persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md` + +- `[2026-09-15]` ⚠ **ana-docker resolves NO `.internal` names** — its `/etc/resolv.conf` is `1.1.1.1`/`1.0.0.1`, not the fleet AdGuard. LiteLLM only reaches `irv-ml1.nh3.internal` because of a hand-pinned `extra_hosts` in its compose. New gateway aliases therefore use **raw IPs**; adding a hosts entry would mean recreating the container and bouncing the gateway for every consumer. Fleet-wide DNS fix is unowned. + +- `[2026-09-15]` ⚠ **Two Homepage ASR cards point at the dead wg0 lifeline `10.100.79.3`** (retired at the headscale cutover): the old irv-ml1 `parakeet` :8765 and `Speaches ASR` :8204. The irv-ml1 parakeet is **genuinely still running** (healthz 200 on `100.64.0.6` and `10.6.110.50`) — only its href is stale. So the fleet now has **two** Parakeets. Retiring the irv-ml1 one is the operator's call; not torn down. + - `[2026-09-15]` **Mesh membership retired for fv-ml1 and nh3-dev — six nodes left, each with a job.** fv-ml1 gets break-glass rejoin instead of standing membership; nh3-dev's retirement also removed the nh3-scale masquerade exception it had required. Exactly one live reusable pre-auth key remains fleet-wide. → `persistent-memory.d/2026-09-15-fv-mesh-watchdog.md` - `[2026-09-15]` **FV cross-site routing fixed — one OPNsense outbound-NAT rule had been scoped to Anaheim only.** fv-ml1 now reaches NH3/ESH/IRV/ANA/mesh/internet; four rules, all `src=10.251.50.0/24`. The diagnostic signature is the valuable part: every layer looks correct and the discriminator is that *every other site pair works*. → `persistent-memory.d/2026-09-15-fv-cross-site-snat.md` diff --git a/servers/fv-ml1/README.md b/servers/fv-ml1/README.md index 95b47d8..cc91281 100644 --- a/servers/fv-ml1/README.md +++ b/servers/fv-ml1/README.md @@ -108,7 +108,16 @@ it on the card with room or evicting a dormant one first. seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`, `ms32-24b-angel`, `qwen3.5-122b`, `qwopus3.5-122b`, `qwen35-vl`, `qwen36-vl`, `qwen36-27b-aeon`, `qwen-image-bench`, `vibevoice`, `comfyui`, `kokoro`, -`parakeet`, `vllm-qwen3`. +`vllm-qwen3`. + +**GPU 3 — utility card:** + +| Container | Port | Serves | Notes | +|-----------|------|--------|-------| +| `parakeet` | 8300 | Parakeet-TDT 0.6B v3 int8 (25 languages) | ASR via sherpa-onnx, LiteLLM `ext-stt` / `whisper-1`. Relocated from irv-ml1 2026-09-15. `stacks/parakeet/`. | + +⚠ The other three cards run 85-98% full, so GPU 3 is where a new small seat goes +until something bigger claims it. **Retired:** - `llama-swap` (former GGUF multiplexer on :9292) — replaced by dedicated diff --git a/stacks/parakeet/.env.example b/stacks/parakeet/.env.example index 67da72d..ab7339f 100644 --- a/stacks/parakeet/.env.example +++ b/stacks/parakeet/.env.example @@ -1,4 +1,4 @@ -# Parakeet ASR stack tunables. Copy to `.env` on irv-ml1 before deploying. +# Parakeet ASR stack tunables. Copy to `.env` on fv-ml1 before deploying. # # cp .env.example .env # # edit as needed @@ -7,29 +7,42 @@ # Image tag. Bump when you change the Dockerfile / app.py so docker caches # cleanly. -PARAKEET_TAG=sherpa-onnx-v2 +PARAKEET_TAG=sherpa-onnx-v4 -# Host port for the FastAPI server (container listens on 8000) -PARAKEET_PORT=8765 +# Which GPU to pin. fv-ml1 GPU 3 is the utility card — 0/1/2 carry the vLLM +# serving seats and sit at 85-98% VRAM, so this is the only one with room. +# The container sees whichever card this names as cuda:0 internally. +PARAKEET_GPU=3 -# Bind address. 0.0.0.0 exposes on all interfaces including the WG tunnel IP -# (10.100.79.3). Use 127.0.0.1 to restrict to local-only. +# Host port for the FastAPI server (container listens on 8000). 8300 is +# fv-ml1's established parakeet port; the 80xx range belongs to the vLLM seats. +PARAKEET_PORT=8300 + +# Bind address. 0.0.0.0 exposes on all interfaces. Use 127.0.0.1 to restrict +# to local-only — but LiteLLM on ana-docker reaches this over the LAN, so it +# has to be 0.0.0.0 for the gateway alias to work. PARAKEET_BIND=0.0.0.0 # Host path for the ONNX model files — encoder/decoder/joiner/tokens.txt. # Downloaded by the entrypoint on first run if absent. Must exist before -# first `up` (directory, not files). -PARAKEET_MODELS_DIR=/worktank/parakeet/models +# first `up` (directory, not files). Regenerable — exclude from restic. +PARAKEET_MODELS_DIR=/tank/parakeet/models -# Which sherpa-onnx release tarball to fetch on first boot. Default is the -# int8-quantized English-only v2 (~400 MB). Switch to the v3 tarball below -# to cover 25 European languages at a similar size: -# https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8.tar.bz2 -PARAKEET_MODEL_URL=https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2 +# Which sherpa-onnx release tarball to fetch on first boot. +# v3 (default, 464 MiB) — 25 European languages +# v2 — English only, swap the URL below +# https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2 +PARAKEET_MODEL_URL=https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8.tar.bz2 -# ONNX Runtime execution provider. `cuda` uses the GPU (requires nvidia -# runtime + matching CUDA/cuDNN in the image). `cpu` falls back to CPU — -# fine for low-volume dev use; ~4-8× slower on this host. +# ONNX Runtime execution provider. `cuda` uses the GPU (requires the nvidia +# container runtime + matching CUDA/cuDNN in the image). `cpu` falls back to +# CPU. +# +# ⚠ ORT's CUDA EP FALLS BACK TO CPU SILENTLY when it cannot initialise — the +# server still answers 200 and still returns correct text, just slowly. So +# `PROVIDER=cuda` is a REQUEST, not a guarantee, and the only honest check is +# to watch `nvidia-smi` during a transcription and confirm a process appears on +# the pinned card. See README § Verifying the GPU is actually in use. PARAKEET_PROVIDER=cuda # CPU threads per recognizer session. Irrelevant when provider=cuda; diff --git a/stacks/parakeet/README.md b/stacks/parakeet/README.md index 261ec65..91f245a 100644 --- a/stacks/parakeet/README.md +++ b/stacks/parakeet/README.md @@ -4,12 +4,26 @@ NVIDIA Parakeet-TDT 0.6B (int8 ONNX) served by our own thin FastAPI wrapper over [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) (ONNX Runtime + CUDA). -**Server:** irv-ml1 (Irvine, WireGuard-only) -**Port:** 8765 (container 8000) -**GPU:** both exposed (`NVIDIA_VISIBLE_DEVICES=all`); sherpa-onnx uses -whichever CUDA ExecutionProvider picks -**Image:** `local/parakeet:sherpa-onnx-v1` — built from `Dockerfile` + +**Server:** fv-ml1 (Fountain Valley, `10.251.50.54`) — moved from irv-ml1 2026-09-15 +**Port:** 8300 (container 8000) +**GPU:** **3**, pinned explicitly via `device_ids` — the utility card +**Image:** `local/parakeet:sherpa-onnx-v4` — built from `Dockerfile` + `app.py` + `entrypoint.sh` in this directory; **we own all the code** +**Model:** `parakeet-tdt-0.6b-v3` int8, 25 European languages (~464 MiB) + +## Why GPU 3 + +fv-ml1 has four RTX PRO 6000 Blackwell Max-Q (96 GB each). Three carry the +vLLM serving seats and run 85–98 % full; GPU 3 is the utility card and was +empty (2 MiB) at placement time. A 0.6 B int8 ASR model is a rounding error +next to those seats, but it still has to go somewhere that is not fighting +them for VRAM. + +⚠ The pin is `deploy.resources.reservations.devices[].device_ids: ["3"]`, +the fleet convention — **not** `count: all`, which is what the dead on-host +stub used and which would have handed this seat all four cards. Inside the +container the pinned card presents as `cuda:0`, which is what sherpa-onnx's +CUDA execution provider takes by default. ## Why not the FastAPI community wrappers @@ -35,80 +49,93 @@ recognizer API is a three-line call. | Host path | Container path | Purpose | Restic? | |---|---|---|---| -| `/worktank/parakeet/models/` | `/models` | ONNX encoder+decoder+joiner+tokens (~400 MB int8) | excluded (regenerable — re-downloads from the URL on first run if absent) | +| `/tank/parakeet/models/` | `/models` | ONNX encoder+decoder+joiner+tokens (~464 MiB int8) | excluded (regenerable — re-downloads from the URL on first run if absent) | -## First-time deploy on irv-ml1 +## ⚠ Verifying the GPU is actually in use + +**ONNX Runtime's CUDA execution provider falls back to CPU silently.** It logs +a warning if you are looking, keeps the process alive, answers `200`, and +returns *correct transcriptions* — just far slower. So `PROVIDER=cuda` in +`.env` is a request, not a guarantee, and "the service is up and the text is +right" does **not** establish that the GPU is doing the work. + +Blackwell is the reason this matters here rather than being pedantry: these +cards are `sm_120`, newer than the compute capabilities ORT's prebuilt CUDA +binaries have historically shipped kernels for, and the irv-ml1 host this +stack came from was Ampere `sm_86`. The move is exactly the kind that turns a +green service into a CPU service without a single error. + +The honest check is to watch the card while a transcription runs: ```bash -# 1. Push compose + Dockerfile + app + entrypoint -scripts/deploy-stack.sh irv-ml1 parakeet +# on fv-ml1 — terminal 1 +watch -n0.2 'nvidia-smi --query-compute-apps=pid,process_name,used_memory \ + --format=csv -i 3' -# 2. Make sure the models dir exists (one-time, already done from the -# earlier Shadowfita deploy; this is idempotent) -ssh -t irv-ml1 'sudo mkdir -p /worktank/parakeet/models && \ - sudo chown -R lkraven:lkraven /worktank/parakeet' - -# 3. Build the image and bring up. First boot does a ~400 MB model -# download via the entrypoint; allow 1–2 minutes before /healthz -# flips healthy. -ssh irv-ml1 ' - cd /opt/docker/compose/parakeet && \ - cp -n .env.example .env && \ - docker compose config >/dev/null && \ - docker compose build && \ - docker compose up -d && \ - docker compose logs -f --tail=30 -' +# terminal 2 — send real audio, not silence +curl -s -F file=@sample.wav http://127.0.0.1:8300/v1/audio/transcriptions ``` -## Smoke test +A process must appear **on GPU 3** for the duration. If GPU 3 stays empty, the +CUDA EP did not initialise and you are on CPU regardless of what `.env` says. +Confirm with the container's own startup log, which names the providers ORT +actually registered: ```bash -# Over WG from the workstation -curl -F "file=@sample.wav" http://10.100.79.3:8765/transcribe -# → {"text": "hello world"} - -# OpenAI-shape alias (for clients that only know /v1/audio/transcriptions) -curl -F "file=@sample.wav" http://10.100.79.3:8765/v1/audio/transcriptions +docker logs parakeet 2>&1 | grep -i 'provider\|cuda\|onnxruntime' ``` -## Switching to the v3 (multilingual) model +Timing alone is **not** sufficient evidence either way: the int8 model is fast +enough on a 96-thread EPYC that a CPU fallback still looks brisk on short +clips. Use the process check as the discriminator and treat throughput as a +secondary signal. -The env var `PARAKEET_MODEL_URL` picks the release tarball. To swap -from the English-only v2 to the 25-language v3: +## LiteLLM alias + +Reached fleet-wide through the gateway rather than by name, engine-neutral so +the backend can be swapped without touching consumers — the same pattern as +`ext-tts`: + +| alias | mode | backend | +|---|---|---| +| `ext-stt` | `audio_transcription` | `http://10.251.50.54:8300/v1` | +| `whisper-1` | `audio_transcription` | same — OpenAI-compatible name so stock SDK clients work unchanged | + +⚠ **The alias uses a raw IP on purpose.** `ana-docker` (where LiteLLM runs) +resolves no `.internal` names at all — its `/etc/resolv.conf` points at +`1.1.1.1`/`1.0.0.1`, and the only reason the `ext-tts` backend resolves is a +hand-pinned `extra_hosts: irv-ml1.nh3.internal:10.6.110.50` in the LiteLLM +compose. Adding a second hosts entry would mean recreating the container and +bouncing the gateway for every consumer; an IP costs nothing and cannot go +stale silently. See the DNS follow-up in `persistent-memory.md`. + +Aliases live in LiteLLM's **Postgres store** (`store_model_in_db: true`), not +in `config.yaml` — that is where the `ext-tts` family lives too, and it means +adding one needs no gateway restart. It also means `config.yaml` is not a +complete picture of what the gateway serves: check `/v1/models` or +`/model/info`, never just the file. + +## Deploy ```bash -ssh irv-ml1 ' - cd /opt/docker/compose/parakeet && \ - sed -i "s|v2-int8|v3-int8|" .env && \ - # Wipe the v2 weights so the entrypoint re-downloads v3 on next up: - rm -f /worktank/parakeet/models/*.onnx /worktank/parakeet/models/tokens.txt && \ - docker compose up -d && \ - docker compose logs -f --tail=30 -' +scripts/deploy-stack.sh fv-ml1 parakeet --compose +# on fv-ml1, first time only: +# cp .env.example .env # then edit +docker compose -f /opt/docker/compose/parakeet/compose.yaml build +docker compose -f /opt/docker/compose/parakeet/compose.yaml up -d ``` -## Upgrade sherpa-onnx or change the base image +First boot downloads ~464 MiB of ONNX weights into `/tank/parakeet/models/`; +the healthcheck's `start_period` is 300 s to cover it. Subsequent starts skip +the download. -Bump `PARAKEET_TAG` in `.env` to force a rebuild of the local image -after editing the `Dockerfile`, then: +## Switching model variants -```bash -scripts/deploy-stack.sh irv-ml1 parakeet -ssh irv-ml1 'cd /opt/docker/compose/parakeet && docker compose build && docker compose up -d' -``` +One line in `.env`, then `docker compose up -d` (not `restart` — the model URL +is read by the entrypoint at container creation) and delete the old files from +`/tank/parakeet/models/` so the entrypoint re-downloads: -Model files under `/worktank/parakeet/models/` are preserved across -image rebuilds. +- **v3** (default) — 25 European languages +- **v2** — English only; slightly better on English-only material -## File layout - -``` -stacks/parakeet/ -├── Dockerfile # CUDA 12.8 + cuDNN 9 base, sherpa-onnx-cu12 wheel -├── app.py # FastAPI — ~60 lines -├── entrypoint.sh # downloads model on first run, then uvicorn -├── compose.yaml # one service, bind-mounts the models dir -├── .env.example # template; real .env lives on the server -└── README.md # this file -``` +Both are ~460 MiB int8 tarballs from the same k2-fsa release page. diff --git a/stacks/parakeet/app.py b/stacks/parakeet/app.py index b8357ab..a8b9dbf 100644 --- a/stacks/parakeet/app.py +++ b/stacks/parakeet/app.py @@ -6,7 +6,7 @@ Load the encoder/decoder/joiner/tokens once at startup; serve: GET /healthz — used by the docker healthcheck No VAD chunking, no Silero preprocessing — parakeet-tdt handles long-form natively -and the int8 ONNX model on a 24 GB GPU eats everything we're likely to throw at it. +and the int8 ONNX model is a rounding error against this host's 96 GB cards. """ from __future__ import annotations @@ -14,6 +14,7 @@ from __future__ import annotations import io import logging import os +import time from pathlib import Path import numpy as np @@ -59,8 +60,36 @@ def _load_recognizer() -> sherpa_onnx.OfflineRecognizer: ) +def _warm(rec: "sherpa_onnx.OfflineRecognizer") -> None: + """Decode one throwaway buffer before the server accepts traffic. + + ⚠ NOT an optimisation — it moves a 45 s stall out of the first real request. + ONNX Runtime's CUDA EP compiles and autotunes its kernels lazily, on the first + decode, and on this host (RTX PRO 6000 Blackwell, sm_120) that measured **45.7 s** + while every subsequent call was ~0.48 s. Without this, the first caller after any + container restart sees a 45 s hang and most clients — LiteLLM's default request + timeout included — give up long before it returns, which reads as "the service is + broken" rather than "the service is warming". + + The healthcheck's `start_period` (300 s) is what makes paying it here safe. + """ + try: + t0 = time.monotonic() + stream = rec.create_stream() + # 1 s of silence at 16 kHz: enough to force the full encoder/decoder/joiner + # path to compile, cheap enough not to matter. + stream.accept_waveform(16000, np.zeros(16000, dtype=np.float32)) + rec.decode_stream(stream) + logger.info("warmup decode complete in %.1fs — CUDA kernels compiled", time.monotonic() - t0) + except Exception: + # A failed warmup must not stop the server: the model is loaded and real + # requests would still work, just with the stall back on the first caller. + logger.exception("warmup decode failed; first real request will absorb the stall") + + app = FastAPI(title="Parakeet ASR (sherpa-onnx)") recognizer = _load_recognizer() +_warm(recognizer) def _decode(raw: bytes) -> str: diff --git a/stacks/parakeet/compose.yaml b/stacks/parakeet/compose.yaml index ae8aef8..b078fe8 100644 --- a/stacks/parakeet/compose.yaml +++ b/stacks/parakeet/compose.yaml @@ -6,7 +6,18 @@ # prebuilt int8 quantized Parakeet-TDT from k2-fsa — and wrote our own ~50-line # wrapper we own end-to-end. # -# Model weights (~400 MB int8) download on first run via the entrypoint to +# HOST: fv-ml1, GPU 3 (relocated from irv-ml1 2026-09-15). GPU 3 is the utility +# card — the other three carry the vLLM serving seats and run 85-98% full, so a +# seat placed anywhere else would fight them for VRAM. +# +# ⚠ GPU pin is `deploy.resources.reservations.devices[].device_ids`, the fleet +# convention — NOT `runtime: nvidia` + NVIDIA_VISIBLE_DEVICES, and NOT +# `count: all` (which is what the dead on-host stub did, and would have let this +# tiny ASR seat see all four cards including the three that are full). +# device_ids ["3"] presents that card as cuda:0 INSIDE the container, which is +# what sherpa-onnx's CUDAExecutionProvider takes by default. +# +# Model weights (~460 MB int8) download on first run via the entrypoint to # ${PARAKEET_MODELS_DIR}/ (persistent host bind mount). Subsequent starts skip # the download. # @@ -25,11 +36,9 @@ services: dockerfile: Dockerfile container_name: parakeet restart: unless-stopped - runtime: nvidia ports: - "${PARAKEET_BIND:-0.0.0.0}:${PARAKEET_PORT}:8000" environment: - - NVIDIA_VISIBLE_DEVICES=0 - MODEL_DIR=/models - MODEL_URL=${PARAKEET_MODEL_URL} - PROVIDER=${PARAKEET_PROVIDER:-cuda} @@ -37,17 +46,31 @@ services: - LOG_LEVEL=${PARAKEET_LOG_LEVEL:-INFO} volumes: - ${PARAKEET_MODELS_DIR}:/models + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["${PARAKEET_GPU:-3}"] + capabilities: [gpu] + networks: + - tnet healthcheck: # Image ships wget (apt) but not curl — use wget so the check actually runs. test: ["CMD-SHELL", "wget -q -O /dev/null http://localhost:8000/healthz || exit 1"] interval: 30s timeout: 10s retries: 3 - # First boot may include a ~400 MB model download. + # First boot may include a ~460 MB model download. start_period: 300s labels: - homepage.group=AI - Audio Tools - homepage.name=Parakeet ASR - homepage.icon=mdi-microphone - - homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (irv-ml1) - - homepage.href=http://irv-ml1.nh3.internal:${PARAKEET_PORT} + - homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (fv-ml1 GPU 3) + - homepage.href=http://10.251.50.54:${PARAKEET_PORT} + +networks: + tnet: + name: traefik-net + external: true