feat(parakeet): stand up Parakeet STT on fv-ml1 GPU 3 + LiteLLM ext-stt/whisper-1

Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card
and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry
the vLLM seats at 84-95.5 GB of 96.

Changes:

- compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used
  `count: all`, which would have handed a 0.6B ASR seat all four cards);
  join traefik-net; port 8300; homepage href to the live FV address.
- .env.example: default to the v3 int8 model (25 European languages, 464 MiB)
  rather than English-only v2; models to /tank/parakeet/models.
- app.py: warm the recognizer at startup before uvicorn accepts traffic.

The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes
lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at
45.1s on a second container) against ~0.50s warm. A 45s first request is
indistinguishable from a hang and LiteLLM's default timeout abandons it long
before it returns. Decoding 1s of silence at load moves the cost inside the
healthcheck's 300s start_period; first real request after restart is now 0.65s.

Verification, because "provider=cuda" in the log is only an echo of the env var:
ORT falls back to CPU silently and still returns correct text, so the service
being up and the transcript being right establishes nothing. The discriminator is
a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS
sentence transcribes near-exactly (positive), 3s of digital silence returns
empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread
0.47-0.65s, single-stream, one clip: a smoke measurement with its harness
stated, not a benchmark.

Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1`
(OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's
Postgres store where the ext-tts family already lives — no gateway restart, and
config.yaml is consequently not a complete picture of what the gateway serves.
Both verified end to end.

The aliases use a raw IP deliberately: ana-docker resolves no .internal names at
all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a
hand-pinned extra_hosts entry. A second hosts entry would mean recreating the
container and bouncing the gateway for every consumer.

Also records the svos_miranda plugin validation pass and its structural findings,
and notes that the irv-ml1 parakeet is still running — there are two now, and
retiring the old one is the operator's call.

persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md

@@ -0,0 +1,86 @@
+# Parakeet STT on fv-ml1 GPU 3 (2026-09-15)
+
+Operator asked for an STT service on fv-ml1's utility GPU plus a LiteLLM alias.
+
+## What it is
+
+`stacks/parakeet/` — Parakeet-TDT 0.6B **v3** int8 ONNX (25 European languages,
+464 MiB) under sherpa-onnx, behind ~90 lines of FastAPI we own. Container
+`parakeet`, port **8300**, **GPU 3** pinned by `device_ids`. Image
+`local/parakeet:sherpa-onnx-v4` (5.09 GB).
+
+Not greenfield: the stack already existed, targeting irv-ml1. Retargeted rather
+than rewritten — the Ampere→Blackwell move was the only real question.
+
+## Why GPU 3
+
+GPU 0 = 84/96 GB, GPU 1 = 92.9/96, GPU 2 = 95.5/96 (the vLLM seats). **GPU 3 was
+at 2 MiB.** The dead on-host stub used `count: all`, which would have handed this
+seat all four cards; replaced with an explicit `device_ids: ["3"]` per the fleet
+convention. Inside the container the pinned card presents as `cuda:0`, which is
+what ORT's CUDA EP takes by default.
+
+## ⚠ The finding worth keeping: a 45-second first decode
+
+ONNX Runtime's CUDA EP compiles and autotunes lazily, on the **first decode**, not
+at session creation. On sm_120:
+
+| | measured |
+|---|---|
+| first decode, cold container | **45.7 s** (n=1), reproduced at **45.1 s** on a second container |
+| warm, 8.52 s clip | **0.50 s** median (n=5: 0.65 / 0.53 / 0.48 / 0.47 / 0.50) |
+
+≈17× realtime warm, single-stream, one 8.52 s clip, int8, GPU 3 idle otherwise.
+That is a smoke measurement with its harness stated, **not** a benchmark — no
+concurrency sweep, no length sweep, one clip.
+
+A 45 s first request is indistinguishable from a hang to any caller, and LiteLLM's
+default timeout would abandon it. `_warm()` in `app.py` now decodes 1 s of silence
+before uvicorn accepts traffic, so the cost lands inside the healthcheck's 300 s
+`start_period`. First real request after restart: **0.65 s**.
+
+## ⚠⚠ "provider=cuda" is not evidence the GPU is being used
+
+ORT's CUDA EP **falls back to CPU silently** — the process lives, answers 200, and
+returns *correct text*, just slowly. Our own log line `loading OfflineRecognizer
+(provider=cuda...)` merely echoes the env var and proves nothing.
+
+The discriminator that actually settles it:
+
+```
+nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv -i 3
+-> 1588301, /opt/venv/bin/python3, 922 MiB
+```
+
+Timing is **not** a sufficient check either — the int8 model is fast enough on a
+96-thread EPYC that a CPU fallback still looks brisk on short clips.
+
+Controls run, both directions:
+- **positive** — known TTS sentence in, near-exact transcript out (two word errors,
+  both attributable to the source audio: an inserted "um", "Foun Valley").
+- **null** — 3 s of digital silence → `{"text": ""}`. The instrument does not
+  manufacture signal.
+
+## LiteLLM
+
+Two aliases, both `mode: audio_transcription` → `http://10.251.50.54:8300/v1`:
+`ext-stt` (engine-neutral fleet name, mirrors `ext-tts`) and `whisper-1`
+(OpenAI-compatible drop-in). Both verified end-to-end through the gateway.
+
+Registered via `POST /model/new`, i.e. the **Postgres store**, not `config.yaml` —
+that is where the `ext-tts` family lives, and it needs no gateway restart.
+⚠ Corollary: `config.yaml` is NOT a complete picture of what the gateway serves
+(it lists 35 models; the gateway serves 40, and carries stale entries like
+`granite-4.1-8b`). Read `/v1/models` or `/model/info`, never just the file.
+
+⚠ **Raw IP on purpose** — see the ana-docker DNS row in the index.
+
+## Loose ends
+
+- The **irv-ml1 parakeet is still running** (healthz 200 on `100.64.0.6:8765`).
+  Two Parakeets now. Retiring the old one is the operator's call — not touched.
+- `/opt/docker/compose/parakeet` and `/tank/parakeet` normalised to `root:docker
+  2775`; the rest of fv-ml1's deploy tree is still `lkraven:lkraven` (it was not
+  part of the 5-host normalisation).
+- `servers/fv-ml1/README.md` is still broadly stale — it claims 2 GPUs and a
+  2026-07-22 stack list. Only the parakeet/GPU-3 rows were corrected.

persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md

@@ -0,0 +1,64 @@
+# svos_miranda Hermes plugin — validation pass (2026-09-15)
+
+svos-dev asked infra-ops to run `hermes plugins validate` → `doctor` → `compat`
+on `/home/lkraven/development/svos/hermes_plugin/` and report before enabling.
+Hermes Agent v0.21.1 (2026.9.7), local `b88e6776`, on nh3-dev.
+
+## The blocker (found, fixed by svos-dev at `c964e64`)
+
+Three absolute intra-package imports — `from hermes_plugin._vendored`, `.forward`,
+`.jwt` — pinned the package to its **source directory name**. The documented
+install renames it to `svos_miranda`, and Hermes loads directory plugins under the
+`hermes_plugins.<dir>` namespace; in neither case does a top-level `hermes_plugin`
+exist. Fix: three relative imports.
+
+⚠ **The harness hid this from three different readers.** My first `validate` passed
+the import only because my cwd was the SVOS repo root. svos-dev's test suite
+imports `hermes_plugin.*` from that same root, and their editable install resolves
+the name from anywhere on the box — it only reproduced for them once `sys.path` was
+stripped. Same class as `feedback_filters_that_silently_narrow_the_window`: the
+instrument carried the result.
+
+## ⚠ Two of the three commands CANNOT pass this plugin, ever
+
+Neither is fixable from the plugin side. Both are now documented in its README.
+
+- **`validate`** — two independent causes. Its `RecordingContext.get_config`
+  (`hermes_cli/plugin_validate.py:219-222`) returns the **default for every key**,
+  ignoring `config.yaml` entirely, so `dispatch_key` is always `""`. And its
+  `register_tool` returns `None`, which the plugin's INV-P6 guard correctly reads
+  as a name collision — so even with a key supplied it raises on the first tool.
+- **`doctor`** — runs `register()` under a **temp `HERMES_HOME`** with sockets
+  blocked, so no config exists there either.
+
+## Tool-level gotchas worth remembering
+
+- ⚠ **`hermes plugins doctor` exits 0 even when it prints ERROR.** Needs `--ci`.
+- ⚠ **`hermes plugins compat <nonexistent-path>` prints ✓ and exits 0.** A typo'd
+  path reads as a pass. (The instrument itself is sound — verified with a throwaway
+  plugin importing a real deprecated path, which it flagged with file:line, exit 1.)
+- ⚠ **`doctor`'s sandbox registry starts EMPTY — 0 entries, no built-ins.** So
+  doctor cannot detect tool-name collisions at all. `validate`'s separate static
+  "built-in tool collisions" check is what covers that.
+- The real `PluginContext.register_tool` (`hermes_cli/plugins.py:449-491`) returns
+  a truthy `PluginRegistration` on success — confirmed against the live runtime.
+
+## Roster verified another way
+
+Since neither command can supply config, a probe mirroring validate's context but
+returning real settings and a truthy handle gave: **8 tools** with
+`repo_read_enabled: true`, **7** with false or omitted, names matching
+`plugin.yaml` exactly, zero hooks/middleware/commands. All nine settings-validation
+controls (quoted booleans, `"90 s"`, zero/negative timeouts, empty/whitespace
+strings) raise errors naming their own key.
+
+## A false finding I caught on myself
+
+A probe registering `read_file` got back a `PluginRegistration` instead of the
+expected refusal — which looked like the plugin's collision reading was wrong. It
+was not: doctor's sandbox holds no built-ins, so nothing was claimed and **my
+positive case was not positive.** Reported as untested rather than as a finding.
+
+## Where enabling stands
+
+Blocked on the operator only. See the OPEN row in the index for the sequence.

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-09-15 ~01:15 PT (FV cross-site routing FIXED; mesh membership RETIRED for both fv-ml1 and nh3-dev — six nodes remain, each with a job; fv-ml1 carries a break-glass rejoin instead. Next session: drain svos-dev, then stand up an STT.)_
+_Last updated: 2026-09-15 ~01:45 PT (Parakeet STT LIVE on fv-ml1 GPU 3 + LiteLLM `ext-stt`/`whisper-1`; svos_miranda plugin validated and its blocker fixed by svos-dev — enabling is the operator's call.)_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -117,11 +117,25 @@ no longer deployed sidecars here. See Recent decisions.)
 
 _As of 2026-09-15 ~01:15 PT._
 
-### ⭐ NEXT SESSION'S NAMED WORK (operator, at snapshot time)
+### ⭐ BOTH NAMED JOBS CLOSED (2026-09-15 ~01:45 PT)
-1. **Drain svos-dev** — one althing message unread since 00:52 PT, pinged five times,
+1. **svos-dev drained** — four messages, thread closed from their end. Ran
-   never read. `/althing:inbox` first.
+   `hermes plugins validate/doctor/compat` on `svos_miranda`; found and A/B-proved a
-2. **Stand up an STT** (speech-to-text) service. Nothing started: no placement decided,
+   load blocker (absolute intra-package imports), which svos-dev fixed at `c964e64`.
-   no model chosen, no stack authored. Greenfield.
+   ⚠ **Enabling is still pending and is the OPERATOR'S call** — see the decision row below.
+2. **STT stood up** — Parakeet on fv-ml1 GPU 3, `ext-stt` / `whisper-1` in LiteLLM.
+   Not greenfield after all: `stacks/parakeet/` already existed (sherpa-onnx + our own
+   FastAPI wrapper, previously on irv-ml1) and was retargeted rather than rewritten.
+
+### ⏳ OPEN — svos_miranda enable (needs the operator)
+Everything technical is done. Enabling requires editing the operator's live
+`~/.hermes/config.yaml` (this session was guardrailed out of that write, then
+unblocked — but the go/no-go is his) and svos-dev restarts `:8770` after, which takes
+the SVOS board down until the roster verifies. Sequence when he says go: install to
+`~/.hermes/plugins/svos_miranda`; add `plugins.enabled` + `entries.svos_miranda.settings`
+(dispatch key from `secret get nh3-dev/svos/bifrost-dispatch-key`, `repo_read_enabled: true`);
+set `platform_toolsets["api_server"] = ["svos_miranda"]` and `agent.disabled_toolsets`;
+**re-derive that list after registration** from `GET http://127.0.0.1:8765/v1/toolsets`
+(28 rows now, `svos_miranda` joins once it registers); hand off to svos-dev for the restart.
 
 ### Fleet networking — closed out this session
 - **FV cross-site routing FIXED.** fv-ml1 reaches NH3/ESH/ANA/IRV/internet via outbound
@@ -161,6 +175,14 @@ hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`).
 
 ## Recent decisions
 
+- `[2026-09-15]` **Parakeet STT live on fv-ml1 GPU 3, behind LiteLLM `ext-stt` / `whisper-1`.** Retargeted the existing `stacks/parakeet/` (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ **ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s** — every later call ~0.5 s; a startup warmup in `app.py` now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was **verified by a process on GPU 3 (922 MiB), not by the `provider=cuda` log line**, because ORT falls back to CPU silently and still returns correct text. Silence → `""` (null control), known sentence → near-exact (positive control). → `persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md`
+
+- `[2026-09-15]` **`svos_miranda` Hermes plugin validated; found its load blocker.** Absolute intra-package imports (`from hermes_plugin.x`) could not resolve at the documented install name — fixed by svos-dev at `c964e64`. ⚠ **`hermes plugins validate` and `doctor` can NEVER pass this plugin**, by construction: validate's probe stub is config-blind AND returns `None` from `register_tool` (which the plugin's guard reads as a collision), and doctor runs under a temp `HERMES_HOME` with no config. ⚠ `doctor` exits **0** on ERROR (use `--ci`); `compat` reads a **nonexistent path as a pass**. Roster verified 8/7 by a probe supplying real settings. → `persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md`
+
+- `[2026-09-15]` ⚠ **ana-docker resolves NO `.internal` names** — its `/etc/resolv.conf` is `1.1.1.1`/`1.0.0.1`, not the fleet AdGuard. LiteLLM only reaches `irv-ml1.nh3.internal` because of a hand-pinned `extra_hosts` in its compose. New gateway aliases therefore use **raw IPs**; adding a hosts entry would mean recreating the container and bouncing the gateway for every consumer. Fleet-wide DNS fix is unowned.
+
+- `[2026-09-15]` ⚠ **Two Homepage ASR cards point at the dead wg0 lifeline `10.100.79.3`** (retired at the headscale cutover): the old irv-ml1 `parakeet` :8765 and `Speaches ASR` :8204. The irv-ml1 parakeet is **genuinely still running** (healthz 200 on `100.64.0.6` and `10.6.110.50`) — only its href is stale. So the fleet now has **two** Parakeets. Retiring the irv-ml1 one is the operator's call; not torn down.
+
 - `[2026-09-15]` **Mesh membership retired for fv-ml1 and nh3-dev — six nodes left, each with a job.** fv-ml1 gets break-glass rejoin instead of standing membership; nh3-dev's retirement also removed the nh3-scale masquerade exception it had required. Exactly one live reusable pre-auth key remains fleet-wide. → `persistent-memory.d/2026-09-15-fv-mesh-watchdog.md`
 
 - `[2026-09-15]` **FV cross-site routing fixed — one OPNsense outbound-NAT rule had been scoped to Anaheim only.** fv-ml1 now reaches NH3/ESH/IRV/ANA/mesh/internet; four rules, all `src=10.251.50.0/24`. The diagnostic signature is the valuable part: every layer looks correct and the discriminator is that *every other site pair works*. → `persistent-memory.d/2026-09-15-fv-cross-site-snat.md`

servers/fv-ml1/README.md

@@ -108,7 +108,16 @@ it on the card with room or evicting a dormant one first.
 seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`,
 `ms32-24b-angel`, `qwen3.5-122b`, `qwopus3.5-122b`, `qwen35-vl`, `qwen36-vl`,
 `qwen36-27b-aeon`, `qwen-image-bench`, `vibevoice`, `comfyui`, `kokoro`,
-`parakeet`, `vllm-qwen3`.
+`vllm-qwen3`.
+
+**GPU 3 — utility card:**
+
+| Container | Port | Serves | Notes |
+|-----------|------|--------|-------|
+| `parakeet` | 8300 | Parakeet-TDT 0.6B v3 int8 (25 languages) | ASR via sherpa-onnx, LiteLLM `ext-stt` / `whisper-1`. Relocated from irv-ml1 2026-09-15. `stacks/parakeet/`. |
+
+⚠ The other three cards run 85-98% full, so GPU 3 is where a new small seat goes
+until something bigger claims it.
 
 **Retired:**
 - `llama-swap` (former GGUF multiplexer on :9292) — replaced by dedicated

stacks/parakeet/.env.example

@@ -1,4 +1,4 @@
-# Parakeet ASR stack tunables. Copy to `.env` on irv-ml1 before deploying.
+# Parakeet ASR stack tunables. Copy to `.env` on fv-ml1 before deploying.
 #
 #   cp .env.example .env
 #   # edit as needed
@@ -7,29 +7,42 @@
 
 # Image tag. Bump when you change the Dockerfile / app.py so docker caches
 # cleanly.
-PARAKEET_TAG=sherpa-onnx-v2
+PARAKEET_TAG=sherpa-onnx-v4
 
-# Host port for the FastAPI server (container listens on 8000)
+# Which GPU to pin. fv-ml1 GPU 3 is the utility card — 0/1/2 carry the vLLM
-PARAKEET_PORT=8765
+# serving seats and sit at 85-98% VRAM, so this is the only one with room.
+# The container sees whichever card this names as cuda:0 internally.
+PARAKEET_GPU=3
 
-# Bind address. 0.0.0.0 exposes on all interfaces including the WG tunnel IP
+# Host port for the FastAPI server (container listens on 8000). 8300 is
-# (10.100.79.3). Use 127.0.0.1 to restrict to local-only.
+# fv-ml1's established parakeet port; the 80xx range belongs to the vLLM seats.
+PARAKEET_PORT=8300
+
+# Bind address. 0.0.0.0 exposes on all interfaces. Use 127.0.0.1 to restrict
+# to local-only — but LiteLLM on ana-docker reaches this over the LAN, so it
+# has to be 0.0.0.0 for the gateway alias to work.
 PARAKEET_BIND=0.0.0.0
 
 # Host path for the ONNX model files — encoder/decoder/joiner/tokens.txt.
 # Downloaded by the entrypoint on first run if absent. Must exist before
-# first `up` (directory, not files).
+# first `up` (directory, not files). Regenerable — exclude from restic.
-PARAKEET_MODELS_DIR=/worktank/parakeet/models
+PARAKEET_MODELS_DIR=/tank/parakeet/models
 
-# Which sherpa-onnx release tarball to fetch on first boot. Default is the
+# Which sherpa-onnx release tarball to fetch on first boot.
-# int8-quantized English-only v2 (~400 MB). Switch to the v3 tarball below
+#   v3 (default, 464 MiB) — 25 European languages
-# to cover 25 European languages at a similar size:
+#   v2                    — English only, swap the URL below
-#   https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8.tar.bz2
+# https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2
-PARAKEET_MODEL_URL=https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2
+PARAKEET_MODEL_URL=https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8.tar.bz2
 
-# ONNX Runtime execution provider. `cuda` uses the GPU (requires nvidia
+# ONNX Runtime execution provider. `cuda` uses the GPU (requires the nvidia
-# runtime + matching CUDA/cuDNN in the image). `cpu` falls back to CPU —
+# container runtime + matching CUDA/cuDNN in the image). `cpu` falls back to
-# fine for low-volume dev use; ~4-8× slower on this host.
+# CPU.
+#
+# ⚠ ORT's CUDA EP FALLS BACK TO CPU SILENTLY when it cannot initialise — the
+# server still answers 200 and still returns correct text, just slowly. So
+# `PROVIDER=cuda` is a REQUEST, not a guarantee, and the only honest check is
+# to watch `nvidia-smi` during a transcription and confirm a process appears on
+# the pinned card. See README § Verifying the GPU is actually in use.
 PARAKEET_PROVIDER=cuda
 
 # CPU threads per recognizer session. Irrelevant when provider=cuda;

stacks/parakeet/README.md

@@ -4,12 +4,26 @@ NVIDIA Parakeet-TDT 0.6B (int8 ONNX) served by our own thin FastAPI
 wrapper over [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)
 (ONNX Runtime + CUDA).
 
-**Server:** irv-ml1 (Irvine, WireGuard-only)
+**Server:** fv-ml1 (Fountain Valley, `10.251.50.54`) — moved from irv-ml1 2026-09-15
-**Port:** 8765 (container 8000)
+**Port:** 8300 (container 8000)
-**GPU:** both exposed (`NVIDIA_VISIBLE_DEVICES=all`); sherpa-onnx uses
+**GPU:** **3**, pinned explicitly via `device_ids` — the utility card
-whichever CUDA ExecutionProvider picks
+**Image:** `local/parakeet:sherpa-onnx-v4` — built from `Dockerfile` +
-**Image:** `local/parakeet:sherpa-onnx-v1` — built from `Dockerfile` +
 `app.py` + `entrypoint.sh` in this directory; **we own all the code**
+**Model:** `parakeet-tdt-0.6b-v3` int8, 25 European languages (~464 MiB)
+
+## Why GPU 3
+
+fv-ml1 has four RTX PRO 6000 Blackwell Max-Q (96 GB each). Three carry the
+vLLM serving seats and run 85–98 % full; GPU 3 is the utility card and was
+empty (2 MiB) at placement time. A 0.6 B int8 ASR model is a rounding error
+next to those seats, but it still has to go somewhere that is not fighting
+them for VRAM.
+
+⚠ The pin is `deploy.resources.reservations.devices[].device_ids: ["3"]`,
+the fleet convention — **not** `count: all`, which is what the dead on-host
+stub used and which would have handed this seat all four cards. Inside the
+container the pinned card presents as `cuda:0`, which is what sherpa-onnx's
+CUDA execution provider takes by default.
 
 ## Why not the FastAPI community wrappers
 
@@ -35,80 +49,93 @@ recognizer API is a three-line call.
 
 | Host path | Container path | Purpose | Restic? |
 |---|---|---|---|
-| `/worktank/parakeet/models/` | `/models` | ONNX encoder+decoder+joiner+tokens (~400 MB int8) | excluded (regenerable — re-downloads from the URL on first run if absent) |
+| `/tank/parakeet/models/` | `/models` | ONNX encoder+decoder+joiner+tokens (~464 MiB int8) | excluded (regenerable — re-downloads from the URL on first run if absent) |
 
-## First-time deploy on irv-ml1
+## ⚠ Verifying the GPU is actually in use
+
+**ONNX Runtime's CUDA execution provider falls back to CPU silently.** It logs
+a warning if you are looking, keeps the process alive, answers `200`, and
+returns *correct transcriptions* — just far slower. So `PROVIDER=cuda` in
+`.env` is a request, not a guarantee, and "the service is up and the text is
+right" does **not** establish that the GPU is doing the work.
+
+Blackwell is the reason this matters here rather than being pedantry: these
+cards are `sm_120`, newer than the compute capabilities ORT's prebuilt CUDA
+binaries have historically shipped kernels for, and the irv-ml1 host this
+stack came from was Ampere `sm_86`. The move is exactly the kind that turns a
+green service into a CPU service without a single error.
+
+The honest check is to watch the card while a transcription runs:
 
 ```bash
-# 1. Push compose + Dockerfile + app + entrypoint
+# on fv-ml1 — terminal 1
-scripts/deploy-stack.sh irv-ml1 parakeet
+watch -n0.2 'nvidia-smi --query-compute-apps=pid,process_name,used_memory \
+    --format=csv -i 3'
 
-# 2. Make sure the models dir exists (one-time, already done from the
+# terminal 2 — send real audio, not silence
-#    earlier Shadowfita deploy; this is idempotent)
+curl -s -F file=@sample.wav http://127.0.0.1:8300/v1/audio/transcriptions
-ssh -t irv-ml1 'sudo mkdir -p /worktank/parakeet/models && \
-                sudo chown -R lkraven:lkraven /worktank/parakeet'
-
-# 3. Build the image and bring up. First boot does a ~400 MB model
-#    download via the entrypoint; allow 1–2 minutes before /healthz
-#    flips healthy.
-ssh irv-ml1 '
-  cd /opt/docker/compose/parakeet && \
-  cp -n .env.example .env && \
-  docker compose config >/dev/null && \
-  docker compose build && \
-  docker compose up -d && \
-  docker compose logs -f --tail=30
-'
 ```
 
-## Smoke test
+A process must appear **on GPU 3** for the duration. If GPU 3 stays empty, the
+CUDA EP did not initialise and you are on CPU regardless of what `.env` says.
+Confirm with the container's own startup log, which names the providers ORT
+actually registered:
 
 ```bash
-# Over WG from the workstation
+docker logs parakeet 2>&1 | grep -i 'provider\|cuda\|onnxruntime'
-curl -F "file=@sample.wav" http://10.100.79.3:8765/transcribe
-# → {"text": "hello world"}
-
-# OpenAI-shape alias (for clients that only know /v1/audio/transcriptions)
-curl -F "file=@sample.wav" http://10.100.79.3:8765/v1/audio/transcriptions
 ```
 
-## Switching to the v3 (multilingual) model
+Timing alone is **not** sufficient evidence either way: the int8 model is fast
+enough on a 96-thread EPYC that a CPU fallback still looks brisk on short
+clips. Use the process check as the discriminator and treat throughput as a
+secondary signal.
 
-The env var `PARAKEET_MODEL_URL` picks the release tarball. To swap
+## LiteLLM alias
-from the English-only v2 to the 25-language v3:
+
+Reached fleet-wide through the gateway rather than by name, engine-neutral so
+the backend can be swapped without touching consumers — the same pattern as
+`ext-tts`:
+
+| alias | mode | backend |
+|---|---|---|
+| `ext-stt` | `audio_transcription` | `http://10.251.50.54:8300/v1` |
+| `whisper-1` | `audio_transcription` | same — OpenAI-compatible name so stock SDK clients work unchanged |
+
+⚠ **The alias uses a raw IP on purpose.** `ana-docker` (where LiteLLM runs)
+resolves no `.internal` names at all — its `/etc/resolv.conf` points at
+`1.1.1.1`/`1.0.0.1`, and the only reason the `ext-tts` backend resolves is a
+hand-pinned `extra_hosts: irv-ml1.nh3.internal:10.6.110.50` in the LiteLLM
+compose. Adding a second hosts entry would mean recreating the container and
+bouncing the gateway for every consumer; an IP costs nothing and cannot go
+stale silently. See the DNS follow-up in `persistent-memory.md`.
+
+Aliases live in LiteLLM's **Postgres store** (`store_model_in_db: true`), not
+in `config.yaml` — that is where the `ext-tts` family lives too, and it means
+adding one needs no gateway restart. It also means `config.yaml` is not a
+complete picture of what the gateway serves: check `/v1/models` or
+`/model/info`, never just the file.
+
+## Deploy
 
 ```bash
-ssh irv-ml1 '
+scripts/deploy-stack.sh fv-ml1 parakeet --compose
-  cd /opt/docker/compose/parakeet && \
+# on fv-ml1, first time only:
-  sed -i "s|v2-int8|v3-int8|" .env && \
+#   cp .env.example .env    # then edit
-  # Wipe the v2 weights so the entrypoint re-downloads v3 on next up:
+docker compose -f /opt/docker/compose/parakeet/compose.yaml build
-  rm -f /worktank/parakeet/models/*.onnx /worktank/parakeet/models/tokens.txt && \
+docker compose -f /opt/docker/compose/parakeet/compose.yaml up -d
-  docker compose up -d && \
-  docker compose logs -f --tail=30
-'
 ```
 
-## Upgrade sherpa-onnx or change the base image
+First boot downloads ~464 MiB of ONNX weights into `/tank/parakeet/models/`;
+the healthcheck's `start_period` is 300 s to cover it. Subsequent starts skip
+the download.
 
-Bump `PARAKEET_TAG` in `.env` to force a rebuild of the local image
+## Switching model variants
-after editing the `Dockerfile`, then:
 
-```bash
+One line in `.env`, then `docker compose up -d` (not `restart` — the model URL
-scripts/deploy-stack.sh irv-ml1 parakeet
+is read by the entrypoint at container creation) and delete the old files from
-ssh irv-ml1 'cd /opt/docker/compose/parakeet && docker compose build && docker compose up -d'
+`/tank/parakeet/models/` so the entrypoint re-downloads:
-```
 
-Model files under `/worktank/parakeet/models/` are preserved across
+- **v3** (default) — 25 European languages
-image rebuilds.
+- **v2** — English only; slightly better on English-only material
 
-## File layout
+Both are ~460 MiB int8 tarballs from the same k2-fsa release page.
-
-```
-stacks/parakeet/
-├── Dockerfile           # CUDA 12.8 + cuDNN 9 base, sherpa-onnx-cu12 wheel
-├── app.py               # FastAPI — ~60 lines
-├── entrypoint.sh        # downloads model on first run, then uvicorn
-├── compose.yaml         # one service, bind-mounts the models dir
-├── .env.example         # template; real .env lives on the server
-└── README.md            # this file
-```

stacks/parakeet/app.py

@@ -6,7 +6,7 @@ Load the encoder/decoder/joiner/tokens once at startup; serve:
   GET  /healthz                      — used by the docker healthcheck
 
 No VAD chunking, no Silero preprocessing — parakeet-tdt handles long-form natively
-and the int8 ONNX model on a 24 GB GPU eats everything we're likely to throw at it.
+and the int8 ONNX model is a rounding error against this host's 96 GB cards.
 """
 
 from __future__ import annotations
@@ -14,6 +14,7 @@ from __future__ import annotations
 import io
 import logging
 import os
+import time
 from pathlib import Path
 
 import numpy as np
@@ -59,8 +60,36 @@ def _load_recognizer() -> sherpa_onnx.OfflineRecognizer:
     )
 
 
+def _warm(rec: "sherpa_onnx.OfflineRecognizer") -> None:
+    """Decode one throwaway buffer before the server accepts traffic.
+
+    ⚠ NOT an optimisation — it moves a 45 s stall out of the first real request.
+    ONNX Runtime's CUDA EP compiles and autotunes its kernels lazily, on the first
+    decode, and on this host (RTX PRO 6000 Blackwell, sm_120) that measured **45.7 s**
+    while every subsequent call was ~0.48 s. Without this, the first caller after any
+    container restart sees a 45 s hang and most clients — LiteLLM's default request
+    timeout included — give up long before it returns, which reads as "the service is
+    broken" rather than "the service is warming".
+
+    The healthcheck's `start_period` (300 s) is what makes paying it here safe.
+    """
+    try:
+        t0 = time.monotonic()
+        stream = rec.create_stream()
+        # 1 s of silence at 16 kHz: enough to force the full encoder/decoder/joiner
+        # path to compile, cheap enough not to matter.
+        stream.accept_waveform(16000, np.zeros(16000, dtype=np.float32))
+        rec.decode_stream(stream)
+        logger.info("warmup decode complete in %.1fs — CUDA kernels compiled", time.monotonic() - t0)
+    except Exception:
+        # A failed warmup must not stop the server: the model is loaded and real
+        # requests would still work, just with the stall back on the first caller.
+        logger.exception("warmup decode failed; first real request will absorb the stall")
+
+
 app = FastAPI(title="Parakeet ASR (sherpa-onnx)")
 recognizer = _load_recognizer()
+_warm(recognizer)
 
 
 def _decode(raw: bytes) -> str:

stacks/parakeet/compose.yaml

@@ -6,7 +6,18 @@
 # prebuilt int8 quantized Parakeet-TDT from k2-fsa — and wrote our own ~50-line
 # wrapper we own end-to-end.
 #
-# Model weights (~400 MB int8) download on first run via the entrypoint to
+# HOST: fv-ml1, GPU 3 (relocated from irv-ml1 2026-09-15). GPU 3 is the utility
+# card — the other three carry the vLLM serving seats and run 85-98% full, so a
+# seat placed anywhere else would fight them for VRAM.
+#
+# ⚠ GPU pin is `deploy.resources.reservations.devices[].device_ids`, the fleet
+# convention — NOT `runtime: nvidia` + NVIDIA_VISIBLE_DEVICES, and NOT
+# `count: all` (which is what the dead on-host stub did, and would have let this
+# tiny ASR seat see all four cards including the three that are full).
+# device_ids ["3"] presents that card as cuda:0 INSIDE the container, which is
+# what sherpa-onnx's CUDAExecutionProvider takes by default.
+#
+# Model weights (~460 MB int8) download on first run via the entrypoint to
 # ${PARAKEET_MODELS_DIR}/ (persistent host bind mount). Subsequent starts skip
 # the download.
 #
@@ -25,11 +36,9 @@ services:
       dockerfile: Dockerfile
     container_name: parakeet
     restart: unless-stopped
-    runtime: nvidia
     ports:
       - "${PARAKEET_BIND:-0.0.0.0}:${PARAKEET_PORT}:8000"
     environment:
-      - NVIDIA_VISIBLE_DEVICES=0
       - MODEL_DIR=/models
       - MODEL_URL=${PARAKEET_MODEL_URL}
       - PROVIDER=${PARAKEET_PROVIDER:-cuda}
@@ -37,17 +46,31 @@ services:
       - LOG_LEVEL=${PARAKEET_LOG_LEVEL:-INFO}
     volumes:
       - ${PARAKEET_MODELS_DIR}:/models
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids: ["${PARAKEET_GPU:-3}"]
+              capabilities: [gpu]
+    networks:
+      - tnet
     healthcheck:
       # Image ships wget (apt) but not curl — use wget so the check actually runs.
       test: ["CMD-SHELL", "wget -q -O /dev/null http://localhost:8000/healthz || exit 1"]
       interval: 30s
       timeout: 10s
       retries: 3
-      # First boot may include a ~400 MB model download.
+      # First boot may include a ~460 MB model download.
       start_period: 300s
     labels:
       - homepage.group=AI - Audio Tools
       - homepage.name=Parakeet ASR
       - homepage.icon=mdi-microphone
-      - homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (irv-ml1)
+      - homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (fv-ml1 GPU 3)
-      - homepage.href=http://irv-ml1.nh3.internal:${PARAKEET_PORT}
+      - homepage.href=http://10.251.50.54:${PARAKEET_PORT}
+
+networks:
+  tnet:
+    name: traefik-net
+    external: true