diff --git a/persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md b/persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md
new file mode 100644
index 0000000..0eefdd7
--- /dev/null
+++ b/persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md
@@ -0,0 +1,86 @@
+# Parakeet STT on fv-ml1 GPU 3 (2026-09-15)
+
+Operator asked for an STT service on fv-ml1's utility GPU plus a LiteLLM alias.
+
+## What it is
+
+`stacks/parakeet/` — Parakeet-TDT 0.6B **v3** int8 ONNX (25 European languages,
+464 MiB) under sherpa-onnx, behind ~90 lines of FastAPI we own. Container
+`parakeet`, port **8300**, **GPU 3** pinned by `device_ids`. Image
+`local/parakeet:sherpa-onnx-v4` (5.09 GB).
+
+Not greenfield: the stack already existed, targeting irv-ml1. Retargeted rather
+than rewritten — the Ampere→Blackwell move was the only real question.
+
+## Why GPU 3
+
+GPU 0 = 84/96 GB, GPU 1 = 92.9/96, GPU 2 = 95.5/96 (the vLLM seats). **GPU 3 was
+at 2 MiB.** The dead on-host stub used `count: all`, which would have handed this
+seat all four cards; replaced with an explicit `device_ids: ["3"]` per the fleet
+convention. Inside the container the pinned card presents as `cuda:0`, which is
+what ORT's CUDA EP takes by default.
+
+## ⚠ The finding worth keeping: a 45-second first decode
+
+ONNX Runtime's CUDA EP compiles and autotunes lazily, on the **first decode**, not
+at session creation. On sm_120:
+
+| | measured |
+|---|---|
+| first decode, cold container | **45.7 s** (n=1), reproduced at **45.1 s** on a second container |
+| warm, 8.52 s clip | **0.50 s** median (n=5: 0.65 / 0.53 / 0.48 / 0.47 / 0.50) |
+
+≈17× realtime warm, single-stream, one 8.52 s clip, int8, GPU 3 idle otherwise.
+That is a smoke measurement with its harness stated, **not** a benchmark — no
+concurrency sweep, no length sweep, one clip.
+
+A 45 s first request is indistinguishable from a hang to any caller, and LiteLLM's
+default timeout would abandon it. `_warm()` in `app.py` now decodes 1 s of silence
+before uvicorn accepts traffic, so the cost lands inside the healthcheck's 300 s
+`start_period`. First real request after restart: **0.65 s**.
+
+## ⚠⚠ "provider=cuda" is not evidence the GPU is being used
+
+ORT's CUDA EP **falls back to CPU silently** — the process lives, answers 200, and
+returns *correct text*, just slowly. Our own log line `loading OfflineRecognizer
+(provider=cuda...)` merely echoes the env var and proves nothing.
+
+The discriminator that actually settles it:
+
+```
+nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv -i 3
+-> 1588301, /opt/venv/bin/python3, 922 MiB
+```
+
+Timing is **not** a sufficient check either — the int8 model is fast enough on a
+96-thread EPYC that a CPU fallback still looks brisk on short clips.
+
+Controls run, both directions:
+- **positive** — known TTS sentence in, near-exact transcript out (two word errors,
+ both attributable to the source audio: an inserted "um", "Foun Valley").
+- **null** — 3 s of digital silence → `{"text": ""}`. The instrument does not
+ manufacture signal.
+
+## LiteLLM
+
+Two aliases, both `mode: audio_transcription` → `http://10.251.50.54:8300/v1`:
+`ext-stt` (engine-neutral fleet name, mirrors `ext-tts`) and `whisper-1`
+(OpenAI-compatible drop-in). Both verified end-to-end through the gateway.
+
+Registered via `POST /model/new`, i.e. the **Postgres store**, not `config.yaml` —
+that is where the `ext-tts` family lives, and it needs no gateway restart.
+⚠ Corollary: `config.yaml` is NOT a complete picture of what the gateway serves
+(it lists 35 models; the gateway serves 40, and carries stale entries like
+`granite-4.1-8b`). Read `/v1/models` or `/model/info`, never just the file.
+
+⚠ **Raw IP on purpose** — see the ana-docker DNS row in the index.
+
+## Loose ends
+
+- The **irv-ml1 parakeet is still running** (healthz 200 on `100.64.0.6:8765`).
+ Two Parakeets now. Retiring the old one is the operator's call — not touched.
+- `/opt/docker/compose/parakeet` and `/tank/parakeet` normalised to `root:docker
+ 2775`; the rest of fv-ml1's deploy tree is still `lkraven:lkraven` (it was not
+ part of the 5-host normalisation).
+- `servers/fv-ml1/README.md` is still broadly stale — it claims 2 GPUs and a
+ 2026-07-22 stack list. Only the parakeet/GPU-3 rows were corrected.
diff --git a/persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md b/persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md
new file mode 100644
index 0000000..edd68c2
--- /dev/null
+++ b/persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md
@@ -0,0 +1,64 @@
+# svos_miranda Hermes plugin — validation pass (2026-09-15)
+
+svos-dev asked infra-ops to run `hermes plugins validate` → `doctor` → `compat`
+on `/home/lkraven/development/svos/hermes_plugin/` and report before enabling.
+Hermes Agent v0.21.1 (2026.9.7), local `b88e6776`, on nh3-dev.
+
+## The blocker (found, fixed by svos-dev at `c964e64`)
+
+Three absolute intra-package imports — `from hermes_plugin._vendored`, `.forward`,
+`.jwt` — pinned the package to its **source directory name**. The documented
+install renames it to `svos_miranda`, and Hermes loads directory plugins under the
+`hermes_plugins.
` namespace; in neither case does a top-level `hermes_plugin`
+exist. Fix: three relative imports.
+
+⚠ **The harness hid this from three different readers.** My first `validate` passed
+the import only because my cwd was the SVOS repo root. svos-dev's test suite
+imports `hermes_plugin.*` from that same root, and their editable install resolves
+the name from anywhere on the box — it only reproduced for them once `sys.path` was
+stripped. Same class as `feedback_filters_that_silently_narrow_the_window`: the
+instrument carried the result.
+
+## ⚠ Two of the three commands CANNOT pass this plugin, ever
+
+Neither is fixable from the plugin side. Both are now documented in its README.
+
+- **`validate`** — two independent causes. Its `RecordingContext.get_config`
+ (`hermes_cli/plugin_validate.py:219-222`) returns the **default for every key**,
+ ignoring `config.yaml` entirely, so `dispatch_key` is always `""`. And its
+ `register_tool` returns `None`, which the plugin's INV-P6 guard correctly reads
+ as a name collision — so even with a key supplied it raises on the first tool.
+- **`doctor`** — runs `register()` under a **temp `HERMES_HOME`** with sockets
+ blocked, so no config exists there either.
+
+## Tool-level gotchas worth remembering
+
+- ⚠ **`hermes plugins doctor` exits 0 even when it prints ERROR.** Needs `--ci`.
+- ⚠ **`hermes plugins compat ` prints ✓ and exits 0.** A typo'd
+ path reads as a pass. (The instrument itself is sound — verified with a throwaway
+ plugin importing a real deprecated path, which it flagged with file:line, exit 1.)
+- ⚠ **`doctor`'s sandbox registry starts EMPTY — 0 entries, no built-ins.** So
+ doctor cannot detect tool-name collisions at all. `validate`'s separate static
+ "built-in tool collisions" check is what covers that.
+- The real `PluginContext.register_tool` (`hermes_cli/plugins.py:449-491`) returns
+ a truthy `PluginRegistration` on success — confirmed against the live runtime.
+
+## Roster verified another way
+
+Since neither command can supply config, a probe mirroring validate's context but
+returning real settings and a truthy handle gave: **8 tools** with
+`repo_read_enabled: true`, **7** with false or omitted, names matching
+`plugin.yaml` exactly, zero hooks/middleware/commands. All nine settings-validation
+controls (quoted booleans, `"90 s"`, zero/negative timeouts, empty/whitespace
+strings) raise errors naming their own key.
+
+## A false finding I caught on myself
+
+A probe registering `read_file` got back a `PluginRegistration` instead of the
+expected refusal — which looked like the plugin's collision reading was wrong. It
+was not: doctor's sandbox holds no built-ins, so nothing was claimed and **my
+positive case was not positive.** Reported as untested rather than as a finding.
+
+## Where enabling stands
+
+Blocked on the operator only. See the OPEN row in the index for the sequence.
diff --git a/persistent-memory.md b/persistent-memory.md
index d7d8b56..862b46f 100644
--- a/persistent-memory.md
+++ b/persistent-memory.md
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
-_Last updated: 2026-09-15 ~01:15 PT (FV cross-site routing FIXED; mesh membership RETIRED for both fv-ml1 and nh3-dev — six nodes remain, each with a job; fv-ml1 carries a break-glass rejoin instead. Next session: drain svos-dev, then stand up an STT.)_
+_Last updated: 2026-09-15 ~01:45 PT (Parakeet STT LIVE on fv-ml1 GPU 3 + LiteLLM `ext-stt`/`whisper-1`; svos_miranda plugin validated and its blocker fixed by svos-dev — enabling is the operator's call.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -117,11 +117,25 @@ no longer deployed sidecars here. See Recent decisions.)
_As of 2026-09-15 ~01:15 PT._
-### ⭐ NEXT SESSION'S NAMED WORK (operator, at snapshot time)
-1. **Drain svos-dev** — one althing message unread since 00:52 PT, pinged five times,
- never read. `/althing:inbox` first.
-2. **Stand up an STT** (speech-to-text) service. Nothing started: no placement decided,
- no model chosen, no stack authored. Greenfield.
+### ⭐ BOTH NAMED JOBS CLOSED (2026-09-15 ~01:45 PT)
+1. **svos-dev drained** — four messages, thread closed from their end. Ran
+ `hermes plugins validate/doctor/compat` on `svos_miranda`; found and A/B-proved a
+ load blocker (absolute intra-package imports), which svos-dev fixed at `c964e64`.
+ ⚠ **Enabling is still pending and is the OPERATOR'S call** — see the decision row below.
+2. **STT stood up** — Parakeet on fv-ml1 GPU 3, `ext-stt` / `whisper-1` in LiteLLM.
+ Not greenfield after all: `stacks/parakeet/` already existed (sherpa-onnx + our own
+ FastAPI wrapper, previously on irv-ml1) and was retargeted rather than rewritten.
+
+### ⏳ OPEN — svos_miranda enable (needs the operator)
+Everything technical is done. Enabling requires editing the operator's live
+`~/.hermes/config.yaml` (this session was guardrailed out of that write, then
+unblocked — but the go/no-go is his) and svos-dev restarts `:8770` after, which takes
+the SVOS board down until the roster verifies. Sequence when he says go: install to
+`~/.hermes/plugins/svos_miranda`; add `plugins.enabled` + `entries.svos_miranda.settings`
+(dispatch key from `secret get nh3-dev/svos/bifrost-dispatch-key`, `repo_read_enabled: true`);
+set `platform_toolsets["api_server"] = ["svos_miranda"]` and `agent.disabled_toolsets`;
+**re-derive that list after registration** from `GET http://127.0.0.1:8765/v1/toolsets`
+(28 rows now, `svos_miranda` joins once it registers); hand off to svos-dev for the restart.
### Fleet networking — closed out this session
- **FV cross-site routing FIXED.** fv-ml1 reaches NH3/ESH/ANA/IRV/internet via outbound
@@ -161,6 +175,14 @@ hardened for ha-dev (`d1769ed` ff); `kb` KB-search tool (`68fa80f`).
## Recent decisions
+- `[2026-09-15]` **Parakeet STT live on fv-ml1 GPU 3, behind LiteLLM `ext-stt` / `whisper-1`.** Retargeted the existing `stacks/parakeet/` (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ **ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s** — every later call ~0.5 s; a startup warmup in `app.py` now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was **verified by a process on GPU 3 (922 MiB), not by the `provider=cuda` log line**, because ORT falls back to CPU silently and still returns correct text. Silence → `""` (null control), known sentence → near-exact (positive control). → `persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md`
+
+- `[2026-09-15]` **`svos_miranda` Hermes plugin validated; found its load blocker.** Absolute intra-package imports (`from hermes_plugin.x`) could not resolve at the documented install name — fixed by svos-dev at `c964e64`. ⚠ **`hermes plugins validate` and `doctor` can NEVER pass this plugin**, by construction: validate's probe stub is config-blind AND returns `None` from `register_tool` (which the plugin's guard reads as a collision), and doctor runs under a temp `HERMES_HOME` with no config. ⚠ `doctor` exits **0** on ERROR (use `--ci`); `compat` reads a **nonexistent path as a pass**. Roster verified 8/7 by a probe supplying real settings. → `persistent-memory.d/2026-09-15-svos-miranda-plugin-validation.md`
+
+- `[2026-09-15]` ⚠ **ana-docker resolves NO `.internal` names** — its `/etc/resolv.conf` is `1.1.1.1`/`1.0.0.1`, not the fleet AdGuard. LiteLLM only reaches `irv-ml1.nh3.internal` because of a hand-pinned `extra_hosts` in its compose. New gateway aliases therefore use **raw IPs**; adding a hosts entry would mean recreating the container and bouncing the gateway for every consumer. Fleet-wide DNS fix is unowned.
+
+- `[2026-09-15]` ⚠ **Two Homepage ASR cards point at the dead wg0 lifeline `10.100.79.3`** (retired at the headscale cutover): the old irv-ml1 `parakeet` :8765 and `Speaches ASR` :8204. The irv-ml1 parakeet is **genuinely still running** (healthz 200 on `100.64.0.6` and `10.6.110.50`) — only its href is stale. So the fleet now has **two** Parakeets. Retiring the irv-ml1 one is the operator's call; not torn down.
+
- `[2026-09-15]` **Mesh membership retired for fv-ml1 and nh3-dev — six nodes left, each with a job.** fv-ml1 gets break-glass rejoin instead of standing membership; nh3-dev's retirement also removed the nh3-scale masquerade exception it had required. Exactly one live reusable pre-auth key remains fleet-wide. → `persistent-memory.d/2026-09-15-fv-mesh-watchdog.md`
- `[2026-09-15]` **FV cross-site routing fixed — one OPNsense outbound-NAT rule had been scoped to Anaheim only.** fv-ml1 now reaches NH3/ESH/IRV/ANA/mesh/internet; four rules, all `src=10.251.50.0/24`. The diagnostic signature is the valuable part: every layer looks correct and the discriminator is that *every other site pair works*. → `persistent-memory.d/2026-09-15-fv-cross-site-snat.md`
diff --git a/servers/fv-ml1/README.md b/servers/fv-ml1/README.md
index 95b47d8..cc91281 100644
--- a/servers/fv-ml1/README.md
+++ b/servers/fv-ml1/README.md
@@ -108,7 +108,16 @@ it on the card with room or evicting a dormant one first.
seats, safe to leave: `mistral-medium-3.5`, `mistral-small-4(-heretic)`,
`ms32-24b-angel`, `qwen3.5-122b`, `qwopus3.5-122b`, `qwen35-vl`, `qwen36-vl`,
`qwen36-27b-aeon`, `qwen-image-bench`, `vibevoice`, `comfyui`, `kokoro`,
-`parakeet`, `vllm-qwen3`.
+`vllm-qwen3`.
+
+**GPU 3 — utility card:**
+
+| Container | Port | Serves | Notes |
+|-----------|------|--------|-------|
+| `parakeet` | 8300 | Parakeet-TDT 0.6B v3 int8 (25 languages) | ASR via sherpa-onnx, LiteLLM `ext-stt` / `whisper-1`. Relocated from irv-ml1 2026-09-15. `stacks/parakeet/`. |
+
+⚠ The other three cards run 85-98% full, so GPU 3 is where a new small seat goes
+until something bigger claims it.
**Retired:**
- `llama-swap` (former GGUF multiplexer on :9292) — replaced by dedicated
diff --git a/stacks/parakeet/.env.example b/stacks/parakeet/.env.example
index 67da72d..ab7339f 100644
--- a/stacks/parakeet/.env.example
+++ b/stacks/parakeet/.env.example
@@ -1,4 +1,4 @@
-# Parakeet ASR stack tunables. Copy to `.env` on irv-ml1 before deploying.
+# Parakeet ASR stack tunables. Copy to `.env` on fv-ml1 before deploying.
#
# cp .env.example .env
# # edit as needed
@@ -7,29 +7,42 @@
# Image tag. Bump when you change the Dockerfile / app.py so docker caches
# cleanly.
-PARAKEET_TAG=sherpa-onnx-v2
+PARAKEET_TAG=sherpa-onnx-v4
-# Host port for the FastAPI server (container listens on 8000)
-PARAKEET_PORT=8765
+# Which GPU to pin. fv-ml1 GPU 3 is the utility card — 0/1/2 carry the vLLM
+# serving seats and sit at 85-98% VRAM, so this is the only one with room.
+# The container sees whichever card this names as cuda:0 internally.
+PARAKEET_GPU=3
-# Bind address. 0.0.0.0 exposes on all interfaces including the WG tunnel IP
-# (10.100.79.3). Use 127.0.0.1 to restrict to local-only.
+# Host port for the FastAPI server (container listens on 8000). 8300 is
+# fv-ml1's established parakeet port; the 80xx range belongs to the vLLM seats.
+PARAKEET_PORT=8300
+
+# Bind address. 0.0.0.0 exposes on all interfaces. Use 127.0.0.1 to restrict
+# to local-only — but LiteLLM on ana-docker reaches this over the LAN, so it
+# has to be 0.0.0.0 for the gateway alias to work.
PARAKEET_BIND=0.0.0.0
# Host path for the ONNX model files — encoder/decoder/joiner/tokens.txt.
# Downloaded by the entrypoint on first run if absent. Must exist before
-# first `up` (directory, not files).
-PARAKEET_MODELS_DIR=/worktank/parakeet/models
+# first `up` (directory, not files). Regenerable — exclude from restic.
+PARAKEET_MODELS_DIR=/tank/parakeet/models
-# Which sherpa-onnx release tarball to fetch on first boot. Default is the
-# int8-quantized English-only v2 (~400 MB). Switch to the v3 tarball below
-# to cover 25 European languages at a similar size:
-# https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8.tar.bz2
-PARAKEET_MODEL_URL=https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2
+# Which sherpa-onnx release tarball to fetch on first boot.
+# v3 (default, 464 MiB) — 25 European languages
+# v2 — English only, swap the URL below
+# https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2
+PARAKEET_MODEL_URL=https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8.tar.bz2
-# ONNX Runtime execution provider. `cuda` uses the GPU (requires nvidia
-# runtime + matching CUDA/cuDNN in the image). `cpu` falls back to CPU —
-# fine for low-volume dev use; ~4-8× slower on this host.
+# ONNX Runtime execution provider. `cuda` uses the GPU (requires the nvidia
+# container runtime + matching CUDA/cuDNN in the image). `cpu` falls back to
+# CPU.
+#
+# ⚠ ORT's CUDA EP FALLS BACK TO CPU SILENTLY when it cannot initialise — the
+# server still answers 200 and still returns correct text, just slowly. So
+# `PROVIDER=cuda` is a REQUEST, not a guarantee, and the only honest check is
+# to watch `nvidia-smi` during a transcription and confirm a process appears on
+# the pinned card. See README § Verifying the GPU is actually in use.
PARAKEET_PROVIDER=cuda
# CPU threads per recognizer session. Irrelevant when provider=cuda;
diff --git a/stacks/parakeet/README.md b/stacks/parakeet/README.md
index 261ec65..91f245a 100644
--- a/stacks/parakeet/README.md
+++ b/stacks/parakeet/README.md
@@ -4,12 +4,26 @@ NVIDIA Parakeet-TDT 0.6B (int8 ONNX) served by our own thin FastAPI
wrapper over [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)
(ONNX Runtime + CUDA).
-**Server:** irv-ml1 (Irvine, WireGuard-only)
-**Port:** 8765 (container 8000)
-**GPU:** both exposed (`NVIDIA_VISIBLE_DEVICES=all`); sherpa-onnx uses
-whichever CUDA ExecutionProvider picks
-**Image:** `local/parakeet:sherpa-onnx-v1` — built from `Dockerfile` +
+**Server:** fv-ml1 (Fountain Valley, `10.251.50.54`) — moved from irv-ml1 2026-09-15
+**Port:** 8300 (container 8000)
+**GPU:** **3**, pinned explicitly via `device_ids` — the utility card
+**Image:** `local/parakeet:sherpa-onnx-v4` — built from `Dockerfile` +
`app.py` + `entrypoint.sh` in this directory; **we own all the code**
+**Model:** `parakeet-tdt-0.6b-v3` int8, 25 European languages (~464 MiB)
+
+## Why GPU 3
+
+fv-ml1 has four RTX PRO 6000 Blackwell Max-Q (96 GB each). Three carry the
+vLLM serving seats and run 85–98 % full; GPU 3 is the utility card and was
+empty (2 MiB) at placement time. A 0.6 B int8 ASR model is a rounding error
+next to those seats, but it still has to go somewhere that is not fighting
+them for VRAM.
+
+⚠ The pin is `deploy.resources.reservations.devices[].device_ids: ["3"]`,
+the fleet convention — **not** `count: all`, which is what the dead on-host
+stub used and which would have handed this seat all four cards. Inside the
+container the pinned card presents as `cuda:0`, which is what sherpa-onnx's
+CUDA execution provider takes by default.
## Why not the FastAPI community wrappers
@@ -35,80 +49,93 @@ recognizer API is a three-line call.
| Host path | Container path | Purpose | Restic? |
|---|---|---|---|
-| `/worktank/parakeet/models/` | `/models` | ONNX encoder+decoder+joiner+tokens (~400 MB int8) | excluded (regenerable — re-downloads from the URL on first run if absent) |
+| `/tank/parakeet/models/` | `/models` | ONNX encoder+decoder+joiner+tokens (~464 MiB int8) | excluded (regenerable — re-downloads from the URL on first run if absent) |
-## First-time deploy on irv-ml1
+## ⚠ Verifying the GPU is actually in use
+
+**ONNX Runtime's CUDA execution provider falls back to CPU silently.** It logs
+a warning if you are looking, keeps the process alive, answers `200`, and
+returns *correct transcriptions* — just far slower. So `PROVIDER=cuda` in
+`.env` is a request, not a guarantee, and "the service is up and the text is
+right" does **not** establish that the GPU is doing the work.
+
+Blackwell is the reason this matters here rather than being pedantry: these
+cards are `sm_120`, newer than the compute capabilities ORT's prebuilt CUDA
+binaries have historically shipped kernels for, and the irv-ml1 host this
+stack came from was Ampere `sm_86`. The move is exactly the kind that turns a
+green service into a CPU service without a single error.
+
+The honest check is to watch the card while a transcription runs:
```bash
-# 1. Push compose + Dockerfile + app + entrypoint
-scripts/deploy-stack.sh irv-ml1 parakeet
+# on fv-ml1 — terminal 1
+watch -n0.2 'nvidia-smi --query-compute-apps=pid,process_name,used_memory \
+ --format=csv -i 3'
-# 2. Make sure the models dir exists (one-time, already done from the
-# earlier Shadowfita deploy; this is idempotent)
-ssh -t irv-ml1 'sudo mkdir -p /worktank/parakeet/models && \
- sudo chown -R lkraven:lkraven /worktank/parakeet'
-
-# 3. Build the image and bring up. First boot does a ~400 MB model
-# download via the entrypoint; allow 1–2 minutes before /healthz
-# flips healthy.
-ssh irv-ml1 '
- cd /opt/docker/compose/parakeet && \
- cp -n .env.example .env && \
- docker compose config >/dev/null && \
- docker compose build && \
- docker compose up -d && \
- docker compose logs -f --tail=30
-'
+# terminal 2 — send real audio, not silence
+curl -s -F file=@sample.wav http://127.0.0.1:8300/v1/audio/transcriptions
```
-## Smoke test
+A process must appear **on GPU 3** for the duration. If GPU 3 stays empty, the
+CUDA EP did not initialise and you are on CPU regardless of what `.env` says.
+Confirm with the container's own startup log, which names the providers ORT
+actually registered:
```bash
-# Over WG from the workstation
-curl -F "file=@sample.wav" http://10.100.79.3:8765/transcribe
-# → {"text": "hello world"}
-
-# OpenAI-shape alias (for clients that only know /v1/audio/transcriptions)
-curl -F "file=@sample.wav" http://10.100.79.3:8765/v1/audio/transcriptions
+docker logs parakeet 2>&1 | grep -i 'provider\|cuda\|onnxruntime'
```
-## Switching to the v3 (multilingual) model
+Timing alone is **not** sufficient evidence either way: the int8 model is fast
+enough on a 96-thread EPYC that a CPU fallback still looks brisk on short
+clips. Use the process check as the discriminator and treat throughput as a
+secondary signal.
-The env var `PARAKEET_MODEL_URL` picks the release tarball. To swap
-from the English-only v2 to the 25-language v3:
+## LiteLLM alias
+
+Reached fleet-wide through the gateway rather than by name, engine-neutral so
+the backend can be swapped without touching consumers — the same pattern as
+`ext-tts`:
+
+| alias | mode | backend |
+|---|---|---|
+| `ext-stt` | `audio_transcription` | `http://10.251.50.54:8300/v1` |
+| `whisper-1` | `audio_transcription` | same — OpenAI-compatible name so stock SDK clients work unchanged |
+
+⚠ **The alias uses a raw IP on purpose.** `ana-docker` (where LiteLLM runs)
+resolves no `.internal` names at all — its `/etc/resolv.conf` points at
+`1.1.1.1`/`1.0.0.1`, and the only reason the `ext-tts` backend resolves is a
+hand-pinned `extra_hosts: irv-ml1.nh3.internal:10.6.110.50` in the LiteLLM
+compose. Adding a second hosts entry would mean recreating the container and
+bouncing the gateway for every consumer; an IP costs nothing and cannot go
+stale silently. See the DNS follow-up in `persistent-memory.md`.
+
+Aliases live in LiteLLM's **Postgres store** (`store_model_in_db: true`), not
+in `config.yaml` — that is where the `ext-tts` family lives too, and it means
+adding one needs no gateway restart. It also means `config.yaml` is not a
+complete picture of what the gateway serves: check `/v1/models` or
+`/model/info`, never just the file.
+
+## Deploy
```bash
-ssh irv-ml1 '
- cd /opt/docker/compose/parakeet && \
- sed -i "s|v2-int8|v3-int8|" .env && \
- # Wipe the v2 weights so the entrypoint re-downloads v3 on next up:
- rm -f /worktank/parakeet/models/*.onnx /worktank/parakeet/models/tokens.txt && \
- docker compose up -d && \
- docker compose logs -f --tail=30
-'
+scripts/deploy-stack.sh fv-ml1 parakeet --compose
+# on fv-ml1, first time only:
+# cp .env.example .env # then edit
+docker compose -f /opt/docker/compose/parakeet/compose.yaml build
+docker compose -f /opt/docker/compose/parakeet/compose.yaml up -d
```
-## Upgrade sherpa-onnx or change the base image
+First boot downloads ~464 MiB of ONNX weights into `/tank/parakeet/models/`;
+the healthcheck's `start_period` is 300 s to cover it. Subsequent starts skip
+the download.
-Bump `PARAKEET_TAG` in `.env` to force a rebuild of the local image
-after editing the `Dockerfile`, then:
+## Switching model variants
-```bash
-scripts/deploy-stack.sh irv-ml1 parakeet
-ssh irv-ml1 'cd /opt/docker/compose/parakeet && docker compose build && docker compose up -d'
-```
+One line in `.env`, then `docker compose up -d` (not `restart` — the model URL
+is read by the entrypoint at container creation) and delete the old files from
+`/tank/parakeet/models/` so the entrypoint re-downloads:
-Model files under `/worktank/parakeet/models/` are preserved across
-image rebuilds.
+- **v3** (default) — 25 European languages
+- **v2** — English only; slightly better on English-only material
-## File layout
-
-```
-stacks/parakeet/
-├── Dockerfile # CUDA 12.8 + cuDNN 9 base, sherpa-onnx-cu12 wheel
-├── app.py # FastAPI — ~60 lines
-├── entrypoint.sh # downloads model on first run, then uvicorn
-├── compose.yaml # one service, bind-mounts the models dir
-├── .env.example # template; real .env lives on the server
-└── README.md # this file
-```
+Both are ~460 MiB int8 tarballs from the same k2-fsa release page.
diff --git a/stacks/parakeet/app.py b/stacks/parakeet/app.py
index b8357ab..a8b9dbf 100644
--- a/stacks/parakeet/app.py
+++ b/stacks/parakeet/app.py
@@ -6,7 +6,7 @@ Load the encoder/decoder/joiner/tokens once at startup; serve:
GET /healthz — used by the docker healthcheck
No VAD chunking, no Silero preprocessing — parakeet-tdt handles long-form natively
-and the int8 ONNX model on a 24 GB GPU eats everything we're likely to throw at it.
+and the int8 ONNX model is a rounding error against this host's 96 GB cards.
"""
from __future__ import annotations
@@ -14,6 +14,7 @@ from __future__ import annotations
import io
import logging
import os
+import time
from pathlib import Path
import numpy as np
@@ -59,8 +60,36 @@ def _load_recognizer() -> sherpa_onnx.OfflineRecognizer:
)
+def _warm(rec: "sherpa_onnx.OfflineRecognizer") -> None:
+ """Decode one throwaway buffer before the server accepts traffic.
+
+ ⚠ NOT an optimisation — it moves a 45 s stall out of the first real request.
+ ONNX Runtime's CUDA EP compiles and autotunes its kernels lazily, on the first
+ decode, and on this host (RTX PRO 6000 Blackwell, sm_120) that measured **45.7 s**
+ while every subsequent call was ~0.48 s. Without this, the first caller after any
+ container restart sees a 45 s hang and most clients — LiteLLM's default request
+ timeout included — give up long before it returns, which reads as "the service is
+ broken" rather than "the service is warming".
+
+ The healthcheck's `start_period` (300 s) is what makes paying it here safe.
+ """
+ try:
+ t0 = time.monotonic()
+ stream = rec.create_stream()
+ # 1 s of silence at 16 kHz: enough to force the full encoder/decoder/joiner
+ # path to compile, cheap enough not to matter.
+ stream.accept_waveform(16000, np.zeros(16000, dtype=np.float32))
+ rec.decode_stream(stream)
+ logger.info("warmup decode complete in %.1fs — CUDA kernels compiled", time.monotonic() - t0)
+ except Exception:
+ # A failed warmup must not stop the server: the model is loaded and real
+ # requests would still work, just with the stall back on the first caller.
+ logger.exception("warmup decode failed; first real request will absorb the stall")
+
+
app = FastAPI(title="Parakeet ASR (sherpa-onnx)")
recognizer = _load_recognizer()
+_warm(recognizer)
def _decode(raw: bytes) -> str:
diff --git a/stacks/parakeet/compose.yaml b/stacks/parakeet/compose.yaml
index ae8aef8..b078fe8 100644
--- a/stacks/parakeet/compose.yaml
+++ b/stacks/parakeet/compose.yaml
@@ -6,7 +6,18 @@
# prebuilt int8 quantized Parakeet-TDT from k2-fsa — and wrote our own ~50-line
# wrapper we own end-to-end.
#
-# Model weights (~400 MB int8) download on first run via the entrypoint to
+# HOST: fv-ml1, GPU 3 (relocated from irv-ml1 2026-09-15). GPU 3 is the utility
+# card — the other three carry the vLLM serving seats and run 85-98% full, so a
+# seat placed anywhere else would fight them for VRAM.
+#
+# ⚠ GPU pin is `deploy.resources.reservations.devices[].device_ids`, the fleet
+# convention — NOT `runtime: nvidia` + NVIDIA_VISIBLE_DEVICES, and NOT
+# `count: all` (which is what the dead on-host stub did, and would have let this
+# tiny ASR seat see all four cards including the three that are full).
+# device_ids ["3"] presents that card as cuda:0 INSIDE the container, which is
+# what sherpa-onnx's CUDAExecutionProvider takes by default.
+#
+# Model weights (~460 MB int8) download on first run via the entrypoint to
# ${PARAKEET_MODELS_DIR}/ (persistent host bind mount). Subsequent starts skip
# the download.
#
@@ -25,11 +36,9 @@ services:
dockerfile: Dockerfile
container_name: parakeet
restart: unless-stopped
- runtime: nvidia
ports:
- "${PARAKEET_BIND:-0.0.0.0}:${PARAKEET_PORT}:8000"
environment:
- - NVIDIA_VISIBLE_DEVICES=0
- MODEL_DIR=/models
- MODEL_URL=${PARAKEET_MODEL_URL}
- PROVIDER=${PARAKEET_PROVIDER:-cuda}
@@ -37,17 +46,31 @@ services:
- LOG_LEVEL=${PARAKEET_LOG_LEVEL:-INFO}
volumes:
- ${PARAKEET_MODELS_DIR}:/models
+ deploy:
+ resources:
+ reservations:
+ devices:
+ - driver: nvidia
+ device_ids: ["${PARAKEET_GPU:-3}"]
+ capabilities: [gpu]
+ networks:
+ - tnet
healthcheck:
# Image ships wget (apt) but not curl — use wget so the check actually runs.
test: ["CMD-SHELL", "wget -q -O /dev/null http://localhost:8000/healthz || exit 1"]
interval: 30s
timeout: 10s
retries: 3
- # First boot may include a ~400 MB model download.
+ # First boot may include a ~460 MB model download.
start_period: 300s
labels:
- homepage.group=AI - Audio Tools
- homepage.name=Parakeet ASR
- homepage.icon=mdi-microphone
- - homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (irv-ml1)
- - homepage.href=http://irv-ml1.nh3.internal:${PARAKEET_PORT}
+ - homepage.description=Parakeet-TDT speech-to-text via sherpa-onnx (fv-ml1 GPU 3)
+ - homepage.href=http://10.251.50.54:${PARAKEET_PORT}
+
+networks:
+ tnet:
+ name: traefik-net
+ external: true